Guest list issues

So my fiancé and I want our wedding to be more intimate, but he has a huge family. We decided to try to limit our wedding to no more than 100 guests and so far our guest list has about 40 people from my family/friends and 80 from his. We're trying to cut back since we're at 120, so we agreed to not invite family members who aren't involved in our lives/we've never met. This limits who we invite from my family more than his, which I'm ok with.
The issue is that I have never met my fiancés uncle. I know his uncle and his wife live within like 20 minutes of us. In the 5 years we've been together I have never met them so we cut them from our guest list. We told some of my fiancés siblings this and they're saying we "can't just not invite them". Now I'm feeling frustrated and don't know if we should just invite them or not. I feel like it's kind of unfair to exclude some of my family members, but like none of my fh's family members. My fiancé agrees with me, but we both are people pleasers and don't want to cause issues. Any suggestions?