Opinions about invites

My man and I have considered elopement, but both of us have voiced concerns over our families not being there. So we’d like something fairly small.
With that being said, we both come from pretty big families. His parents are divorced and he’s got 8 siblings/half-siblings. He’s already said he probably won’t invite everyone, just the family members he’s actually close to.
I have a very close-knit family on my mom’s side and I know they would all attend. That’s already about 15-20 people alone. My dad’s side is even bigger and would be about 20-25 invited with no idea how many would actually show up, as they all live quite far.However, I was adopted as a baby. It was an open adoption and I grew up being friends with members of my birth family. We haven’t been as close in recent years as we were before (I would go to my grandma’s for weekly family nights in high school), but I have gone to some family events. (For example, my cousin is having a baby. I was invited to the shower and everything since we were close growing up.) I don’t know whether to invite them to be nice or not invite them to keep the wedding small. If I did invite them, that would add another 20-ish to my numbers and I haven’t even gotten to my close friends yet.ALL opinions are welcome! We haven’t decided on anything about our wedding yet, I just want outsider opinions to maybe help me eventually make that decision!