Destination Wedding- who to invite?

We have been having a hard time deciding about how to get married. Our guest list has swelled to like 90 people (I wanted just close family a friends for an intimate wedding, but my Fiance has a lot of extended family and friends of the family that "have" to be invited). We are having difficulties agreeing on type of venue and the guest list amount.

My fiance's parents are helping us pay for it and his Dad just offered to take us to Hawaii and even offset the cost somewhat for my immediate family to fly out (parents and siblings) and we get married there. I want to do that and then have a "reception" or party back home where everyone is invited to celebrate our marriage. Is anyone else doing something like that? Who else are you inviting to your destination wedding? My fiance and his Dad both talked about still inviting extended family members and other people. My whole point of doing the desination wedding was to avoid having anyone besides our family be present for the ceremony (I have some social anxiety and want it to be special with just our families). My fiance is onboard and doesn't mind just having the family present, but he is worried some extended family might be offended not to receive an invite to the ceremony. He thinks we can invite other people, but don't have to worry because they won't show up to Hawaii. I'm worried that we do that some people might actually come and then it will be our immediate family and some random extended family of my fiance's. Is anyone else keeping their destination wedding small with just immediate family? Are you inviting others that you know will come? Or are you inviting people with the thought they won't be able to make it? I feel like it's not rude to just tell anyone who is offended that no one but immediate family is invited, but there will be a separate reception back home for everyone.