Loved ones who invite themselves & others

You're Not Invited, but not because I don't love you!
Before even being engaged we decided we wanted a small wedding with just immediate family, Best Man, and Maid of Honor. We both come from large families, and love them, but do not care for huge crowds. We want an intimate experience that can actually reflect our relationship and not just pleasing the masses. Also note that huge weddings are not common on either side. Both mothers say "do what feels right" but my father (who has severe anxiety and despises large events) seems to be the "You must invite ____ because they are your relative". We, the couple, are paying. What is a kind, nice, neutral way to tell our families that we really just don't want the huge event? I was sure of my response until Dad all-of-a-sudden had an opinion and feelings I don't wish to hurt. THANKS