How small is too small?

Our guest count has cut down considerably from what we were originally planning. We now will be having small, intimate wedding with about 50 guests. My question is, in your opinion how small is to small for a table head count? I have a lot of extenuating circumstances family wise and will only be having 4 people from my fathers side at my wedding(he will not be there). My mother’s family and them do not get along very well. I know they would keep it civil for me but I don’t want anyone to feel tense and not be able to enjoy themselves. Would it be weird to sit them as a table of four by themselves? The rest of our tables are ranging 6-8. I don’t see it as a huge deal but my FH is concerned about hurting anyone’s feelings.