Heavy Hor D'Oeuvres instead of Buffet/Dinner plates

My FH were going to have a buffet style dinner, we were trying to find ways to reduce the cost of the reception the dinner plates were over out budget as well. We looked at the hor d'Oeuvre menu and figured we would cut our cost in half if we do heavy hor d'Oeuvres instead. Our wedding is at 5:00pm the invitations went out and said we would have dinner and cocktails..Do you think our guest with be upset if dinner is not served? We are going to add a carving station the will include ham and turkey, an action station of shrimp ang grits plus 11 different items to choose from. I'm just afraid we won't have enough food.