Day wedding anxiety

Our invitations just went out for our wedding on the Sunday of Labor Day weekend, and there’s been people on my FH’s side who have been pretty judgmental about the fact that it’s a day wedding, instead of the standard evening wedding. The ceremony starts at 11:30, reception ends at 5, and we’re having an after party at the hotel, but people are still like “that’s too early”, “no one will want to drink or dance,” or “expect people to dress super casual” (even though the wedding I’ll be at a very upscale ballroom). There’s literally nothing we can do about timing at this point, and obviously we did this because it’s best for our budget, but I wanted to know what people’s experiences have been at day weddings. Was it still fun even if it didn’t have a night time atmosphere? I’ve never been to one myself so any insight would be helpful.