FH's sister as Best Man

Sure, that's one option! My FH's best man is also a woman. The rest of the groomsmen will be in black suits, so we told her to wear any black dress she wanted.

I attended a wedding last year where there was a groomswoman. She was in a dress that was the color of the groomsmen's suits (grey).

My brother will be on my side, but dressed like the groomsmen

She can wear a bridesmaid dress or a dress in the same color as the groomsman. Maybe see if the bridesmaids style you pick is offered in the groomsmen's color?

That is an option or she can wear a dress in the color or the guys suits

If she is his best woman, it is a given that she will be on his side. Whether she matches the groomsmen, the bridesmaids, or no one, is up to the two of you. All are acceptable.

It depends on her really. My friend's FH has 1 groomsmen who was a woman and she wore a suit like the other groomsmen. Another wedding I went to, the groomswoman wore a dress not like the BMs but in the wedding colors.