How would you distinguish a Man of Honor from the Best Man?

Typically, the Best Man would wear a tie the same color as the other groomsmen, and a Man of Honor would wear one in the same color as the bridesmaids' dresses. However, you can do it however you like.

I guess I should mention that our wedding party only consists of 2-4 people depending upon who decides to attend. So it could very well be just a MoH and BM.

Well, he'll be on your side, so there's that. Perhaps they could have different boutonierres, or different colored vests/ties.

My brother was my MOH. He wore the same suit/vest/tie as the guys who stood up with H.

If it's just him and the BM, you could probably have them wear different colors if you want to mix it up. I just wanted to keep our colors simple b/c he was already going to stand out from the girls, and I didn't want too much going on. Plus I wanted him to be able to take photos with the other guys and not look out of place.