There is no "right to marry" in the US Constitution. Just as there is no "right to drive" in the US Constitution. Both are priveleges extended to qualified persons by each state. And each state has the jurisdiction over who qualifies.
Equal protection of the laws is in the Constitution. So if the law grants drivers licenses, it cannot exclude someone from getting a license just because you hate them.
But what if I hated blind people?

You see, my hate or non-hate is irrelevant to the topic. Blind people cannot drive, ergo they do not get a license to drive unless states want to take that risk. People doing "gay" cannot provide a man and wife (more importantly father & mother) in marriage, so they cannot get a marriage license unless states want to take that risk.
More than just the blind driver implicity share the road. More than just two people married (children) share the marriage contract implicitly. 5 people in DC cannot mandate such widespread damage to the states without their consent.