From the responses so far, it would seem that most do not equate marriage with sex, even though we assume this when people get married.
So why should government discriminate against polygamy then?
Because The People have decided it's not in the public interest.
They're free to change their minds and may do so someday.