The thing is, that what is thought to harm society is often subjective. When we see people injured, deprived of basic rights, etc, that is what should guide policy.
"No man has a natural right to commit aggression on the equal rights of another; and this is all from which the laws ought to restrain him; every man is under the natural duty of contributing to the necessities of the society; and this is all the laws should enforce on him; and, no man having a natural right to be the judge between himself and another, it is his natural duty to submit to the umpirage of an impartial third."
--
Thomas Jefferson; from letter to Francis Gilmer (June 7, 1816)