The Bill of Rights — of which the capstone is the First Amendment — was added to the Constitution in 1791 at the insistence of the Anti-Federalists. Contrary to what most Americans probably think, the Bill of Rights does not grant anyone rights. What the government grants, the government can take away. Since no power was granted to the federal government in the body of the Constitution to infringe what are commonly referred to as Americans’ “First Amendment freedoms,” the government has absolutely no authority to do so. That is why James Madison, who did not initially support adding a bill of rights to the Constitution, said that no “great and important power” could be exercised by Congress unless it was “evidently and necessarily involved in an express power.” The First Amendment merely reinforces the idea that the federal government lacks the authority under the Constitution to abridge Americans’ existing freedoms. The individual rights to freely exercise one’s religion, speak freely, publish freely, peaceably assemble, and petition the government are natural rights independent of the Constitution — just like the right to freedom of association.
The freedom of association simply means that a person has the right to associate, not with whomever he chooses, but with whoever is willing to associate with him. Inherent in the right to associate is the right not to associate. Any person has the right not to associate with whomever he chooses.
In a free society, any person or group of persons has the right to associate with any other person or group of persons willing to associate with him or it on the basis of any standard and for any reason. And likewise, any person or group of persons has the right not to associate with any other person or group of persons on the basis of any standard and for any reason.
It doesn’t matter whether a government bureaucrat or a person who was refused association believes that the actions of the refusing person or group are illogical, unreasonable, irrational, hateful, discriminatory, bigoted, or racist. What matters is freedom.