In order to answer this question, one must consider the effect of taxation. If taxes are not paid when due the property being taxed can be confiscated and resold to recover the taxes. On the surface that sounds fair enough but if you look closely it make taxation - especially on the poor - a vehicle to confiscate property without paying full and just value and then selling it to a wealthier person for a very low cost. That makes taxation a vehicle of unjust confiscation or the destruction of ownership. In fact of law, that was the very reason provided for the refusal of the federal government to submit to state taxation. source:
http://www.willamette.edu/wucl/pdf/review/41-1/gunning.pdf
If churches and tax exempt charities are thus taxed then they can be destroyed or lost to the people. That would infringe on the rights protected by the first amendment. On an extension to this argument any funds that a church or charity receives that are not a direct function of their "statement of service" are indeed taxable. Rent collected on properties is taxable, wages collected by officials are taxable, sales or exchanges of properties, other than to another tax exempt organization are taxable.
Tax exempt does not mean tax free.