The origin of the power given to congress is from the people. Where do you think the power should be centered and from what source?
If the power of Congress is granted by the people, then this implies the people have this power to grant. Who among us may claim such power? Do you claim the power to personally tax your neighbor? To personally write laws which he must obey? If you don't have this power, and neither do any of "the people", how can they delegate this power to someone else? I presume you would not accept me delegating the right to take your car to my brother, on the grounds that I do not have that right to begin with. It is invalid to delegate a right you don't have in the first place, is it not?
I do not see any valid center from which such power could spring, unless God came down from the heavens and asserted it. Man does not have the ability to invent non-existent rights. He may delegate rights he already has, but he cannot validly create new ones, no matter how many people agree to it.