I think you just answered your own question. Congress shouldn't pass laws to restrict as that would be against the first 10 amendments that exist to limit the powers of the Congress and the President. But it doesn't limit the powers of the State.
You sure?
If what you say was true, some states might still have slaves.
Sorry, but s usual, you're wrong.