People on the lower end of the wage scale can't just move. Capital can move a lot more easily than people. Manhattan real estate investor can buy multiple homes that he either rents out or just leaves vacant as an extra home for whenever he's tired of Manhattan.
Besides that, everyone else is going to have the same idea: move to where the mortgages and rents are cheaper. That's a good idea at first, but as I said, capital moves more easily than people. That's why globalization has been such a disaster for the working class. Using your logic, the white working class family man in some small town in Ohio should have just moved with his factory to Mexico or China when the executives decided to relocate the plant.