You're right. Most earn it. Which means they come from the poor and middle classes. So, if you got rid of all of the rich people, all it means is that the poor and middle classes would make more.
It was EXACTLY that kind of thinking that led to the expulsion of the Lebanese from Uganda, the Ibo from Nigeria, and the white farmers from Zimbabwe. If we get rid of the people with all the money, there will be more for the rest of us.
Too bad it doesn't work that way. Rich people are rich because they have habits and values that enable that.