]I disagree with a few phrases of your post, particularly the "better off" notion, but mostly I strongly agree.
It's a position based on contemporary evidence, backed up by demographic data available from the period, and also by first hand accounts by eyewitnesses like Fredrick Law Omstead, an abolitionist, and his travels through the South to Texas in his journal. White immigrant labor had no social safety nets, outside what small charity was available from churches, and had to rely on factory work if any was available when the agricultural jobs ended, which left many of them out in the cold, and with little food, making them susceptible to cholera and other epidemics, which were routine after the waves of immigrants started appearing, and that remained the case until well into the 20th century. Studies show that even 'natives', defined as those here 3 generations or longer, were also affected by the massive influxes; average heights and lifespans after 1820 show an average loss of height of 15% and also a lifespans shortened by the same amount over the periods. While alcoholism was rampant, with a per capita consumption 3 times that of Europeans, it can only partly account for the decline in health.
Perhaps the worst thing about the American experience with enslavement was the association of status with color. For most of human history such was not the case. Slavery was an equal opportunity oppressor.
The Irish would disagree; they suffered from discrimination as well, only worse, since no one regarded them as worth the money to feed or save when they weren't employed. Omstead also notes their treatment, along with poor Germans, compared to slaves. I'll cite some from his diary if anybody cares to know what an eyewitness report from one of the era's more notable abolitionists says about the division of labor then, like who was getting the crappy dangerous jobs and who wasn't, or whose skeletons those are in the banks of the river levees, just covered over where they dropped on the job by the 10's of thousands over the years.
And besides, you would wrong about that; the Arab slavers demanded African male slaves be castrated, via full frontal castration, before purchasing them, and they were in the African slave business far longer than we; don't think they were castrating whites or any other colors the same way. So yes, color did matter, far longer than the colonies existed.
The poisonous consequences of creating an ineffaceable association of appearance and oppression hardly needs elaboration.
This is judging the past by modern popular notions, and doesn't really mean anything as far as history goes; slavery has been around a long time,
I will recommend a good history of early American settlement in the south, where one can indeed find a difference in the way slaves were treated and what rights they had then compared to the growth of chattel slavery and the decline of their rights and liberties, which did worsen over time as their numbers increased. White slavery finally disappeared early on, but was mostly a distinction without a difference, since being 'free' labor meant a more insecure livelihood and a 35% chance of early death from disease and malnutrition, high childhood mortality, etc., while slaves had the same mortality percentages as whites, and their 'value' calculated up to the age of 70 on average, mortality being higher in the south than the north, because of the swamps and year round disease possibilities. Northern cities like New York City had slums where childhood morality ran into 200 to 300 per 1,000, and nobody blinked a eye at it. They were expendable; thousands of replacements landed at the docks every week.
So how did this happen? I believe it was because of the genetic advantages black Africans had, specifically resistance to malaria, that enhanced their chance of surviving the environment in the agricultural regions they were used. Whites (including indentured servants) or Native Americans just dropped like flies; they weren't worth the investment. Black Africans, unfortunately, were hardier stock.
Yes, nothing to disagree with there; they were bought in Africa from black slave traders there because of their resistance to some tropical diseases, which, ironically is also related to their sickle cell anemia susceptibility.