Elitists in the U.K most likely referred to the Irish as "trash" when they literally starved them into submission after the potato famine of the 1840's. The poor Irish who managed to get to the U.S. were quickly scooped up by the Yankees when they got off the boats and used as cannon fodder in the Civil War.
The "elitists" in the UK were more concerned about the underclasses in the UK than Ireland.They had very real issues with the poor, of which there were many.
The much vaunted empire worked for some but not for many.
If you had money to invest in slaves or other goods it was a great time. If you didnt have that then it brought you little benefit.
There seem to have been competing visions for the new world at this point. The english aristocratic vision of a penal colony acting as a safety valve for the UK underclass and the Quaker vision of a theocratic state based on Quaker theology. Both visions were based on class although one was closer to what we would call acceptable today.
As a discussion point it is interesting that the US today has one of the largest prison populations in the World. The prison population is a commodity. Is that a million miles away from the Virginia penal colony ?