The north didnt "invade the south" to start the war. The "south" attacked Ft Sumter. Also Im not really sure how you invade your own country. The fact that the majority of the war was fought on southern soil doesnt mean the north "invaded"
1. Their place in society was threatened. You might be poor but you werent the lowest in society. It's the same reason jim crow was allowed to exist and why racism (among other reasons) exists today.
“It will be to the non-slaveholder, equally with the largest slaveholder, the obliteration of caste and the deprivation of important privileges,” he cautioned. “The color of the white man is now, in the South, a title of nobility in his relations as to the negro,” he reminded his readers. “In the Southern slaveholding States, where menial and degrading offices are turned over to be per formed exclusively by the Negro slave, the status and color of the black race becomes the badge of inferiority, and the poorest non-slaveholder may rejoice with the richest of his brethren of the white race, in the distinction of his color. He may be poor, it is true; but there is no point upon which he is so justly proud and sensitive as his privilege of caste; and there is nothing which he would resent with more fierce indignation than the attempt of the Abolitionist to emancipate the slaves and elevate the Negroes to an equality with himself and his family.”
2. There was a genuine fear of a slave revolt.
"Fear of a slave rebellion was palpable. The establishment of a black republic in Haiti and the insurrections, threatened and real, of Gabriel Prosser, Denmark Vesey, and Nat Turner stoked the fires.
John Brown’s raid at Harper’s Ferry sent shock waves through the south. Throughout the decades leading up to 1860, slavery was a burning national issue, and political battles raged over the admission of new states as slave or free. Compromises were struck – the Missouri Compromise, the Compromise of 1850 – but the controversy could not be laid to rest."
etc.
Address by historian and author Gordon Rhea about why non-slaveholding southerners fought for the Confederacy
www.battlefields.org