Relatively few southerners owned slaves, but it was a part of the culture. However, economically, it was becoming less and less tenable for the big slave owners to have such a large part of their wealth tied up in slaves when that wealth could be put to a more lucrative purpose. My guess is slavery would have died out from economic pressures in a few generations anyway.
Incorrect.
AND correct at the same time.
Slavery was more lucrative in the 1850s than it had ever been at any point in its history.
The profits that could be had by a Southern cotton grower were extreme. Even those who didn't grow cotton but relied on other labor-intensive crops, such as tobacco and rice, were generally doing very well.
At the same time, you're correct, the writing was on the wall for slavery economically, only most people didn't realize it.
Why?
Because the British HATED buying American cotton when they had vast regions of their empire where cotton would thrive -- primarily India and Egypt. Egyptian cotton, which was just starting to come into its own in the 1860s, was also a different type, and produced much finer cloth. Even had the South not tried for independence, British cotton began hitting the market hard in the 1870s. Southern cotton production was a fraction of what it had been pre-war, and even so the prices were so depressed that most growers lost money throughout the 1870s.
Then there was the double pronged nastiness of soil depletion and the coming boll wevil.
Vast areas of the South were just starting to show the effects of soil depletion. You simply can't keep growing the same crop on the same soil year after year and expect the soil to hold up.