In the 1850's, 1/4 of Southern white families owned slaves while the other 3/4 did not. About 88% of the total slave population was in the South's Confederacy, which was made up of about 3,500,000 people total. People who didn't own slaves were still willing to fight because they got most of their products from slaves and they didn't want to change that.
