From Bitterly Divided: The South’s Inner Civil War, by David Williams (New Press, 2010), Kindle pp. 245-246:
Though the conflict may have been a rich man’s war, it was not as much of a poor man’s fight as the rich tried to make it. That was true for North and South. On both sides, the lowest of the lower classes tended to be as adamant as the rich in their refusal to fight—or refusal to fight for their region’s dominant regime. In the South, while most Confederate soldiers were nonslaveholders and poorer than their slaveholding neighbors, southerners even poorer still were more likely to dodge the draft, desert, or serve in the Union army. As for the North, James McPherson, in his Battle Cry of Freedom, presents evidence suggesting that the poorest northerners were among the least likely to serve. It was in fact their resistance to the draft, and northern dissent generally, that goes a long way toward explaining how a Confederacy at war with itself as well as the North was able to survive for as long as it did….
Despite the North’s population advantage of two to one, only about a million native-born northerners served in the Union military—roughly the same as the number of southerners who served the Confederacy. Nearly a fourth of the Union armed forces were made up of immigrants, and almost another fourth were southerners, black and white. It was, in the end, southerners who gave the Union armies their numerical superiority on the battlefield. Given the limits of support Lincoln was able to muster in the North, the war’s resolution largely came down to Southerners themselves. Had all soldiers from the South fought for the South, or more precisely for the Richmond regime, the result would have been at least parity on the battlefield and perhaps Confederate victory.