Are you arguing that slavery did not contribute significantly to the wealth of the North, first by using slaves directly, then by integrating advantageously with the South’s economy and the slave trade?
My condensed argument is that Slavery per se does not make countries rich. Rather I am only arguing that slavery, in many countries including the USA, Brazil, and others world-wide, did not contribute to the long term economic success of the nation nearly as much as many claim it does (or solely claim it does in the USA).
We don't have a counterfactual Twin Earth where those countries did not have slaves, but we know for sure slavery is not necessary for a country to be successful. And if we look at countries that had way more slaves, the advantage of simply having slave labor seems hugely insufficient for a country to be successful.
It is possible that the USA had slaves and was able to economically leverage them far greater than all the other countries that had slaves, but this seems unlikely, and such a scenario would also be the biggest gain for a counterfactual scenario that the USA would have been successful without them, too.