maybe not on an SSD but it definitely helps a lot on HDD by virtue of having far less disk traffic. The kernel's method for figuring out which modules to load is effectively to load every single module that might be compatible with a given device in series and then ask the module for its opinion before unloading it, and then once it has a list of all (self-reported) compatible modules for a given device it picks one and reloads it.
IDK; at the time i was using gentoo, in which it's natural not to have more modules than necessary because part of the installation process involves generating your own kernel configuration.
Even though it's not the normal way to install debian there ought to be some sort of way to build your own custom kernels and modules without interferance from the package manager (or you can just run it all manually and hope that you dont end up in a conflict with apt). Gentoo is the only distro where it's mandatory but im pretty sure this is supported on just about every distribution since it would be necessary for maintainers.
