This very much. If I'm using your library, I've already committed to it's architecture and API with all its flaws. And my users don't care for the technology I use. Even if they're not that good, I can build wrapper over the worst aspects and then just focus on building features. New features are nice, but I'm more interested in getting security updates for the current version that I have.

When it's time to go for a refactoring, the trade-off between costs and returns are worth it as you can go for years between those huge refactors.