This is a tough problem to solve. I currently use NixOS with impermanence [1] which is one way of solving it. Impermanence, in short, allows you to explicitely opt-in the files and directories that are persisted after each reboot.

So while my `~` is not "clean" in the sense that all my config files are in `~/.config`, it is "clean" in the sense that all files in `~` are there for a reason (i.e., `~` is not polluted by the old config/cache of uninstalled packages).

[1] https://github.com/nix-community/impermanence