"use virtualenvs and install your dependencies from an unfiltered, unsupervised, untrusted source" is certainly the solution most aligned with the rest of the industry but I struggle to see any technical benefit from it. Other than perhaps "move faster and break more things".
Virtualenvs don't require using unfiltered/unsupervised/untrusted sources. They're a place to install things into, not a place to get things from.
The specific model that distros could adopt, if we went this route, is that each Python package builds into a .whl, they're build-dependencies of applications, and applications install .whls into a virtualenv at build time. You'd still restrict packages to come from the distro with the usual policies (built from source code, compliant with licensing, not contacting the network at build time, etc. etc.).
So, for instance, installing "python-somelib" would get you a /usr/share/python-wheels/somelib-1.0.whl, built from source. The build process of "someapp" would create a /usr/share/someapp/venv, pip install that wheel into that virtualenv, and then symlink /usr/bin/someapp to /usr/share/someapp/venv/bin/someapp.
The PEP goes into more details about why virtualenvs are recommended, and I can give you a whole host of subtle reasons, but that's not really the point - the point is that it's defensible, not that it's perfect, and that it's very easy to implement with what works today. So if you ask the PSF to come up with something that magically solves the problems and makes people sad if necessary, this is literally what they're going to come up with. If you don't like that outcome (and there are good reasons not to like it!), then you shouldn't ask for them to arbitrarily pick an outcome some people won't like.
The problem here is that distros only supply one version of the package which means you gain nothing by having that same version installed in multiple virtualenvs.
So I guess the problem actually lies in the python library ecosystem that's becoming a npm like dependency hell.
The relevant question here is probably whether the library ecosystem is like that because the packaging tooling sucks or is it the other way around?
- Almost all distros don't just supply one version of a package. They try to avoid it, but I can't recall one with a hard rule against it. For instance, Debian packages multiple versions of autoconf https://packages.debian.org/search?keywords=autoconf2 , the Linux kernel, etc.
- One reason that distros try not to install multiple versions of a package is that it's hard to specify which one you want. If you have, say, requests 1.0 and 2.0 installed, which one does "import requests" get you? The virtualenv approach, where "import requests" simply does not work in an un-virtualenv'd Python, avoids this problem entirely. Each application can independently build-depend on python-requests-1 or python-requests-2, and each user can create their own virtualenv and pip install /usr/share/wheels/requests-1.whl or requests-2.whl as they prefer.
- Even if you do not have different versions of the same library, it may well be the case that you have two different libraries with the same importable name. As a great example, see CJ Wright's talk from PackagingCon last week "Will the Real Slugify Please Stand Up" https://pretalx.com/packagingcon-2021/talk/P3983F/ (I expect they'll post videos online soon). tl;dr there are three packages you can get via "import slugify", and they expose different APIs.
- The possibility you haven't accounted for is "The library ecosystem is like that because things are fast-moving, because people have actual problems they want to solve, and upgrading dependencies and sorting out conflicts is work." It would be great for that work to be done, but we go back to the problem I mentioned at the top - limited volunteer time. In the absence of time to engineer things perfectly, your options are to ship something that's engineered imperfectly or decide not to ship it. We went through the dark ages of "We'll ship the next Debian release when all the bugs are solved" over a decade ago, and it turns out that this doesn't actually help users in any way.
> The possibility you haven't accounted for is "The library ecosystem is like that because things are fast-moving, because people have actual problems they want to solve, and upgrading dependencies and sorting out conflicts is work."
Sorry, I was overly aggressive in my previous comments. All I want is the Python ecosystem to acknowledge some people prefer stable over fast moving, recognize the importance of that and work towards a solution that isn't horrible for their use case.
We all depend on someone somewhere deep in our stack caring for stability, even when we don't realize it.
They have acknowledged it, noted the use case, and decided that they are unable to support it. Python has this in common with most other programming languages.
Providing long-term backwards compatibility with the ability to share libraries and update them system-wise is difficult and imposes very high costs on an ecosystem. Only a small number of tech stacks have ever done this (for example, C and Perl) - and not only are they so old that it wasn’t a conscious choice, they have stagnated as a result.
