I am not one to install python packages using my distro's package manager, but I totally agree with the sentiment that we need a more standard build/dependency management system in python. I like poetry, and I think most people are heading that way, but it doesn't seem to play super nice with pyenv (which is a critical tool) a lot of the time, and I think that a first party endorsement of the "one true build system" a la golang or rust would be a huge step.
I landed with poetry as well. The issue with python dependency managment in my book is, that it is incredibly (and needlessly) hard to learn how to do it properly. It took me years to figure out how things should work and there are still issues now and then that cost me a few hours to figure out and restore things.
Meanwhile when I use Rust all these things are taken care of in cargo. It is part of the language. There is one right way to do things and the way is supported by comfortable tooling, that works so well that you literally don't even consider thinking about anything else.
The way python does dependencies is totally unpythonic. The fact that it is 2021 and this isn't fixed or at least the number one priority of things that need fixing sheds a dark shadow on the whole language – a language that I like to use.
Poetry is good. But it isn't as good as cargo, because it also has to deal with all the legacy cruft. To run code developed with poetry on a non-poetry system you have to figure out all the ways of dealing with envs, paths and such.
Issues like these get me a little fired up, because the collective brainpower wasted on something that should have been elegantly solved in one place is gigantic.
To run code developed with Poetry, you shouldn’t even know it’s developed with Poetry. It should be released as a source distribution or a wheel on PyPI or as a conda package if it contains nontrivial binary extensions. These distributions should be then either packaged by the OS or they should be installed with all their dependencies into a separate virtual environment in /opt.
Sure, if I release the code I tend to do just that. However a lot of the code I have been writing was to be used on servers where installing poetry was not an option. That means there was no straightforward, well documented or otherwise easy way to copy the project onto the server an "just have it work". And this was not due to me not taking care of my dev environment.
Now I know how to do this, so this is not a problem. My complaint was mostly, that this was a waste of time.
If installing Poetry wasn’t an option, deploying a frozen set of dependencies should be still relatively easy with plain ”pip freeze” and “pip install -r”.
I’m not a Python dev. I do need to occasionally run things written in Python. I made the mistake of trying to get pip + Conda + pyenv (or whatever) to install a fairly simple tool. I have no idea how the dev got their setup working, but it was totally and utterly unreproducable, even after they sat down on my computer for several hours. In that amount of time, I could have probably rewritten it in PHP (that’s actually exactly what I ended up doing while they attempted to get a Docker container running).
Needless to say, I will only use the distro package manager these days. I know the versions are (probably) compatible, maintainers will usually backport security patches, etc. You get none of that using whatever flavor of the week python package manager.
I think this actually gets to the crux of the matter. The existing python dependency management tools (especially the new shiny ones like poetry) are very much designed by python devs, for python devs. What you're describing is a totally different use case, which is running released software in a sensible way.
The distro package managers are probably the best place for that, but bridging the gap between them and the python ecosystem is an obvious challenge.
It's not really the fault of Python that somebody fucked their environment beyond belief. I've always used pip and almost never had a problem. A "fairly simple" application should not require the tools you mentioned.
That, my friend, is called a filter bubble. And your comment is not productive btw. I for one never "fucked my environment", I only ever used distro packages and I still couldn't make a simple script even start without it spitting cryptic error messages about modules or paths.
I'm not a Python dev; I just needed to run a tool that didn't have any other alternatives.
Meanwhile, I can download scripts written in a range of other languages and just fire 1-2 commands and the thing will work.
Same BTW. And I started having a Docker image for each Python script that I need to run (I'm not a Python dev). Took me a while but I've mostly tamed the beast. And at least I can run the occasional script that has no alternatives.
I had to learn this by FUBAR'ing my system a long time ago, but my setup process for working with a python package from PyPI (i.e. not installable by or updated enough in OS's package manager) nowadays is:
- sudo apt install pip3
- pip3 install --user --upgrade pipenv
(In workdir):
- pipenv install --three package
- pipenv run package --option
Works like a charm and doesn't mess with my system.
This is especially frustrating to read because one of the main selling points of Conda is reproducibility. In data science teams, I've found it indispensable for making sure people can all run each others' project code.
So for anyone reading this in the future: don't try to use Pyenv to install Conda. Pyenv tries to set up shims for every binary in the Conda env, which will likely break your PATH.
Pyenv supports installing Conda because Anaconda used to be "just" a Python distribution.
They can otherwise coexist without trouble on the same system.
My understanding of the problem is that Pyenv attempts to detect the contents of "/bin" relative to the top level of every Python installation that it manages.
It does this so that it can set up its shim to handle any executable that gets installed in any Pyenv-managed environment.
This is how Pyenv creates the "foobar is not available in your current environment, but is available in x.y.z" message. It's also a much more reliable solution than trying to explicitly whitelist every possible script that might get installed.
The problem is that this was only designed to work for Python executables and scripts installed by Pip. Conda environments can contain a lot more than that; it's not hard to end up with an entire C compiler toolchain in there (possibly even both GCC and LLVM) or even Coreutils.
If Pyenv detects `bin/gcc` in a Conda env, it will set up a system-wide shim for GCC, which no longer passes the `gcc` command along to the OS, but intercepts it, only to inform you that no such command exists in the current env!
So it's not that Pyenv hoses Conda envs. It's that Pyenv can hose PATH if you have it manage a Conda installation, and if that Conda installation ends up with non-Python stuff in `/bin`.
Obviously I don't know what exactly was broken when you tried to set up that application. But this particular adverse interaction bit me at work a few years ago, and ever since then I have insisted that Pyenv should never manage a Conda installation.
I think that's a reasonable policy anyway, in light of the facts that:
1) Conda isn't really a "Python distribution" anymore.
2) The Pyenv installer just runs the opaque Conda installer script and there's basically no way to control the version that gets installed.
3) They are different tools that serve different purposes and it doesn't make sense to have one manage the other anyway.
4) You probably shouldn't use the Python that's installed in the base Conda environment anyway. You need that to run Conda itself, and you want to keep the list of requirements small to make sure that updates can progress cleanly. It's basically the same as any Linux package manager like APT. Except of course, those tools don't generally support "environments" other than chroot.
Both Conda and pyenv are for managing (their own) Python installations to run different Python versions in parallel, it doesn't really compute to try and use both at the same time.
Yes. I tried to use Gitpod (a cloud development env platform) which ships with pyenv. Poetry would very stubbornly only use the system python and completely ignore whatever the pyenv shim was pointing to.
Golang is a great model to follow. I think the fundamental thing that python (and ruby, when I still used it) has taught me is to run a mile if a language is without a robust dependency management system. The pain is just not worth it, even for a nice language with an otherwise robust ecosystem.
With golang, it's not just the dependency management side, gofmt (and various bits of other tooling) is also an incredible blessing. Yes, prescriptivism can sometimes feel weird to us computer nerds, but sometimes it's just much better to have a canonical way of doing common tasks. It means onboarding people is much easier, and infrastructure/code/whatever is more reusable.
Go is terrible for distributions to package due to the poor library versioning. Also, static linking makes security updates really expensive to build at scale.
