Before I found poetry I worked at a place that had python projects that were too complicated for pip to download dependencies for. I had developed a tool that, for our projects, could build a set of wheels to deploy a project. I had thought about what a general solution would look like but before implementing anything I found poetry.
Poetry struck me as one of those 80% solutions that might get in the way of a 100% solution. I used it for a number of little projects and felt it was good enough but my analysis was that its dependency resolution strategy was pretty slow and that I didn't believe it was 100% correct.
For my own Python projects I've been really slumming it lately. I make venv's with pycharm and let pycharm manage my dependencies. I also have written a number of "single file Python" programs that I use on my Linux server to do things like burn DTS CD-Rs with complete CD-Text information. For those ones I just stick to the standard library. One of these days I'm going to have to deploy a real python project to a server and I'll have to think of something then.
I convinced myself that the right way to resolve dependencies in Python is to do the same thing maven does and form a graph of the dependencies of all the software versions that could possibly satisfy the constraints.
For wheel files it is practical to do because you can download the metadata in 2 or 3 http range requests out of a wheel. eggs are a problem because you are expected to download and run the python scripts inside the egg before you know anything about the egg. fortunately (1) wheels have mostly replaced eggs and (2) you can build wheels as to the needs of your organization in a private "wheelhouse" used for builds in your organization.
Since it takes a long time to fetch that information it really does have to have a local cache like maven and maybe even a metadata database. It would be nice to see python build cleaned up.
> Poetry struck me as one of those 80% solutions that might get in the way of a 100% solution. I used it for a number of little projects and felt it was good enough but my analysis was that its dependency resolution strategy was pretty slow and that I didn't believe it was 100% correct.
I think part of the problem is that almost all the solutions in the Python ecosystem (yours included if, like you say, you ever need to deploy it) are 80% solutions. It's just each one handles a different 80% of the problem. The last company I worked at also has a reasonable solution involving manual venvs and a bit of scripting on the side that worked fine until we ran into a bug that we couldn't reproduce for ages because one of our dependencies had randomly updated at some point, and we had no way of locking them. 80% fine, 20% a nightmare that I never want to touch again with a bargepole. Even looking through this thread, most people are suggesting solutions with enormous caveats to practical use.
In Poetry's defence, I think the 80% that it caters for tends to be a pretty good 80% for the majority of people, so it's still a good recommendation if you're setting up a new project that's not going too far outside the norms. But I think the 80% thing is definitely a valid criticism of it and most of the rest of the Python packaging ecosystem.
The problem is that people are concluding that "python sucks, let's use another language" every day. If you look at the replies to my comment that is what the rest of them say. That's what people concluded at that place that was having trouble building complex systems.
From that point of view it is not pip vs poetry but pip vs gopm or pip vs rubygems, or...
Maven and npm are perceived as 100% solutions in the Java and Javascript worlds respectively. Sure there are gradle and yarn but both of those are trying to be 100% solutions that are better than the other 100% solutions.
I completely agree, I say all that as someone who is happy to pragmatically work with Python, but who is also very tired of the ecosystem and wouldn't choose it for much outside of data science. Which is unpleasant to admit, because it was my first language, and I did a lot of fun stuff with it.
Working on a python gig targeting an airgapped windows server, different OS architecture from everything else on-hand, restrictions on building and deploying VM's, poor dev/prod workflows etc.
After a couple of months of app dev we ran into weeks of grinding through building a reproducible process (including a sneakernet step) for python dependency resolution and binary builds to get the payload dropped on the server. It was brittle and a bit overwhelming.
Go was "big enough" at the time to be a reasonable next step and after a few weeks of porting we had a cross-compilation process up and were done.
Go actually _shits me to tears_ as a language compared to python, rust, or even Lua, but as a back-end dev it's still where I do most of my work as I'm nearly guaranteed to be able to i) predictably solve something in a way i can share it with a team and ii) get it deployed without too much pain.
I know there's a large part of the Python community that loves Poetry but I've never understood why. It adds a lot of complexity and I've never found it does a good job of solving complex problems, where as requirements and constraints offered by Pip are simple and when you have complex problems provide just enough (and no more) functionality to solve them.
For very simply requirements you can just have a simple requirements file or setup.cfg that lists what packages you need and use pip install.
As your requirements get bigger you can start putting sensible lower bounds on everything. This can be achieved manually or if you're struggling to extend an existing environment run pip freeze and replace all the == with >=.
When you start wanting to have reproducible environments or you're having to keep multiple requirements in sync with each other such as prod requirements and development requirements you use a constraints file. Install all your requirements at once, run pip freeze and save the output as a constraints file, now always specify -c constraints.txt when you pip install in any environment unless you are updating the dependencies in your environment.
Better still, you want to add 1 package without changing an of your other dependencies, just run pip install new_package -c constraints.txt and you either install it or find out what it's not compatible with.
It's not perfect but in my experience other tools have always seemed to add a lot of complexity but only solve a narrow set of problems, which I can usually just workaround in some other way with Pip.
Main thing I want is lock files. I want to know exactly when full dependency closure changes and to see diffs when it does (and be able to reproduce why closure changed). Otherwise you get weird issues with packages updating when not expected. I was surprised linked James Bennett post didn’t mention that at all.
I’ve liked poetry in general, just takes a really long time and sometimes its solver gets stuck on a bad package.
I recently developed a tox plug-in, tox-pin-deps [1], that integrates the pip-compile workflow with dependencies specified in tox.ini (per environment). This was in response to both poetry and Pipenv being non-standard and consistent stumbling blocks for ramping up new devs on the team.
Despite the shiny, I seem to always find my way back to standard tools and text files.
Poetry was the best pragmatic tool for python at one point, but it certainly was the developers project. That is to say, it introduced different and unfamiliar version specifiers to python, these are common place in other ecosystems, but this definitely was one of the barriers to success. Another came down to the applications code style and architecture, poetry is not written well, it tried to reinvent everything from the ground up in python package management, including it's own dependency resolution algorithm and cli library. I never could use it in isolation either, always had to combine it with other tools like Make. I once looked to contribute to poetry, it didn't take long for me to realise that it was poorly written and most of the new development efforts (e.g. the plugin system) were slowly developed by the project creator seemingly in isolation, meanwhile the rest of the development on the project slowly ground to a snails pace, at one point recently no new version (major, minor or patch) had been released in almost 18 months while the bug reports, issues and PRs stacked up, none addressed. Some of the most egregious issues were bugs in the private package repository support, some providers just wouldn't work and this can be a mandatory requirement for some organisations. One of the worst and most damning things about poetry, when it was actually releasing updates, was the regressions, it happened every single time, without fail, you only need to search GitHub issues for the proof. There are so many things that it could do that it does not, probably due to a poorly written foundation, a lack of leadership or organisation, almost definitely all of the above. Now it's just a tool that half works and has created a lot of animosity towards tools attempting to right it's wrongs. More than anything I am disappointed, I struggle to put my frustrations and whole feelings here, this is really what failure looks like from a thousand angles.
Recently I've ventured out to pdm. It is truly a sublime experience. It is everything poetry should have been and with a mere fraction of development team and effort, it really shows how much of a waste poetry is. Pdm has given me hope again. I only wish that more tools like Pycharm would support it, but really the experience isn't missing much at all. For those of you who are feeling the pain of poetry, I can't convey enough how much more pdm is, it's not perfect but if you try it in earnest and you won't be disappointed.
Somehow I became the “poetry guy” at work, and I spend a meaningful amount of time debugging shit for people because the environment somehow got corrupted or something. If there’s no issues I guess it’s fine (dependency resolution time notwithstanding), but when you experience a persistent bug or environment issue, it takes nothing short of expertise to figure out how to sort it out without losing half a day.
Wasn't poetry supposed to be the one that fixed all that?
I haven't tried it, but I've suffered the state of other python tooling and whenever I rant about it to others they're always like "but we have poetry now, and it's good, you've just gotta use that and then everything's sunny and happy"
I'm a big fan of (and small contributor to) pip-tools, but both poetry and pipenv offer management of more stuff, which understandably appeals to folks seeking a simple comprehensible workflow.
Pip-tools is also a bit lower level, offering flexibility and compatibility which I relish, but also requiring more attention from the user to set things up as they wish.
If you or anyone else enjoying pip-tools is a Zsh user and interested in trying out my higher level functions to ease interactive use of pip-tools, venvs, and also isolated app installs (like pipx), I would love some feedback on zpy: https://github.com/AndydeCleyre/zpy
I'm very happy to answer any questions about it right here or as GitHub issues.
Poetry struck me as one of those 80% solutions that might get in the way of a 100% solution. I used it for a number of little projects and felt it was good enough but my analysis was that its dependency resolution strategy was pretty slow and that I didn't believe it was 100% correct.
For my own Python projects I've been really slumming it lately. I make venv's with pycharm and let pycharm manage my dependencies. I also have written a number of "single file Python" programs that I use on my Linux server to do things like burn DTS CD-Rs with complete CD-Text information. For those ones I just stick to the standard library. One of these days I'm going to have to deploy a real python project to a server and I'll have to think of something then.
I convinced myself that the right way to resolve dependencies in Python is to do the same thing maven does and form a graph of the dependencies of all the software versions that could possibly satisfy the constraints.
For wheel files it is practical to do because you can download the metadata in 2 or 3 http range requests out of a wheel. eggs are a problem because you are expected to download and run the python scripts inside the egg before you know anything about the egg. fortunately (1) wheels have mostly replaced eggs and (2) you can build wheels as to the needs of your organization in a private "wheelhouse" used for builds in your organization.
Since it takes a long time to fetch that information it really does have to have a local cache like maven and maybe even a metadata database. It would be nice to see python build cleaned up.