You can specify different dependencies in pyproject.toml and then use --extra with pip-compile to pull them in. I do this with dev and test sections for example. Constraints seem like they work like you'd expect here. pip-compile also has the -c flag if you want to use that.
I guess I don't know what you mean by OS stuff, but maybe this works for that too?
Hmm. So you generate a requirements.txt and a dev-requirements.txt. It sounds like they have overlapping dependencies? I can't get them to be "layered" where the dev-requirements.txt only has the dev dependencies.