fabfile.py (Fabric) could be used as a Makefile in Python. If you don't ever need to ssh to other machines ti run your tasks, you could use pyinvoke library directly (tasks.py).
https://www.fabfile.org/
It is easy to add command line arguments to the tasks, configure them using files (json, yaml), environment variables, to split the task definitions into several modules/namespaces.
Having used fabric in the past, I've always found it just as easy to use a shell script and make files.
There's always some level of bootstrapping a project (installing packages/software, compiling libraries and dependencies) where it's easier to just to write a shell script than to program python to do. E.g. How do you get fabric installed on a system?
There's also been this longevity of sorts that Make seems to have gotten right. People just keep going back to it because it's simple.
I've been moving away from using shell scripts in a tools/ directory to using Python Invoke (http://www.pyinvoke.org), which is the library underlying fabric.
I used bash scripts for years, but for a lot of reasons made the switch:
- It was always painful to create small libraries of functions used across multiple scripts in a project
- It's difficult to consistently configure them with per-user settings. I've written bash implementations of settings management, Invoke handles this for me.
- I'd still have to reach for Python whenever I needed to do anything with arrays or dicts, etc.
- Getting error handling correct can be a chore
Invoke has a lot of nice to haves to:
- Autogenerated help output
- Autocomplete out of the box
- Very easy to add tasks, just a Python function
- Easy to run shell code when needed
- Very powerful config management when needed
- Supports namespacing, task dependencies, running multiple tasks at once and deduplicating them
It's not perfect, but it's a lot better than my hand rolled scripts were.
Groxx replies hits the point. I might work with a small number of platforms, but the "super-simple" qualifier is the point. The point at which you need dictionaries (associative arrays) in your install script, not to mention settings management beyond a make include is the point at which you've outgrown make.
it's also far, far, far easier to make it work predictably on multiple platforms. and easier to understand and change later. that can get nightmarishly hard in make/bash, once you go outside the super-simple realm.
Wow I've never seen such bloated python project before. It has 10 dependancies with 2 additional optional dependancies and the introduction/tutorial is absurdly overspecific.
The first example they use to describe the tool is:
> Cufflinks is a tool to assemble transcripts, calculate abundance and conduct a differential expression analysis on RNA-Seq data. his example shows how to create a typical Cufflinks workflow with Snakemake. It assumes that mapped RNA-Seq data for four samples 101-104 is given as bam files.
This is epitome of non-programmer programming, colour me disappointed.
And yet it's still less crufty than the "by hackers for hackers" GNU Automake and less over engineered than the "made by real professional programmers at a real big tech company" Luigi. Would love the hear if you have any suggestions for actual alternatives for doing this type of automation beyond what make can neatly deal with rather than just going "eww.. it has dependencies"; "eww.. it's made by bioinformaticists".
It is easy to add command line arguments to the tasks, configure them using files (json, yaml), environment variables, to split the task definitions into several modules/namespaces.