Request for session: workflows with make / snakemake

[staeiou]

Or other tools, NextFlow?

[ickc]

I might be able to help on GNU-make depending on my schedule. (I currently have a group meeting colliding with exactly the same time slot. But I occasionally can skip it and come here.)

Ideas:

• Assumption: ignore the most common case of building source codes but focus on make syntax and broad range of usage in general (because for those who maintain a project, they should already be using make.)

• why make and its limitation

  • building dependency graph

  • trivial to parallelize if above wrote correctly

  • Limited to 1 node (no MPI. However combining mpibash and make might be able to overcome this---my next idea to try.)

  • UNIX-first, if not only

• installing make on OSes

  • Windows (need help here)

  • macOS: brew (because of GPLv3, macOS's old)

  • Linux: nothing

• makefile naming convention

• running make

  • Standard/common targets?

  • Shell completion?

• makefile programming

  • simple terminology, syntax and pattern rules

    • demo: wildcard of all .md and create .pdf, or something similar

  • var. and env. var.

  • branching depending on env. (macOS/Linux), overriding env. var.

  • code reuse

    • include

    • "template"

  • Advanced use:

    • automatic variables

    • Secondary expansion

    • function

    • meta-programming

  • Unusual use:

    • Use PHONY target for bundling tiny bash programs in a bigger program using make

    • Use absolute path to create dependencies outside current dir.

• references

  • bash guide

  • make maintainer's blog

• examples: either

  • use existing examples from my projects (easy on presenter, difficult to follow by the audiences), or

  • ideally, create a repo from the ground up including the presentation and examples (time consuming on presenter, and likely to create a half-finished product that the audiences might not be easy to follow as well)

[karldw]

Would it be possible to talk about choosing between these tools, or at least why you use make? GNU make vs NextFlow vs snakemake vs drake vs all the other build automation tools.

For GNU make on Windows, one option is Rtools, which people might already have. Another is proposed in this gist, but I haven't tested it.

[ickc]

Would it be possible to talk about choosing between these tools, or at least why you use make? GNU make vs NextFlow vs snakemake vs drake vs all the other build automation tools.

Are you asking me?

For GNU make on Windows, one option is Rtools, which people might already have. Another is proposed in this gist, but I haven't tested it.

And concerning using UNIX tools on Windows (caveat: I ain't Windows developer), the general approach seems to be:

• Cygwin
• Windows Subsystem for Linux (64-bit only)
• Install Ubuntu using Hyper-V (64-bit and the "Pro-variants" of Windows only). Other VM solutions is also ok but Hyper-V has minimal overhead.

But make (and bash) are very UNIXy, so using them on Windows feels odd.

Digression: I think it is counter-productive to expect every topic to be covering all the bases. For example, in the past there's a topic on data-analysis using Python/R, where stuffs are taught in parallel using both languages. It will be easier just to focus on Python in a row, vice versa. Same idea could be applied in this situation: rather than trying to make make relevant to Windows users, probably one should just ignore Windows users in that presentation (instructions can be provided to turn their Windows into UNIX if they want to), and then probably another tool for automation is shown, where Windows users can be benefited from (or may even be only applicable to Windows.)

[karldw]

@ickc, sure, if you have any thoughts on choosing an automation tool, I'd be happy to hear them! If you're all about GNU Make, that's great too.

[ickc]

Among the ones I've heard of: autotools, CMake, SCons. I have some experiences with the first 2 but not the others (nor those in your list). But the first 2 seems to be specialized in building softwares so it might not be suitable for general use.

May be we should back up by a little bit and ask what are the kinds of automation this suggestion is really about? May be give an example or two. And then may be make a list of tools (that some people here know and can teach) that might solve these problems. Providing there's experts willing to teach and using different tools, it will be interesting to see how each tackle the same task, and perhaps different tasks easy to be achieved in one tool but difficult/slow in another. I had briefly trying to find comparison between different build tools but what I read is abstract and brief and cannot help one to determine the proper tool for their own use case.

For example, I personally are looking for a way to do something make-like but scales beyond a single node (that's why I've thought about combining mpibash with make); and also something make-like but might not be file based but data within a Python program, but is similar to make (i.e. specifying rules and what you need but not programming imperatively.)

[ickc]

It also depends on if you want the session to be in-depth tutorials in 1 or 2 tools, or like a survey on many different tools (say 15 min. per tool.) The later one sounds like really fun as an audience, and I really want to be able to make it.