How to handle different versions of modules

[tischi]

Hi, regarding versioning of nextflow modules: this and this discussion seems to indicate that, within nf-core, handling different versions of modules is not completely solved, what do you think?

I am thinking we could indicate the version of a module in its name, e.g. instead of calling the module folder cellpose we call it cellpose-3.1.0 and then we would maintain multiple cellpose modules, one per version.

Since modules can have parameters, it would also be important to document this properly. For the nf-core cellpose module I could find this pointer to the documentation. Since this is pointing to latest it is not immediately clear for which version of CellPose this documentation is valid. I think we would need a mechanism where we always point to the correct documentation for a specific version.

What do you think @BioinfoTongLI @krokicki ?

[BioinfoTongLI]

do you mind explaining your use-case?
I personally would stick with one modules version regardless the version of the software inside, as along as the input and output is always the same. The actually algorithm version should be logged at the execution level (i.e. write the algorithm version into versions.yml).
In this case, when you need to switch version, just update the container path at runtime. It's a bit hacky, but should work.
The other solution I would suggest, if the one above doesn't work, is to have experiemental github branches for less standard version. So, when installing you could do nf-core install -g https://github.com/BioImageTools/nextflow-modules.git -b experimental install xxx

[tischi]

do you mind explaining your use-case?

A life scientist running the same nextflow workflow one year later should readily get the same results (without needing to do anything special or complicated).

So, when installing you could do nf-core install -g https://github.com/BioImageTools/nextflow-modules.git -b experimental install xxx

I guess something like that could also work, one could teach users to work with a specific git commit / tag / branch of the tools repository. Is that what you meant?

[BioinfoTongLI]

I'd say the ultimate solution would be asking users to push their working pipeline somewhere and tag/archive it.
There are certain controls that we can do at the module level. but it's after all just a shell containing algorithms.
Many things could still change after the modules installation (parameter fine-tuning etc.) And those is only saved in the working env of the pipeline, not within each module.

[tischi]

push their working pipeline

Could you elaborate what you specifically mean by "pipeline"? I think if it is just the main.nf it would not be enough, because this contains include statements of modules that may be subject to changes. I thus guess it would need to be (i) the project specific main.nf and (ii) a specific commit of the whole nextflow-modules repository?!

[krokicki]

@tischi The pipelines only download modules when you initially install them. It's not like Python or Maven dependencies which are just cached. Each pipeline actually contains its own static copy of the modules, so there is no need to worry about changes to the nextflow-modules repository. As long as the pipeline repo is versioned, it is reproducible.

[tischi]

@krokicki just very kindly had a short call with me and explained me the difference between a modules repository and a pipeline repository, which I did not know 😬 . That helps a lot and in fact seems to handle many of the versioning worries that I had.