use a lockfile for "old" CI job

[drammock]

@tsbinns @nordme @CarinaFo @scott-huberty

Here's the PR with placeholders. I discussed with @larsoner and he prefers that we try uv first; if it doesn't work or is unweildy we can switch to pixi (see item 3 below for thoughts on this). In addition to the in-line comments, here are a few notes:

1. (perhaps obviously) we'll need to create the lockfile first and commit it to the repo. It should go at tools/uv-ci-old.lock or similar
2. For now the "old" environment will only be used on linux-64 systems, so when creating the lockfile we can skip adding support for other platforms.
3. I'm a bit uncertain if uv.lock is actually going to work; AFAICT uv lock only works in uv's "project mode" where it writes a bunch of stuff to tool.uv.* tables in pyproject.toml. Right now we don't have such tables (and I don't think we should add them in this PR), so we may need to do something a bit unorthodoxed with uv (e.g., write our pins to requirements-ci-old.txt, use uv to convert that to pylock.toml format, and restore the environment from the pylock file). See if you can get it to work with uv, but also feel free to experiment with pixi to see if the flow is simpler/more natural that way.
4. We need to think a bit about how to get mne installed (specifically, the local-to-the-CI clone of mne that reflects the PR being built). It may be possible to get a relative path in the lockfile, but if not, we may need omit mne in the lockfile and add/install the local-to-ci clone of MNE after the environment has been activated.

[drammock]

hey all, here's a snippet based on what we worked out yesterday, for a script to parse pyproject.toml and generate a requirements file from it after hardening the pins:

from pathlib import Path

import requests
import tomllib
from packaging.requirements import Requirement
from packaging.specifiers import SpecifierSet

with open("/path/to/mne/python/pyproject.toml", "rb") as fid:
    toml = tomllib.load(fid)

deps = toml["project"]["dependencies"] + toml["dependency-groups"]["test"]
reqs = [Requirement(dep) for dep in deps]
pinned_reqs = [req for req in reqs if ">" in str(req.specifier)]
unpinned_reqs = list(set(reqs) - set(pinned_reqs))

# TODO triage handling of > vs >=  For now, just assert that we needn't worry
assert all([">=" in str(req) for req in pinned_reqs])
for req in pinned_reqs:
    req.specifier = SpecifierSet(str(req.specifier).replace(">=", "=="))

# TODO make sure pinned version isn't yanked

# recombine pinned and not-pinned
all_reqs = sorted(pinned_reqs + unpinned_reqs, key=str)

outfile = Path("/path/to/mne/python/tools/requirements-ci-old.txt")
outfile.write_text("\n".join([str(req) for req in all_reqs]))

After that we can in theory run this to get a pylock file:

uv pip compile tools/requirements-ci-old.txt --output-file tools/pylock-ci-old.toml --python-version 3.10
mkdir ~/Desktop/some_temp_dir
cd ~/Desktop/some_temp_dir
uv venv --python 3.10
uv pip sync /path/to/mne/python/tools/pylock-ci-old.toml --python-version 3.10

Note the TODO in the first script about pinned version being yanked; I get a warning about scipy 1.11.0 when doing the uv pip sync line. We'll need to figure out what to do about that. Feel free to try some things out and leave comments here saying what worked/didn't work (or open PRs into my branch if you want)

[CarinaFo]

hey all, here's a snippet based on what we worked out yesterday, for a script to parse pyproject.toml and generate a requirements file from it after hardening the pins:

from pathlib import Path

import requests
import tomllib
from packaging.requirements import Requirement
from packaging.specifiers import SpecifierSet

with open("/path/to/mne/python/pyproject.toml", "rb") as fid:
    toml = tomllib.load(fid)

deps = toml["project"]["dependencies"] + toml["dependency-groups"]["test"]
reqs = [Requirement(dep) for dep in deps]
pinned_reqs = [req for req in reqs if ">" in str(req.specifier)]
unpinned_reqs = list(set(reqs) - set(pinned_reqs))

# TODO triage handling of > vs >=  For now, just assert that we needn't worry
assert all([">=" in str(req) for req in pinned_reqs])
for req in pinned_reqs:
    req.specifier = SpecifierSet(str(req.specifier).replace(">=", "=="))

# TODO make sure pinned version isn't yanked

# recombine pinned and not-pinned
all_reqs = sorted(pinned_reqs + unpinned_reqs, key=str)

outfile = Path("/path/to/mne/python/tools/requirements-ci-old.txt")
outfile.write_text("\n".join([str(req) for req in all_reqs]))

After that we can in theory run this to get a pylock file:

uv pip compile tools/requirements-ci-old.txt --output-file tools/pylock-ci-old.toml --python-version 3.10
mkdir ~/Desktop/some_temp_dir
cd ~/Desktop/some_temp_dir
uv venv --python 3.10
uv pip sync /path/to/mne/python/tools/pylock-ci-old.toml --python-version 3.10

Note the TODO in the first script about pinned version being yanked; I get a warning about scipy 1.11.0 when doing the uv pip sync line. We'll need to figure out what to do about that. Feel free to try some things out and leave comments here saying what worked/didn't work (or open PRs into my branch if you want)

regarding the uv pip compile command (uv 0.8.22 (ade2bdbd2 2025-09-23) : pylock-ci-old.toml throws an error:

Expected the output filename to start with pylock. and end with .toml (e.g., pylock.toml, pylock.dev.toml); pylock-ci-old.toml

I renamed the file to pylock.ci-old.toml, that fixed it

[tsbinns]

Have had a go at trying to sort this, here is what I found:

Fixed outdated uv version

An old uv version (0.4; current 0.9) was being used that didn't support the non-standard lockfile name. According to the astral-sh/setup-uv docs, if no version is specified (which it wasn't), it should default to the latest version, but clearly that wasn't happening. Perhaps because an old version of the action was being used (v1; current v7), it didn't support installing uv > 0.4. Have updated the action to v7 and set a minimum version of 0.9 for good measure.

Fixed not being able to sync from some lockfiles

In the SPEC0 job that we would use to create the lockfile, the command had been switched from uv pip compile to uv export. However, the lockfile from export gives the following error in github_actions_dependencies.sh when trying to sync from it:

   Building mne @ file:///home/runner/work/mne-python/mne-python/tools
  × Failed to build `mne @
  │ file:///home/runner/work/mne-python/mne-python/tools`
  ╰─▶ /home/runner/work/mne-python/mne-python/tools/ does not appear to be a
      Python project, as neither `pyproject.toml` nor `setup.py` are present
      in the directory

Seems only the lockfile from uv pip compile works in the way we are trying to sync the env. For that reason, I switched back to creating the lockfile this way.

Fixed missing pip dependency

Also in github_actions_dependencies.sh, we need to add pip to the env, otherwise the subsequent python -m pip install $STD_ARGS $INSTALL_ARGS .$EXTRAS $GROUP_ARG doesn't work. If we use uv pip install pip rather than uv add pip, we can do this without affecting the versions of our 'old' dependencies we get from syncing to the lockfile.

Need to add minimum version pins for test dependencies

In the existing lockfile, there are no test dependencies. If we include --group test in the uv pip compile command, we get these dependencies included, but --resolution lowest-direct means we get some very outdated versions which fail to resolve. I saw there have already been some additional minimum version pins added to the base dependencies. @drammock, can we do the same for the test dependencies causing issues?

Unfortunately, installing these from pip in github_actions_dependencies.sh overwrites some of our 'old' dependency pins, so that doesn't work. An alternative is to write another tools script that modifies the lockfile for a sort of custom resolution, but that'll be a lot more work than just setting some minimum versions for test dependencies.

[drammock]

In the existing lockfile, there are no test dependencies. If we include --group test in the uv pip compile command, we get these dependencies included, but --resolution lowest-direct means we get some very outdated versions which fail to resolve. I saw there have already been some additional minimum version pins added to the base dependencies. @drammock, can we do the same for the test dependencies causing issues?

Yes please add sensible lower pins to test deps as needed

[tsbinns]

Had to adjust some minimum versions further, but the tests are finally passing. Again, I went with what was the latest version available 2 years ago.

As part of #13611, github_actions_check_old.py was added to check that the environment used for testing actually matches the specifications in environment_old.yml. This should be redundant now that we rely on the lockfile. We do make some pip install calls after syncing from the lockfile (e.g., to install MNE in editable mode), but this doesn't affect existing package versions.

We could have a similar check to be super sure the environment matches our expectations, but it might be a bit overkill. WDYT @drammock?

[larsoner]

This should be redundant now that we rely on the lockfile.

In principle yes... but that's assuming that we don't accidentally install other packages later in the pipeline somewhere that could install newer versions. So I think there's value in keeping the check in place after any installation steps complete, right before tests are run. The code is very simple and fast to execute so little cost in keeping it around

[larsoner]

We could have a similar check to be super sure the environment matches our expectations

This part I think would be overkill... just checking the important ones that we want to change + update (using github_actions_check_old.py) seems good enough to me!

[tsbinns]

@larsoner Okay, I can update the check to compare the main dependencies in the env to those listed in the lockfile.

[tsbinns]

Also, I'm only just realising that the existing old env approach included sklearn and pandas, which we don't have in the new approach. The new approach only installs core deps and test deps.

uv's lockfile generation can only be done for whole dep groups at a time, which would mean also adding the full-no-qt group (and likely having to pin a lot more min versions there).

Should we create a new group that only includes sklearn and pandas which we can create the lockfile from, and then just include this in the full groups?

[larsoner]

Yeah I think a new group makes sense here

[tsbinns]

Found when testing that re-compiling the lockfile with --resolution lowest-direct doesn't always lead to the pins in the lockfile being updated if the min version in pyproject.toml is lower than the min version in the lockfile.

If the min version in pyproject.toml is higher than the min version in the lockfile, the lockfile pin gets bumped as expected.

I imagine almost all scenarios would involve increases to the min version in pyproject.toml rather than decreases, but just to be sure the lockfile compiles as expected, I added a check that the versions are as expected.

[tsbinns]

@larsoner Adapted the check that the pyproject.toml min versions and environment versions match.
Also added a check that the lockfile compiles with min versions from pyproject.toml in the first place, since I found this to not always be the case.

I added a new extras group ver-auto-bumped for pandas and scikit-learn. I'm not that fond of the name in case someone wants to change it.

Not sure what else is outstanding. WDYT @drammock?
I'm not happy about having to repeatedly re-activate the venv, but I wasn't successful in finding a fix.

[tsbinns]

I should update the env check that Python is the expected version.