Skip to content

CI testing with Python 3.12 is very slow #87

@apdavison

Description

@apdavison

The build_and_test job for Python 3.12 consistently takes ~19 minutes vs ~55s for 3.14 and ~3 minutes for 3.9.

I asked Claude to investigate this by running tests locally on my machine. It found the following difference:

Metric Python 3.12 Python 3.14 Ratio
Without coverage 6.16s 6.11s 1.0x
With coverage 881.41s (14:41) 24.95s 35x

Investigating further, it found that "the Python 3.12 test performance regression is caused by a known CPython 3.12 bug (python/cpython#127953) where line-number lookup during tracing has quadratic time complexity for large code objects."

This affects the openminds Python package because it contains (auto-generated) files with huge module-level code objects, notably:

  • parcellation_entity_version.py — 285,263 lines (latest) / 276,602 lines (v4)
  • parcellation_entity.py — 35K / 19K lines
  • uberon_parcellation.py — ~28K lines

When pytest-cov enables tracing (via sys.settrace or sys.monitoring), importing these enormous files triggers the quadratic line-number lookup bug in CPython 3.12, turning a 0.4s import into a 4.5 minute operation.

Possible fixes

  1. Drop coverage for 3.12 in CI — Run tests without --cov on 3.12
  2. Pin Python 3.12 to a patched release — There is a backport PR #132268 for 3.12 that fixes this, but it will probably not be included in a 3.12.x release
  3. Restructure the generated code — Split the enormous files into smaller modules so no single code object is so large (this would be a bigger change to the build pipeline)
  4. Replace Python 3.12 with 3.13 in the CI matrix — Since 3.13 has partial fixes and 3.14 works perfectly

Sources:

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions