Open
Conversation
boring-cyborg
bot
added
area:airflow-ctl
area:dev-tools
area:docker-tests
area:helm-chart
Dev-iL
added
full tests needed
This was referenced Mar 14, 2026
Dev-iL
force-pushed
the
2603/py3.14
branch
6 times, most recently
from
b82af37 to
c91f015
Compare
Contributor
|
#protm |
8 tasks
Contributor
+1 |
Contributor
|
Currently trying to get an AWS maintainer to have a look at aws/amazon-redshift-python-driver#272, jumping through a lot of hoops and being forwarded to different folks, but I'm getting close. |
Contributor
|
@Dev-iL can you rebase again? I'll merge after |
Collaborator
Author
|
I removed 4 commits meant to deal with missing upstream 3.14 constraints - to see what happens. |
- constraints are skipped entirely - greenlet pin updated
Importing airflow.models.Connection pulled in airflow.configuration, and airflow.configuration initializes secrets backends at import time. That initialization itself does `from airflow.models import Connection`, so under Python 3.14 the lazy import path could re-enter `airflow.models.__getattr__ before connection.py` had finished defining the Connection class. The result was the ImportError: Module "airflow.models.connection" does not define a "Connection"; plus repeated SQLAlchemy warnings about redefining the model during partial imports. By deferring: `from airflow.configuration import conf, ensure_secrets_loaded` until the method actually needs them, importing the module no longer triggers that cycle. `Connection` gets fully defined first, and only later, when get_connection_from_secrets() runs, do we touch configuration and secrets loading. That keeps behavior the same at runtime, but removes the import-time recursion.
Before this fix there were two separate issues in the migration-test setup for Python 3.14: 1. The migration workflow always passes --airflow-extras pydantic. 2. For Python 3.14, the minimum Airflow version is resolved to 3.2.0 by get_min_airflow_version_for_python.py, and apache-airflow[pydantic]==3.2.0 is not a valid thing to install. So when constraints installation fails, the fallback path tries to install an invalid spec.
Python 3.14 changed the default multiprocessing start method from 'fork' to 'forkserver' on Linux. The forkserver start method is slower because each new process must import modules from scratch rather than copying the parent's address space. This makes `multiprocessing.Manager()` initialization take longer, causing the test to exceed its 10s timeout.
With 'forkserver' (Python 3.14 default on Linux), the child process starts from a clean slate and needs time to import modules and execute the target function before the SIGTERM-ignoring signal handlers are installed. The old sleep(0) was not enough — SIGTERM could arrive before the handlers were in place, killing the process before the test reached the SIGKILL path. Use a multiprocessing.Event to synchronize: the child signals readiness only after its signal handlers are installed. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Python 3.14 changed the default multiprocessing start method from 'fork' to 'forkserver' on Linux. Like 'spawn', 'forkserver' doesn't share the parent's address space, so mock patches applied in the test process are invisible to worker subprocesses. - Skip tests that mock across process boundaries on non-fork methods - Add test_executor_lazy_worker_spawning to verify that non-fork start methods defer worker creation and skip gc.freeze - Make test_multiple_team_executors_isolation and test_global_executor_without_team_name assert the correct worker count for each start method instead of assuming pre-spawning - Remove skip from test_clean_stop_on_signal (works on all methods) and increase timeout from 5s to 30s for forkserver overhead Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The PROD image build installed all provider wheels regardless of Python version compatibility. Providers like google and amazon that exclude Python 3.14 were still passed to pip, causing resolution failures (e.g. ray has no cp314 wheel on PyPI). Two fixes: - get_distribution_specs.py now reads each wheel's Requires-Python metadata and skips incompatible wheels instead of passing them to pip. - The requires-python specifier generation used !=3.14 which per PEP 440 only excludes 3.14.0, not 3.14.3. Changed to !=3.14.* wildcard.
Non-DB core tests use xdist which runs all test types in a single pytest process. With 2059 items across 4 workers, memory accumulates until the OOM killer strikes at ~86% completion (exit code 137). Split core test types into 2 groups (API/Always/CLI and Core/Other/Serialization), similar to how provider tests already use _split_list with NUMBER_OF_LOW_DEP_SLICES. Each group gets ~1000 items, well under the ~1770 threshold where OOM occurs. Update selective_checks test expectations to reflect the 2-group split.
The old code had a check-then-act race (if `os.path.exists` → `os.remove`), which fails when the file doesn't exist at removal time. `contextlib.suppress(FileNotFoundError)` handles this atomically — if the file is missing (never created in this xdist worker, or removed between check and delete), it's silently ignored.
Replace multiprocessing.Process with subprocess.Popen running minimal inline scripts. multiprocessing.Process uses fork(), which duplicates the entire xdist worker memory. At 95% test completion the worker has accumulated hundreds of MBs; forking it triggers the OOM killer (exit code 137) on Python 3.14. subprocess.Popen starts a fresh lightweight process (~10MB) without copying the parent's memory, avoiding the OOM entirely. Also replace the racy ps -ax process counting in TestKillChildProcessesByPids with psutil.pid_exists() checks on the specific PID — the old approach was non-deterministic because unrelated processes could start/stop between measurements.
When a provider declares excluded-python-versions in provider.yaml, every dependency string referencing that provider in pyproject.toml must carry a matching python_version marker. Missing markers cause excluded providers to be silently installed as transitive dependencies (e.g. aiobotocore pulling in amazon on Python 3.14). The new check-excluded-provider-markers hook reads exclusions from provider.yaml and validates all dependency strings in pyproject.toml at commit time, preventing regressions like the one fixed in the previous commit.
This is imported unconditionally in scripts/in_container/in_container_utils.py
When running provider tests (e.g., Providers[google]) on a Python version where the provider is excluded, generate_args_for_pytest removes the test directories but the skip check in _run_test only triggers when its own --ignore filter removes something. Since the directories are already gone, pytest runs with zero test directories and crashes on unrecognized custom arguments like --run-db-tests-only. Add are_all_test_paths_excluded() that explicitly checks whether all test paths for the given test type are in the excluded/suspended provider set, using the same provider.yaml data. Call it early in _run_test to skip before any Docker operations.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The plan is as follows:
Was generative AI tooling used to co-author this PR?
Generated-by: Claude Opus 4.6 following the guidelines
{pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.