fix(api): disable uvloop if PYTHONASYNCIODEBUG=1 to prevent segfault …#61281
fix(api): disable uvloop if PYTHONASYNCIODEBUG=1 to prevent segfault …#61281potiuk merged 10 commits intoapache:mainfrom
Conversation
|
Hi @ephraimbuddy @pierrejeambrun @rawwar |
|
Thanks for the review @jason810496 I hear you on the concern regarding code changes for an edge case. However, I’d like to propose a compromise of Code + Docs for a few key reasons: Solving the mystery of #53105: That issue was closed as 'Not Planned' primarily because the root cause was unknown and it was thought to be environment-specific. This PR proves it’s a core Airflow/uvloop incompatibility on Python 3.13+. By fixing this in code, we officially resolve that long-standing mystery. Segfaults are "silent" killers: Since PYTHONASYNCIODEBUG is a standard Python flag (not Airflow-specific), developers often enable it without checking application-specific docs. If it leads to a hard segmentation fault, there is no log output to tell them why it crashed, leading to more issues being opened here. Minimal Overhead: The fix is a simple defensive check at the entry point. It ensures that when a user asks for 'Debug Mode,' Airflow prioritizes stability (standard loop) over performance (uvloop). How about this: I will keep the defensive code fix to prevent the crash, but I will also add a note to the user-facing documentation as you suggested, explaining the fallback. This provides 'defense-in-depth'—the code prevents the crash, and the docs explain why. Would you be open to this approach? I'm happy to push the doc updates to this PR immediately. |
|
Hi @jason810496, just checking in on the proposal above. |
pierrejeambrun
left a comment
There was a problem hiding this comment.
I've asked a couple questions, there are things I don't understand, maybe i'm missing some context. Thanks
pierrejeambrun
left a comment
There was a problem hiding this comment.
I've asked a couple questions, there are things I don't understand, maybe i'm missing some context. Thanks
|
Just a FYI; I'm using Airflow 3.1.6 with python 3.12.12 and still facing the same segfault issue with the api-server. In my case, I had PYTHONDEVMODE enabled, which essentially sets PYTHONASYNCIODEBUG=1 as per https://docs.python.org/3/library/devmode.html#effects-of-the-python-development-mode. This PR doesn't have any milestone, but will this be deployed in the next patch version 3.1.8? |
Yes, but as this #61281 (comment) comment mentioned, it will just add warning logs and add documentation for user when setting PYTHONASYNCIODEBUG on for api-server. I think in most of the case, we only need to set PYTHONASYNCIODEBUG on for Triggerer to ensure the triggers are well written. |
- Add blank line after warning directive to fix RST error - Quote asyncio with backticks to fix spellcheck
|
Thanks @pierrejeambrun for testing locally and confirming that the asyncio.set_event_loop_policy() approach doesn't prevent the segfault. And thanks @TomTomoki for identifying that this also affects Python 3.12 with PYTHONDEVMODE. I've updated the PR to take a documentation-only approach:
The documentation now clearly warns users about this known limitation and suggests alternatives for debugging. Ready for review |
jason810496
left a comment
There was a problem hiding this comment.
There are just a few final nits regarding the user-facing documentation and the PYTHONDEVMODE check, then the PR will be good to go.
Backport failed to create: v3-1-test. View the failure log Run details
You can attempt to backport this manually by running: cherry_picker 004f790 v3-1-testThis should apply the commit to the v3-1-test branch and leave the commit in conflict state marking After you have resolved the conflicts, you can continue the backport process by running: cherry_picker --continueIf you don't have cherry-picker installed, see the installation guide. |
…nt segfault … (apache#61281) * fix(api): disable uvloop if PYTHONASYNCIODEBUG=1 to prevent segfault (issue apache#61214) * docs(newsfragment): add newsfragment for uvloop/asyncio debug segfault fix (apache#61214) * fix(api): add stacklevel for debug warning and fix rst literals * fix: remove else block and correct newsfragment per review feedback * docs: document PYTHONASYNCIODEBUG/PYTHONDEVMODE incompatibility (Python 3.12+) * docs: fix RST formatting and spellcheck errors - Add blank line after warning directive to fix RST error - Quote asyncio with backticks to fix spellcheck * refactor: simplify to warning-only approach per reviewer feedback * refactor: apply reviewer suggestions --------- (cherry picked from commit 004f790) Co-authored-by: subhash-0000 <122723782+subhash-0000@users.noreply.github.com> Co-authored-by: Jason(Zhe-You) Liu <68415893+jason810496@users.noreply.github.com>
…nt segfault … (#61281) (#61933) * fix(api): disable uvloop if PYTHONASYNCIODEBUG=1 to prevent segfault (issue #61214) * docs(newsfragment): add newsfragment for uvloop/asyncio debug segfault fix (#61214) * fix(api): add stacklevel for debug warning and fix rst literals * fix: remove else block and correct newsfragment per review feedback * docs: document PYTHONASYNCIODEBUG/PYTHONDEVMODE incompatibility (Python 3.12+) * docs: fix RST formatting and spellcheck errors - Add blank line after warning directive to fix RST error - Quote asyncio with backticks to fix spellcheck * refactor: simplify to warning-only approach per reviewer feedback * refactor: apply reviewer suggestions --------- (cherry picked from commit 004f790) Co-authored-by: subhash-0000 <122723782+subhash-0000@users.noreply.github.com>
apache#61281) * fix(api): disable uvloop if PYTHONASYNCIODEBUG=1 to prevent segfault (issue apache#61214) * docs(newsfragment): add newsfragment for uvloop/asyncio debug segfault fix (apache#61214) * fix(api): add stacklevel for debug warning and fix rst literals * fix: remove else block and correct newsfragment per review feedback * docs: document PYTHONASYNCIODEBUG/PYTHONDEVMODE incompatibility (Python 3.12+) * docs: fix RST formatting and spellcheck errors - Add blank line after warning directive to fix RST error - Quote asyncio with backticks to fix spellcheck * refactor: simplify to warning-only approach per reviewer feedback * refactor: apply reviewer suggestions --------- Co-authored-by: Jason(Zhe-You) Liu <68415893+jason810496@users.noreply.github.com>
apache#61281) * fix(api): disable uvloop if PYTHONASYNCIODEBUG=1 to prevent segfault (issue apache#61214) * docs(newsfragment): add newsfragment for uvloop/asyncio debug segfault fix (apache#61214) * fix(api): add stacklevel for debug warning and fix rst literals * fix: remove else block and correct newsfragment per review feedback * docs: document PYTHONASYNCIODEBUG/PYTHONDEVMODE incompatibility (Python 3.12+) * docs: fix RST formatting and spellcheck errors - Add blank line after warning directive to fix RST error - Quote asyncio with backticks to fix spellcheck * refactor: simplify to warning-only approach per reviewer feedback * refactor: apply reviewer suggestions --------- Co-authored-by: Jason(Zhe-You) Liu <68415893+jason810496@users.noreply.github.com>
…nt segfault … (#61281) (#61933) * fix(api): disable uvloop if PYTHONASYNCIODEBUG=1 to prevent segfault (issue #61214) * docs(newsfragment): add newsfragment for uvloop/asyncio debug segfault fix (#61214) * fix(api): add stacklevel for debug warning and fix rst literals * fix: remove else block and correct newsfragment per review feedback * docs: document PYTHONASYNCIODEBUG/PYTHONDEVMODE incompatibility (Python 3.12+) * docs: fix RST formatting and spellcheck errors - Add blank line after warning directive to fix RST error - Quote asyncio with backticks to fix spellcheck * refactor: simplify to warning-only approach per reviewer feedback * refactor: apply reviewer suggestions --------- (cherry picked from commit 004f790) Co-authored-by: subhash-0000 <122723782+subhash-0000@users.noreply.github.com>
apache#61281) * fix(api): disable uvloop if PYTHONASYNCIODEBUG=1 to prevent segfault (issue apache#61214) * docs(newsfragment): add newsfragment for uvloop/asyncio debug segfault fix (apache#61214) * fix(api): add stacklevel for debug warning and fix rst literals * fix: remove else block and correct newsfragment per review feedback * docs: document PYTHONASYNCIODEBUG/PYTHONDEVMODE incompatibility (Python 3.12+) * docs: fix RST formatting and spellcheck errors - Add blank line after warning directive to fix RST error - Quote asyncio with backticks to fix spellcheck * refactor: simplify to warning-only approach per reviewer feedback * refactor: apply reviewer suggestions --------- Co-authored-by: Jason(Zhe-You) Liu <68415893+jason810496@users.noreply.github.com>
apache#61281) * fix(api): disable uvloop if PYTHONASYNCIODEBUG=1 to prevent segfault (issue apache#61214) * docs(newsfragment): add newsfragment for uvloop/asyncio debug segfault fix (apache#61214) * fix(api): add stacklevel for debug warning and fix rst literals * fix: remove else block and correct newsfragment per review feedback * docs: document PYTHONASYNCIODEBUG/PYTHONDEVMODE incompatibility (Python 3.12+) * docs: fix RST formatting and spellcheck errors - Add blank line after warning directive to fix RST error - Quote asyncio with backticks to fix spellcheck * refactor: simplify to warning-only approach per reviewer feedback * refactor: apply reviewer suggestions --------- Co-authored-by: Jason(Zhe-You) Liu <68415893+jason810496@users.noreply.github.com>
apache#61281) * fix(api): disable uvloop if PYTHONASYNCIODEBUG=1 to prevent segfault (issue apache#61214) * docs(newsfragment): add newsfragment for uvloop/asyncio debug segfault fix (apache#61214) * fix(api): add stacklevel for debug warning and fix rst literals * fix: remove else block and correct newsfragment per review feedback * docs: document PYTHONASYNCIODEBUG/PYTHONDEVMODE incompatibility (Python 3.12+) * docs: fix RST formatting and spellcheck errors - Add blank line after warning directive to fix RST error - Quote asyncio with backticks to fix spellcheck * refactor: simplify to warning-only approach per reviewer feedback * refactor: apply reviewer suggestions --------- Co-authored-by: Jason(Zhe-You) Liu <68415893+jason810496@users.noreply.github.com>
This PR fixes a segmentation fault in the Airflow API server that occurs when PYTHONASYNCIODEBUG=1 is set on Python 3.13+.
Problem: uvloop is incompatible with asyncio debug mode on Python 3.13, causing the API server to crash on the first request.
Solution: Added a defensive check in the API server entrypoint (main.py) to detect PYTHONASYNCIODEBUG=1. If debug mode is enabled, uvloop is not installed, and a RuntimeWarning is emitted. Otherwise, uvloop is installed as usual for performance.
Documentation: Added a newsfragment 61214.significant.rst to note this change in the release notes.
This ensures the API server is stable under debug mode while retaining performance when debug mode is off.
Closes: #61214