Always provide a relevant TI context in Dag callback by IndiTheo · Pull Request #61274 · apache/airflow

IndiTheo · 2026-01-31T00:33:13Z

This PR strives to make sense out of TI contexts passed to Dag callbacks. Rather than always passing the last task lexicogrpahically, the new logic passes the last "relevant" task to Dag's resulting state.

Behavior outline:

On regular failure, pass the latest finished task out of the failed ones.
On timeout, pass the task that started last out of the unfinished tasks.
On "tasks deadlocked", pass a task that should have run next but wasn't able to.
On success, pass the latest succeeded task.

There's a possibility it's better to pass no "ti" key at all to Dag-level contexts, or provide a None value. This might break the existing code even more, so currently the PR only improves the relevancy of the task passed. Yet we should consider the option of not passing any task instance in the future.

closes DAG on_failure_callback uses wrong context #26760

Was generative AI tooling used to co-author this PR?

Yes (please specify the tool below)
deepwiki.com was used to find the relevant logic in the repo, and an embedded chat was used to make a skeleton for tests.

Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
When adding dependency, check compliance with the ASF 3rd Party License Policy.
For significant user-facing changes create newsfragment: {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

IndiTheo · 2026-02-04T20:28:07Z

I want to get some feedback on the behavior, whether it's the desired one or if anything can be made better.

I will update the docs to explain the behavior of passed TIs to dag callbacks.

Nataneljpwd

Looks good, I have left a few comments on the changes, overall, great find!

IndiTheo · 2026-03-06T10:00:01Z

Slightly confused about the milestone

eladkal · 2026-03-06T10:47:28Z

Slightly confused about the milestone

I'll set it for 3.1.9 but this is subject to confirmation from the release manager of 3.1.9

kaxil · 2026-03-06T13:34:48Z

Since technically, this is a chance in behavior, let's put it in 3.2, can you add a newsfragment too please @Asquator : https://github.com/apache/airflow/tree/main/airflow-core/newsfragments

Co-authored-by: Elad Kalif <45845474+eladkal@users.noreply.github.com>

eladkal · 2026-03-08T08:19:02Z

static checks are failing

IndiTheo · 2026-03-08T13:58:14Z

The test is failing because my newsfragment has multiple lines, and it can't...

Do I change the type to significant, or avoid the elaborate explanations by including just one general sentence?

Generally I'm not sure why we have to limit this to just one line... I don't think it's a significant change but a minor improvement, and I still wanted to include some details.

IndiTheo · 2026-03-08T15:10:00Z

Newsfragment validation should pass now.

boring-cyborg · 2026-03-08T17:46:51Z

Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions.

* Skeleton solution * Handle more cases * Refactored None handling * Fixed task name * Materialized tasks for double iteration * Use default argument in min * Use default argument in max * Use default argument in max * Changed end_date to start_date * Fix logic for on_success_callback * Handle the case where ti has no start_date * Logic for DAG testing * Cosmetics * Simplified logic * Pass last succeeded TI on success * Added template method * Cosmetics * More cosmetics * Lint * Added default keys for mypy check to succeed * Updated the docs * Fixed timezone aware datetime comparison * Clarified Dag run timeouts in the docs * Fix condition to look for failures across all relevant TIs * Mypy & ruff * Timezone awareness for deadlocked tasks case * Select from all tis for success callback * Improved deadlock callback logic * Mypy & ruff * Made the test more explicit * Format * Updated backward docs version to 3.1.9 according to milestone * Revert "Updated backward docs version to 3.1.9 according to milestone" This reverts commit 4defea7. * Type hint Thank you Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com> * Reapply "Updated backward docs version to 3.1.9 according to milestone" This reverts commit a31476a. * Removed redundant iter() call * Removed redundant logic in deadlock case * Updated backward docs version to 3.2.0 according to milestone Co-authored-by: Elad Kalif <45845474+eladkal@users.noreply.github.com> * Added a newsfragment entry * Removed trailing whitespaces * Shortened nesfragment to just one line --------- Co-authored-by: TheoS <asquator@proton.me> Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com> Co-authored-by: Elad Kalif <45845474+eladkal@users.noreply.github.com>

TheoS added 3 commits January 29, 2026 19:21

Skeleton solution

41823b4

Handle more cases

1c820ba

Refactored None handling

30c9e7e

IndiTheo requested review from XD-DENG and ashb as code owners January 31, 2026 00:33

boring-cyborg Bot added the area:Scheduler including HA (high availability) scheduler label Jan 31, 2026

TheoS and others added 3 commits January 31, 2026 02:44

Fixed task name

2bd3e09

Materialized tasks for double iteration

e3ca084

Merge branch 'main' into bugfix/improve-dag-callback-context

17ef75f

IndiTheo changed the title ~~Always provide a relevant task instance in Dag callback~~ Always provide a relevant TI context in Dag callback Jan 31, 2026

Use default argument in min

9cad5ae

IndiTheo commented Jan 31, 2026

View reviewed changes

Comment thread airflow-core/src/airflow/jobs/scheduler_job_runner.py Outdated

Use default argument in max

947025a

IndiTheo marked this pull request as draft February 4, 2026 19:43

Use default argument in max

b9272ff

IndiTheo marked this pull request as ready for review February 4, 2026 20:26

Merge branch 'main' into bugfix/improve-dag-callback-context

fc48d5a

IndiTheo mentioned this pull request Feb 4, 2026

DAG on_failure_callback uses wrong context #26760

Closed

2 tasks

Merge branch 'main' into bugfix/improve-dag-callback-context

8ef4b9c

Nataneljpwd reviewed Feb 5, 2026

View reviewed changes

TheoS added 5 commits February 6, 2026 20:50

Changed end_date to start_date

a564078

Fix logic for on_success_callback

05aae04

Handle the case where ti has no start_date

90cc202

Logic for DAG testing

a218556

Cosmetics

7e6dcbb

IndiTheo requested a review from Nataneljpwd February 7, 2026 12:07

IndiTheo added 3 commits February 7, 2026 14:07

Merge branch 'main' into bugfix/improve-dag-callback-context

a7bf75b

Merge branch 'main' into bugfix/improve-dag-callback-context

bb23ffc

Merge branch 'main' into bugfix/improve-dag-callback-context

2c42728

TheoS and others added 3 commits March 6, 2026 11:39

Removed redundant iter() call

fcc73e0

Removed redundant logic in deadlock case

08253d9

Merge branch 'main' into bugfix/improve-dag-callback-context

c45f152

eladkal modified the milestones: Airflow 3.2.0, Airflow 3.1.9 Mar 6, 2026

eladkal added the backport-to-v3-1-test label Mar 6, 2026

kaxil modified the milestones: Airflow 3.1.9, Airflow 3.2.0 Mar 6, 2026

kaxil removed the backport-to-v3-1-test label Mar 6, 2026

eladkal reviewed Mar 6, 2026

View reviewed changes

Comment thread airflow-core/docs/administration-and-deployment/logging-monitoring/callbacks.rst Outdated

IndiTheo and others added 2 commits March 6, 2026 16:37

Updated backward docs version to 3.2.0 according to milestone

2198ac7

Co-authored-by: Elad Kalif <45845474+eladkal@users.noreply.github.com>

Added a newsfragment entry

ad5f8ff

kaxil approved these changes Mar 6, 2026

View reviewed changes

eladkal approved these changes Mar 8, 2026

View reviewed changes

TheoS and others added 2 commits March 8, 2026 08:27

Removed trailing whitespaces

7d58717

Merge branch 'main' into bugfix/improve-dag-callback-context

9b91104

Shortened nesfragment to just one line

45213d5

Merge branch 'main' into bugfix/improve-dag-callback-context

b6041fe

eladkal merged commit 43c17c6 into apache:main Mar 8, 2026
73 checks passed

eladkal mentioned this pull request Apr 8, 2026

Support updating DagRun note from notifier/callback context in Airflow 3 #64890

Open

Conversation

IndiTheo commented Jan 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Was generative AI tooling used to co-author this PR?

Uh oh!

Uh oh!

IndiTheo commented Feb 4, 2026

Uh oh!

Nataneljpwd left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

IndiTheo commented Mar 6, 2026

Uh oh!

eladkal commented Mar 6, 2026

Uh oh!

kaxil commented Mar 6, 2026

Uh oh!

Uh oh!

eladkal commented Mar 8, 2026

Uh oh!

IndiTheo commented Mar 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

IndiTheo commented Mar 8, 2026

Uh oh!

Uh oh!

boring-cyborg Bot commented Mar 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

IndiTheo commented Jan 31, 2026 •

edited

Loading

IndiTheo commented Mar 8, 2026 •

edited

Loading