Skip to content

Fix bug in scheduler with Job dependency leakage#369

Merged
cigamit merged 1 commit into
mainfrom
dep_leakage
Jun 4, 2026
Merged

Fix bug in scheduler with Job dependency leakage#369
cigamit merged 1 commit into
mainfrom
dep_leakage

Conversation

@cigamit

@cigamit cigamit commented Jun 3, 2026

Copy link
Copy Markdown
Contributor

Job Dependency cascading leakage

This is the infamous job dependency failure bug, where if you launch multiple jobs at once, and one job depends on a project or inventory sync and is the first one processed, 1st, all the current pending jobs stay in pending even if they are not waiting on anything, and 2nd, if the sync fails all the jobs fail with the error about that dependency failing

So generate_dependencies is creating a list of all pending jobs and their dependencies. The problem is that as it loops over each job, it is adding the current running full list of dependencies to each job, instead of just that jobs dependencies.

Ex:

Job 1 = Job 1 deps
Job 2 = Job 1 + Job 2 deps
Job 3 = Job 1 + 2 + 3
etc...

Simple fix, also added some tests to double check this to ensure that dependency leakage doesn't happen. Verified that with reverting the change, the tests do indeed fail and detect the leakage.

@cigamit cigamit requested a review from TheWitness June 3, 2026 22:01
@cigamit cigamit self-assigned this Jun 3, 2026
Copilot AI review requested due to automatic review settings June 3, 2026 22:01
@cigamit cigamit added bug Something isn't working python Pull requests that update python code labels Jun 3, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Fixes a scheduler bug where dependency discovery for multiple pending jobs could “leak” dependencies from earlier jobs into later jobs, incorrectly blocking and/or failing unrelated jobs when a dependency update fails.

Changes:

  • Scope dependent_jobs linking to per-task dependencies (job_deps) instead of a running cumulative list.
  • Add a functional regression test to ensure dependency generation remains isolated per job when multiple jobs are pending.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File Description
awx/main/tests/functional/task_management/test_scheduler.py Adds a regression test asserting each pending job only links to its own inventory update dependency (no cross-job leakage).
awx/main/scheduler/task_manager.py Fixes DependencyManager.generate_dependencies() to attach only the current task’s dependencies instead of a cumulative list.

@cigamit cigamit merged commit 0a24fcb into main Jun 4, 2026
1 check passed
@cigamit cigamit deleted the dep_leakage branch June 4, 2026 19:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working python Pull requests that update python code

Development

Successfully merging this pull request may close these issues.

3 participants