Optimize CI: consolidate workflows, fix caching, speed up e2e tests (47min → 15min)#772
Merged
vikrantpuppala merged 3 commits intomainfrom Apr 14, 2026
Merged
Conversation
Workflow consolidation:
- Delete integration.yml and daily-telemetry-e2e.yml (redundant with
coverage workflow which already runs all e2e tests)
- Add push-to-main trigger to coverage workflow
- Run all tests (including telemetry) in single pytest invocation with
--dist=loadgroup to respect xdist_group markers for isolation
Fix pyarrow cache:
- Remove cache-path: .venv-pyarrow from pyarrow jobs. Poetry always
creates .venv regardless of the cache-path input, so the cache was
never saved ("Path does not exist" error). The cache-suffix already
differentiates keys between variants.
Fix 3.14 post-test DNS hang:
- Add enable_telemetry=False to unit test DUMMY_CONNECTION_ARGS that
use server_hostname="foo". This prevents FeatureFlagsContext from
making real HTTP calls to fake hosts, eliminating ~8min hang from
ThreadPoolExecutor threads timing out on DNS on protected runners.
Improve e2e test parallelization:
- Split TestPySQLLargeQueriesSuite into 3 separate classes
(TestPySQLLargeWideResultSet, TestPySQLLargeNarrowResultSet,
TestPySQLLongRunningQuery) so xdist distributes them across workers
instead of all landing on one.
Speed up slow tests:
- Reduce large result set sizes from 300MB to 100MB (still validates
large fetches, lz4, chunking, row integrity)
- Start test_long_running_query at scale_factor=50 instead of 1 to
skip ramp-up iterations that finish instantly
Co-authored-by: Isaac
Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com>
vikrantpuppala
temporarily deployed
to
azure-prod
GitHub Actions
Inactive
- Use -n 4 instead of -n auto in coverage workflow. The e2e tests are network-bound (waiting on warehouse), not CPU-bound, so 4 workers on a 2-CPU runner is fine and doubles parallelism. - Lower test_long_running_query min_duration from 3 min to 1 min. The test validates long-running query completion — 1 minute is sufficient and saves ~4 min per variant. - Split lz4 on/off loop in test_query_with_large_wide_result_set into separate parametrized test cases so xdist can run them on different workers instead of sequentially in one test. Co-authored-by: Isaac Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com>
vikrantpuppala
temporarily deployed
to
azure-prod
GitHub Actions
Inactive
vikrantpuppala
commented
Apr 13, 2026
vikrantpuppala
commented
Apr 13, 2026
vikrantpuppala
changed the title
vikrantpuppala
commented
Apr 13, 2026
jprakash-db
approved these changes
Apr 14, 2026
Contributor
jprakash-db
left a comment
jprakash-db
left a comment
There was a problem hiding this comment.
Overall LGTM. Thanks for making the changes
Per review feedback from jprakash-db: - Remove mixin classes (LargeWideResultSetMixin, etc) — inline the test methods directly into the test classes in test_driver.py - Remove backward-compat LargeQueriesMixin alias (nothing uses it) - Rename _LargeQueryRowHelper — replaced entirely by inlining - Convert large_queries_mixin.py to just a fetch_rows() helper function Co-authored-by: Isaac Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com>
vikrantpuppala
temporarily deployed
to
azure-prod
GitHub Actions
Inactive
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
integration.ymlanddaily-telemetry-e2e.yml— coverage workflow already runs all e2e tests. Addpush: maintrigger. Run all tests (including telemetry) in a single pytest invocation with--dist=loadgroupfor xdist_group isolation.cache-path: .venv-pyarrow— poetry always creates.venv, so the cache was never saved ("Path does not exist" error). 3.14 PyArrow jobs dropped from 18min → 3min once cache populated.enable_telemetry=Falseto unit test dummy connection args. Unit tests usingserver_hostname="foo"triggered real HTTP calls — on protected runners this caused an 8-min process hang. 3.14 unit tests dropped from 17min → 2min.TestPySQLLargeQueriesSuiteinto 3 separate classes and split lz4 on/off into separate parametrized cases so xdist distributes slow tests across 4 workers.-n 4instead of-n auto(2 CPUs). E2e tests are network-bound (waiting on warehouse), not CPU-bound.test_long_running_querythreshold 3min → 1min, starting scale_factor 1 → 50.Results (measured)
Test plan
SKIP_COVERAGE_CHECK = CI workflow changes only, no source code coverage impact
This pull request was AI-assisted by Isaac.