[CALCITE-7420] Convert RelOptRulesTest to Quidem scripts#4815
Draft
julianhyde wants to merge 51 commits intoapache:mainfrom
Draft
[CALCITE-7420] Convert RelOptRulesTest to Quidem scripts#4815julianhyde wants to merge 51 commits intoapache:mainfrom
julianhyde wants to merge 51 commits intoapache:mainfrom
Conversation
…sistent with Scott.dept
Add infrastructure for rule-based plan tests in Quidem (.iq) files: - Add SubPlanCommand (!sub-plan) that shows a SQL logical plan before and after applying named CoreRules via HepPlanner - Update data() to recursively walk subdirectories for .iq files - Add sql/rule/aggregate.iq with testPullAggregateThroughUnion, migrated from RelOptRulesTest.testPullAggregateThroughUnion Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Move the following tests from RelOptRulesTest.java/xml to sql/rule/aggregate.iq: - testPullAggregateThroughUnion2 (AGGREGATE_UNION_AGGREGATE_SECOND/FIRST) - testPullAggregateThroughUnionAndAddProjects (AGGREGATE_PROJECT_MERGE, AGGREGATE_UNION_AGGREGATE) - testPullAggregateThroughUnionWithAlias (AGGREGATE_PROJECT_MERGE, AGGREGATE_UNION_AGGREGATE) - testAggregateProjectMerge (AGGREGATE_PROJECT_MERGE) - testAggregateGroupingSetsProjectMerge (AGGREGATE_PROJECT_MERGE) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Migrates tests from RelOptRulesTest.java + XML to .iq files: - filter.iq: testFilterProjectTransposeRule2, testFilterProjectTransposeRule3 - join.iq: testAddRedundantSemiJoinRule, testExtractJoinFilterRule - sample.iq: testFilterSampleTransposeWithBernoulli, testFilterSampleTransposeWithSystem, testFilterSampleTransposeWithSystemAndSeed, testSampleToFilter, testSampleToFilterWithSeed - union.iq: testUnionToDistinctRule Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ormat (batch 3) Migrated tests for: - JOIN_DERIVE_IS_NOT_NULL_FILTER_RULE (14 tests) → join-derive.iq - SORT_REMOVE_CONSTANT_KEYS, SORT_REMOVE_REDUNDANT, SORT_REMOVE_DUPLICATE_KEYS (18 tests) → sort-remove.iq - UNION_MERGE, INTERSECT_MERGE, MINUS_MERGE (7 tests) → union.iq - PROJECT_JOIN_JOIN_REMOVE, PROJECT_JOIN_REMOVE (10 tests) → project-join-remove.iq Tests not migrated (require withFactory or withVolcanoPlanner): - testJoinDeriveIsNotNullFilterRule14 (Oracle NVL) - testSortRemoveConstantKeyDoesNotRemoveOrderByRandom (PostgreSQL RANDOM) - testSortRemovalOneKeyConstant (withVolcanoPlanner with collation) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ormat (batch 4) Creates 3 new rule test files and updates 7 existing ones: * aggregate-join-transpose.iq: 31 tests for AGGREGATE_JOIN_TRANSPOSE_EXTENDED * project-join-transpose.iq: 24 tests for PROJECT_FILTER_TRANSPOSE, PROJECT_MERGE, PROJECT_SET_OP_TRANSPOSE, and related rules * reduce-expressions.iq: 31 tests for FILTER_REDUCE_EXPRESSIONS, PROJECT_REDUCE_EXPRESSIONS, JOIN_REDUCE_EXPRESSIONS Also adds descriptions and '!ok'/'Not using !ok' explanations to all tests in rule/*.iq files, per coding standards. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…format (batch 5) Migrate tests from RelOptRulesTest.java and RelOptRulesTest.xml to Quidem .iq scripts in core/src/test/resources/sql/rule/: New files: - aggregate-constants.iq (9 tests): AGGREGATE_ANY_PULL_UP_CONSTANTS, AGGREGATE_VALUES - aggregate-expand-distinct-aggregates.iq (22 tests): AGGREGATE_EXPAND_DISTINCT_AGGREGATES, AGGREGATE_EXPAND_DISTINCT_AGGREGATES_TO_JOIN - aggregate-filter.iq (14 tests): FILTER_AGGREGATE_TRANSPOSE, AGGREGATE_FILTER_TRANSPOSE, AGGREGATE_CASE_TO_FILTER, AGGREGATE_FILTER_TO_CASE, AGGREGATE_REDUCE_FUNCTIONS, AGGREGATE_MERGE, AGGREGATE_MIN_MAX_TO_LIMIT, AGGREGATE_GROUPING_SETS_TO_UNION - filter-into-join.iq (10 tests): FILTER_INTO_JOIN - filter-project-transpose.iq (9 tests): FILTER_PROJECT_TRANSPOSE, PROJECT_WINDOW_TRANSPOSE, PROJECT_TO_LOGICAL_PROJECT_AND_WINDOW, PROJECT_CORRELATE_TRANSPOSE, FILTER_TABLE_FUNCTION_TRANSPOSE - join-project-transpose.iq (19 tests): JOIN_PROJECT_RIGHT/LEFT/BOTH_TRANSPOSE_INCLUDE_OUTER, JOIN_PUSH_EXPRESSIONS, JOIN_EXPAND_OR_TO_UNION_RULE, JOIN_CONDITION_EXPAND_IS_NOT_DISTINCT_FROM, JOIN_REDUCE_EXPRESSIONS, FULL_TO_LEFT_AND_RIGHT_JOIN - join-push-transitive-predicates.iq (21 tests): JOIN_PUSH_TRANSITIVE_PREDICATES - join-sub-query-to-correlate.iq (7 tests): JOIN_SUB_QUERY_TO_CORRELATE - misc-rules.iq (10 tests): LIMIT_MERGE, MULTI_JOIN_OPTIMIZE_BUSHY, MULTI_JOIN_OPTIMIZE, PROJECT_VALUES_MERGE, PROJECT_JOIN_TRANSPOSE, FILTER_TO_CALC, PROJECT_TO_CALC, FILTER_CALC_MERGE, PROJECT_CALC_MERGE, JOIN_CONDITION_PUSH - project-aggregate-merge.iq (7 tests): PROJECT_AGGREGATE_MERGE - semi-join.iq (4 tests): SEMI_JOIN_REMOVE - set-op.iq (29 tests): INTERSECT_TO_SEMI_JOIN, INTERSECT_TO_EXISTS, MINUS_FILTER_TO_FILTER, UNION_FILTER_TO_FILTER, INTERSECT_FILTER_TO_FILTER, MINUS_TO_ANTI_JOIN, AGGREGATE_UNION_TRANSPOSE, JOIN_LEFT/RIGHT_UNION_TRANSPOSE - sort-join-transpose.iq (15 tests): SORT_JOIN_TRANSPOSE, SORT_JOIN_COPY, SORT_PROJECT_TRANSPOSE - unnest.iq (5 tests): UNNEST_DECORRELATE, UNNEST_PROJECT_DECORRELATE Modified existing files: - aggregate-join-transpose.iq: +2 tests - reduce-expressions.iq: +13 tests Tests testReduceCase, testSimplifyItemIsNotNull, testSimplifyItemIsNull are kept in RelOptRulesTest.java because they use withRelBuilderSimplify(false) or withDynamicTable() which are not supported in Quidem .iq format. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… and testPushAggregateThroughJoin3 Add config tokens to SubPlanCommand in QuidemTest, allowing !sub-plan directives to configure SQL-to-RelNode conversion and RelBuilder behavior. Tokens starting with a lowercase letter are config options; UPPERCASE tokens are CoreRules field names. Supported tokens: aggregateUnique=true, bloat=N, expand=true, relBuilderSimplify=false, simplifyValues=false, subQueryRules, trim=true. Use new tokens to migrate testReduceCase (relBuilderSimplify=false) and testPushAggregateThroughJoin3/testPushAggregateThroughOuterJoin11 (aggregateUnique=true) from RelOptRulesTest.java + XML to reduce-expressions.iq and aggregate-join-transpose.iq respectively. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…q format Use the config tokens added in the previous commit to migrate tests from RelOptRulesTest.java to new and existing .iq files: - reduce-expressions.iq: testReduceCaseNullabilityChange, testReduceCasts (relBuilderSimplify=false), testReduceConstants2 (simplifyValues=false) - aggregate.iq: testAggregateRemove2 (aggregateUnique=true) - join-push-transitive-predicates.iq: testInferringPredicatesWithNotOperator InJoinCondition (relBuilderSimplify=false) - filter-sub-query-to-correlate.iq (new): testExpandFilter* and testExpandWhereComparisonCorrelated and testWhereInJoinCorrelated (subQueryRules, relBuilderSimplify=false, trim=true tokens) - project-sub-query-to-correlate.iq (new): testExpandProject* and testSomeWithTwo* (subQueryRules, relBuilderSimplify=false, trim=true) - join-sub-query-to-correlate.iq: testExpandJoinExists, testExpandJoinScalar (subQueryRules) Also fix SubPlanCommand: trim=true must set withTrimUnusedFields on the factory config so that converter.trimUnusedFields() is not a no-op, and must also call flattenTypes then trimUnusedFields after conversion (matching AbstractSqlTester.convertSqlToRel2 behavior). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Previously, tests using sales-schema tables (e.g., sales.emp) or focusing on plan transformation had comments like: # Not using !ok: query uses the sales schema, which is not in the scott database. # Not using !ok: this test focuses on plan transformation (see !sub-plan). This commit enables !ok for all such tests that are compatible with the scott HSQLDB database, adding the expected query results. Sales-schema references (sales.emp, sales.dept, etc.) are converted to unqualified names (emp, dept, etc.) so they run against the scott catalog. Tests that remain disabled are those using non-scott tables (empnullables, deptnullables, DEPT_NESTED), non-deterministic functions (RAND, RAND_INTEGER, CURRENT_TIMESTAMP), or queries that fail in H2 due to implementation limits (very large LIMIT/OFFSET values, EXCEPT with correlated subqueries, ORDER BY null, SINGLE_VALUE aggregation with multiple rows). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…e mismatches against scott HSQLDB For tests where running SQL against scott HSQLDB causes exceptions (TINYINT overflow, type mismatches, dynamic parameters, internal AssertionErrors), use !error, !type, or fix the SQL. - aggregate-expand-distinct-aggregates.iq: SUM(deptno) overflows in some groups (Value 150 out of range) - aggregate-filter.iq: SMALLINT overflow; count(distinct cast(...)) projects as anonymous column - aggregate-join-transpose.iq, filter-sub-query-to-correlate.iq: Remove captured exception stack traces (tests now pass) - filter-project-transpose.iq: Internal AssertionError (type mismatch SMALLINT/TINYINT in windowed aggregate) - join-project-transpose.iq: Type mismatches in joins (cast x to varchar, join on deptno instead of dname for rank comparison) - project-aggregate-merge.iq: TINYINT overflow (Value 7902 out of range for cast(mgr as tinyint)) - project-join-remove.iq: slacker column not in scott database - project-join-transpose.iq: slacker column not in scott database - reduce-expressions.iq: Wrong plan (VARCHAR(20)->VARCHAR(10), EMP->DEPT) - set-op.iq: TINYINT nullability mismatch in INTERSECT - sort-join-transpose.iq: Dynamic parameters (?) cause NPE Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…em .iq format Adds lateDecorrelate=true and operatorTable=BIG_QUERY config tokens to !sub-plan, enabling migration of tests that require these configurations. - reduce-expressions.iq: testDoubleReduction, testDoubleReduction2 (POWER and division constant folding); testSplit, testSplitNull, testSplitNull1, testSplitNull2 (BigQuery SPLIT function reduction) - project-sub-query-to-correlate.iq: testDecorrelationWithProject (EXISTS in project expanded to left join, then late decorrelation) RelOptRulesTest.java and RelOptRulesTest.xml are now empty of tests. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
julianhyde
force-pushed
the
7420-rule-test
branch
from
2fc9f68 to
4672ab2
Compare
julianhyde
force-pushed
the
7420-rule-test
branch
from
c126cc3 to
dc23322
Compare
julianhyde
force-pushed
the
7420-rule-test
branch
from
fe268b9 to
16588d2
Compare
julianhyde
force-pushed
the
7420-rule-test
branch
from
16588d2 to
416f3f8
Compare
Migrate tests from RelOptRulesTest.java + XML to Quidem .iq scripts. Remove preRule= token from SubPlanCommand; use a single HepPlanner pass instead. New .iq files: limit-merge.iq, union-to-values.iq Updated .iq files: aggregate.iq, aggregate-distinct-test.iq, aggregate-expand-distinct-aggregates.iq, aggregate-join-remove.iq, filter.iq, filter-into-join.iq, filter-project-transpose.iq, reduce-expressions.iq, semi-join.iq, set-op.iq QuidemTest.java: add inSubQueryThreshold=N token; remove preRule= token. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
julianhyde
force-pushed
the
7420-rule-test
branch
from
7859d9f to
a1a5a38
Compare
Migrate 15 tests from RelOptRulesTest.java + XML to Quidem .iq scripts: - reduce-expressions.iq: testReduceNestedCaseWhen, testReduceNot - aggregate.iq: testDigestOfApproximateDistinctAggregateCall, testRemoveDistinctOnAgg - aggregate-within-distinct.iq (new file): testWithinDistinct, testWithinDistinctNoThrow, testWithinDistinctUniformDistinctKeys, testWithinDistinctUniformDistinctKeysNoThrow, testWithinDistinctCountDistinct, testWithinDistinctFilteredAggs, testWithinDistinctPreservesNonDistinctAggFilters, testWithinDistinctPreservesNonDistinctCollation, testWithinDistinctFilteredAggsUniformDistinctKeys, testWithinDistinctFilteredAggsUniformDistinctKeysNoThrow, testWithinDistinctFilteredAggsSameFilter Also adds 'throwIfNotUnique=false' config token to SubPlanCommand in QuidemTest.java, which applies AggregateExpandWithinDistinctRule.config.withThrowIfNotUnique(false) when the token precedes AGGREGATE_EXPAND_WITHIN_DISTINCT in the rule list. Also carries forward earlier-session changes: - filter-into-join.iq: testStrengthenJoinType - filter-project-transpose.iq (new): testFilterProjectTransposePreventedByCorrelation, testProjectCorrelateTransposeRuleLeftCorrelate - filter.iq: testPushFilterPastAggWithGroupingSets1/2 (with correct pre-rules PROJECT_MERGE, FILTER_PROJECT_TRANSPOSE) - project-filter-transpose.iq (new): testPushProjectPastFilter3b/3c Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
julianhyde
force-pushed
the
7420-rule-test
branch
from
5f75b0b to
2365b5e
Compare
…use on its own line For SQL fragments in rule/*.iq files where any line exceeded 80 characters, reformat the entire query: put each clause (select, from, join, where) on its own line, indent subqueries 4 spaces, and keep 'on' on the same line as 'join'. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Migrate 6 SQL-based filter tests from RelOptRulesTest.java + XML to Quidem .iq scripts: - testFilterIntoJoinMissingVariableCor → filter-into-join.iq - testExpandFilterDisjunctionForJoinInput → misc-rules.iq - testManyFiltersOnTopOfMultiJoinShouldCollapse → misc-rules.iq - testMergeFilterWithJoinCondition → misc-rules.iq - testPullConstantIntoFilter → misc-rules.iq - testSimplifyFilter → misc-rules.iq RelBuilder-based filter tests (testFilterSortTranspose*, testFilterRemoveIsNotDistinctFromRule, etc.) are not migrated. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Migrated 16 tests from RelOptRulesTest.java + XML to Quidem .iq scripts: In reduce-expressions.iq (6 tests): - testCasePushIsAlwaysWorking - testProjectOverWithAvg - testReduceConstantsWindow - testReduceConstantsWithMultipleOrderByWindow - testReduceNullableToNotNull - testReduceValuesToEmpty New prune-empty.iq (10 tests): - testEmptyAggregate - testEmptyAggregateEmptyKey - testEmptyIntersect - testEmptyMinus, testEmptyMinus2, testEmptyMinus3 - testEmptyProject, testEmptyProject2 - testEmptySort - testEmptySortLimitZero PruneEmptyRules fields are referenced as "PruneEmptyRules.INSTANCE_NAME" in !sub-plan arguments (already supported by getCoreRule()). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
julianhyde
force-pushed
the
7420-rule-test
branch
from
1c21f79 to
9a16c9e
Compare
Migrate 30 SQL-based tests from RelOptRulesTest.java + XML to Quidem .iq: prune-empty.iq (8 tests): testLeftEmptyFullJoin, testLeftEmptyInnerJoin, testLeftEmptyLeftJoin, testLeftEmptyRightJoin, testRightEmptyFullJoin, testRightEmptyInnerJoin, testRightEmptyLeftJoin, testRightEmptyRightJoin filter-sub-query-to-correlate.iq (11 tests): testExistsWithAtLeastOneRowSubQuery, testExistsWithNoRowSubQuery, testInOptimizationBothNotNull, testInWithNoRowSubQuery, testNotInNullableKey, testNotInNullableSubqueryColumn, testNotInWithNoRowSubQuery, testNotUniqueWithNoRowSubQuery, testSomeWithGreaterThanNoRowSubQuery, testSomeWithLessThanOrEqualNoRowSubQuery, testUniqueWithNoRowSubQuery union.iq (4 tests): testPullConstantThroughUnion, testPullConstantThroughUnion2, testPullConstantThroughUnion3, testPullConstantThroughUnionSameTypeNullableField reduce-expressions.iq (7 tests): testCorrelationScalarAggAndFilter, testPullConstantIntoJoin, testPullConstantIntoProject, testPullConstantIntoProjectWithIsNotDistinctFrom, testPullConstantIntoProjectWithIsNotDistinctFromForNull, testPullUpPredicatesFromProject2, testPullUpPredicatesFromUnionWithProject Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Migrate 30 tests from RelOptRulesTest.java + XML to Quidem .iq scripts: - filter-sub-query-to-correlate.iq: 20 tests using withSubQueryRules().withLateDecorrelate(true) including testAll, testAnyInProjectNonNullable, testAnyInProjectNullable, testDecorrelateExists, testDecorrelateTwoExists, testDecorrelateTwoIn, testDecorrelateUncorrelatedInAndCorrelatedExists, testSelectAnyCorrelated, testSelectNotInCorrelated, testSome, testSomeWithEquality, testSomeWithEquality2, testSomeWithNotEquality, testWhereAnyCorrelatedInSelect, testWhereExpressionInCorrelated, testWhereExpressionInCorrelated2, testWhereInCorrelated, testWhereNotInCorrelated, testWhereNotInCorrelated2, testWhereOrSubQuery - aggregate.iq: 10 AggregateMerge tests (testAggregateMerge1-10) using pre-rules AGGREGATE_PROJECT_MERGE + PROJECT_MERGE and main rules AGGREGATE_PROJECT_MERGE + AGGREGATE_MERGE Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Migrate 32 tests from RelOptRulesTest.java + XML to Quidem .iq scripts. Add topDownGeneralDecorrelate=true token to SubPlanCommand. New files: - top-down-decorrelate.iq: 20 tests for TopDownGeneralDecorrelator (CALCITE-7031, CALCITE-7397, CALCITE-7411) - sort-join-copy.iq: 3 tests for SortJoinCopyRule (CALCITE-7089) Existing files extended: - join-push-transitive-predicates.iq: 3 tests (CALCITE-2200, CALCITE-2205, CALCITE-6432) - prune-empty.iq: 3 tests (CALCITE-7071, CALCITE-7396) - aggregate.iq: 2 tests (CALCITE-5000, CALCITE-7116) - join.iq: 1 test (CALCITE-7089) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…en it detects an out-of-order line This change makes it easier for both Claude and humans to re-order lines. Lint should continue to display the file name and line number of the source (i.e. the current position of the out-of-order line).
Migrate 31 tests from RelOptRulesTest.java + XML to Quidem .iq scripts: - push-aggregate-through-union.iq (20 tests): AGGREGATE_UNION_TRANSPOSE - pull-constant-through-aggregate.iq (8 tests): AGGREGATE_PROJECT_PULL_UP_CONSTANTS - aggregate.iq (+3 tests): AggregateExtractProjectRule Add AggregateExtractProjectRule to getCoreRule() in QuidemTest.java. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Migrate 52 tests from RelOptRulesTest.java to .iq format: - 49 tests removed from Java/XML (3 new .iq entries added, 46 already in .iq files had their Java methods and XML entries removed) - New tests: testPushJoinCondDownToProject (filter-into-join.iq), testReduceDecimal (reduce-expressions.iq), testPushAggregateThroughSemiJoin (semi-join.iq) - Add date-range.iq with connectionConfig=true token for DateRangeRules - Add decorrelate.iq, measure.iq, single-values-optimization.iq - Fix testCorrelateWithoutCorrelationToEmpty in union.iq to use !ok - Add 'Not using !ok: ...' comments to 9 tests that lack !ok Also: - Fix connectionConfig=true token in QuidemTest to set CalciteConnectionConfig on the cluster's planner (via withPlannerContext) rather than only on the HepPlanner, so DateRangeRules can find it via getCluster().getPlanner() - Fix sort order of testEmptyTable in prune-empty.iq and testCorrelateWithoutCorrelationToEmpty in union.iq Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Migrate 13 tests from RelOptRulesTest.java + XML to Quidem .iq format. New tokens added to QuidemTest.SubPlanCommand: - bottomUp=true: applies rules with HepMatchOrder.BOTTOM_UP - functionsToReduce=AVG|SUM|...: creates AggregateReduceFunctionsRule with a specific set of reducible SqlKind values - withinDistinctOnly=true: creates AggregateReduceFunctionsRule with extraCondition filtering to WITHIN DISTINCT calls only New files: - sql/rule/hypergraph.iq: testHypergraph0/1/2 (using bottomUp=true) - sql/rule/aggregate-reduce-functions.iq: 8 tests for AggregateReduceFunctionsRule with various function sets Tests added to existing files: - join-project-transpose.iq: testJoinProjectTranspose1 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
The name 'sub-plan' was opaque; 'transform' better describes what the command does: convert SQL to a relational plan and optionally apply named rules. Updates all 1,458 occurrences across 43 .iq files, the command handler in CoreQuidemTest, and Javadoc in QuidemTest. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
SubPlanCommand was a 230-line nested static class inside QuidemTest with no access to instance state. Extracting it to TransformCommand.java makes it easier to navigate, and removes 18 imports from QuidemTest that were only needed by the nested class.
Extract three helpers from the monolithic execute() method: - parseArgs(String) → Config: token parsing and validation - buildRelNode(Config, SqlCommand) → RelNode: SQL-to-RelNode conversion, decorrelation, trim, and subQueryRules pre-pass - applyRules(Config, RelNode) → RelNode: HepPlanner rule application and late decorrelation Also introduce a private Config class to carry the parsed state between helpers, replacing the long list of local variables. No change to .iq file syntax. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Rule-specific config tokens are now expressed as RULE_NAME(param=value)
rather than as global flags that are implicitly tied to a specific rule.
The arg string is now split with a paren-aware tokenizer so commas inside
(...) are not treated as token separators.
Migrations:
- functionsToReduce=X → AggregateReduceFunctionsRule(functions=X)
- withinDistinctOnly=true → AggregateReduceFunctionsRule(withinDistinctOnly=true)
- throwIfNotUnique=false, ..., AGGREGATE_EXPAND_WITHIN_DISTINCT
→ ..., AGGREGATE_EXPAND_WITHIN_DISTINCT(throwIfNotUnique=false)
New splitTokens() and parseRuleToken() helpers are added to SubPlanCommand.
The three global config fields (functionsToReduceStr, withinDistinctOnly,
throwIfNotUnique) are removed from parseArgs() and Config.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…!transform Group related config tokens by the Java class they configure: - relBuilderSimplify=false, aggregateUnique=true, bloat=N, simplifyValues=false → RelBuilder(simplify=false, aggregateUnique=true, bloat=N, simplifyValues=false) - expand=true/false, decorrelate=true/false, trim=true, inSubQueryThreshold=N → Sql2Rel(expand=true, decorrelate=true, trim=true, inSubQueryThreshold=N) Migrate 13 .iq files via migrate_transform_groups.py. Remove old flat token handlers from TransformCommand.applyConfigToken. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…stExpandJoinInComposite - testDecorrelateTwoScalar: was @disabled; now active in decorrelate.iq with two-phase !transform (subQueryRules, then lateDecorrelate) - testExpandJoinIn: was @disabled; now active in join-sub-query-to-correlate.iq - testExpandJoinInComposite: moved to join-sub-query-to-correlate.iq inside !if (false) {..!} (CALCITE-1045: composite IN in join ON cannot be executed or rewritten by subQueryRules) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…DecorrelationProducesSingleValues, testIsNullPushDown, testIsNullPushDown2, testExpandJoinDisjunctionForJoinInput Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…InferredFromCorrelate, testExpressionInWindowFunction, testWindowInParenthesis Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…stSortWithDynamicParam Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…linter changes The linter added column aliases (as b, as s, as x) to SQL queries and changed SELECT * to explicit column lists, but the expected !ok output and plan column names were not updated. Update to match actual quidem output. Also reorder testSortWithDynamicParam before testWindowInParenthesis. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Migrate testCartesianProductForDphyp, testChainJoinDphypJoinReorder, testComplexPredicateForDphyp, testCycleJoinDphypJoinReorder, testFullLeftSemiJoinTypeForDphyp, testInnerLeftSemiJoinTypeForDphyp, testLeftInnerSemiJoinTypeForDphyp, testOuterJoinForDphyp, testSemiJoinForDphyp, testStarJoinDphypJoinReorder from RelOptRulesTest.java + RelOptRulesTest.xml to hypergraph.iq. Uses existing Sql2Rel(expand=true, decorrelate=true) and bottomUp=true tokens in the !transform command. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add concise AS aliases to unaliased aggregate/expression SELECT-list items in all .iq files under core/src/test/resources/sql/rule/. Aliases follow these conventions: - count(*) / count(x) → AS c (plan: C=[COUNT()]) - sum(x) → AS s (plan: S=[SUM(...)]) - max(x) → AS m (plan: M=[MAX(...)]) - min(x) → AS n (plan: N=[MIN(...)]) - avg(x) → AS a (plan: A=[AVG(...)]) - arithmetic / literal → AS x (plan: X=[...]) - boolean expression → AS b (plan: B=[...]) - window function → same as underlying aggregate Also adds missing `import static java.lang.Boolean.parseBoolean` to TransformCommand.java (was using the method without the static import). Skip cases left unchanged: EXPR$N from VALUES clauses, ORDER BY planner-added columns, and columns inside inner subquery plan nodes. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Break SQL lines longer than 80 characters and reformat multi-clause inline subqueries that exceed ~55 characters. Changes are purely cosmetic — query semantics are unchanged. Files updated: aggregate-expand-distinct-aggregates.iq, aggregate-filter.iq, decorrelate.iq, filter-project-transpose.iq, hypergraph.iq, project-sub-query-to-correlate.iq, pull-constant-through-aggregate.iq, push-aggregate-through-union.iq, set-op.iq, single-values-optimization.iq, top-down-decorrelate.iq. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add !ok to 9 tests where the justification was vague ("this test
focuses on plan transformation") but the query uses standard
scott tables and produces deterministic results:
aggregate-join-remove (tests 2, 4, 6, 8), limit-merge (test 1),
misc-rules (testWindowInParenthesis), top-down-decorrelate
(testTopDownGeneralDecorrelateForMeasure), join-push-transitive-
predicates (testJoinPushTransitivePredicatesNullabilityIssue)
- Fix 2 wrong comments in set-op.iq that said "focuses on plan
transformation" but real reason is mock-only tables
(EMPNULLABLES, DEPTNULLABLES)
- Replace 2 vague "would produce N rows of window data" comments
with the real reason: arithmetic overflow of TINYINT DEPTNO in
scott when used in a window SUM
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
HSQLDB and H2 format the decimal division result differently (6dp vs full double). Wrap the expression with ROUND(..., 3) so both CoreQuidemTest and CoreQuidemTest2 produce the same output. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See [CALCITE-7420].
This is a draft PR. If you have comments on the strategy or design please make them in the Jira case. I do not want comments on the code right now.