Skip to content

[fix](fe) Fix MATCH crash on alias slots and push down as virtual column#61584

Open
airborne12 wants to merge 1 commit intoapache:masterfrom
airborne12:fix-match-predicate-virtual-column
Open

[fix](fe) Fix MATCH crash on alias slots and push down as virtual column#61584
airborne12 wants to merge 1 commit intoapache:masterfrom
airborne12:fix-match-predicate-virtual-column

Conversation

@airborne12
Copy link
Member

@airborne12 airborne12 commented Mar 20, 2026

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

When MATCH expressions reference alias slots that have lost column metadata
(e.g., CAST(variant_col['subkey'] AS VARCHAR) AS fn), and the MATCH is in
a predicate that cannot be pushed below a join (due to OR with join-dependent
conditions like EXISTS mark or LEFT JOIN null checks), ExpressionTranslator's
visitMatch() crashes with "SlotReference in Match failed to get Column".

Root cause: Alias.toSlot() only preserves originalColumn/originalTable when
its child is a direct SlotReference. When wrapped in Cast/ElementAt, all
metadata is lost. Combined with OR preventing filter pushdown, the MATCH is
stuck at the join layer referencing a metadata-less slot.

Reproducer:

WITH contacts AS (
  SELECT objectId, CAST(overflowProperties['string_8'] AS VARCHAR) AS firstName
  FROM objects_small WHERE portalId = 865815822
),
lists AS (
  SELECT objectId FROM lists_v2 WHERE portalId = 865815822
)
SELECT o.objectId
FROM contacts o LEFT JOIN lists l ON o.objectId = l.objectId
WHERE firstName MATCH_ANY 'john' OR l.objectId IS NOT NULL;
-- ERROR: SlotReference in Match failed to get Column

This PR fixes the issue with two changes:

  1. Graceful fallback in visitMatch(): When the slot has lost column/table
    metadata, fall back to invertedIndex = null instead of throwing. The BE
    evaluates MATCH correctly via slow-path expression evaluation, or the
    virtual column mechanism (below) provides fast-path index evaluation.

  2. New rewrite rule PushDownMatchPredicateAsVirtualColumn: Extracts MATCH
    from join/filter predicates, traces the alias slot back through the Project
    to find the original column expression, and creates a virtual column on
    OlapScan. The BE evaluates the virtual column via inverted index using
    fast_execute(), and the join layer references the boolean result.

Plan transformation:

Before:
  Filter(fn MATCH_ANY 'john' OR l.objectId IS NOT NULL)  ← crashes or slow path
    └── Join → Project[CAST(col) as fn] → OlapScan

After:
  Filter(__match_vc OR l.objectId IS NOT NULL)  ← boolean reference, no crash
    └── Join → Project[fn, __match_vc] → OlapScan[virtualColumns=[(CAST(col) MATCH_ANY 'john')]]
                                                    ↑ inverted index fast path

Release note

Fix MATCH expressions crashing when used with CTE aliases involving type casts
combined with EXISTS/LEFT JOIN and OR conditions. Also enables inverted index
evaluation for such MATCH expressions via virtual column pushdown.

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
      • Verified with variant subcolumn + EXISTS + OR query
      • Verified with LEFT JOIN + OR query
      • Verified explain verbose shows virtualColumn on OlapScan with MATCH expression
      • Verified correct query results match expected output
    • No need to test or manual test. Explain why:
  • Behavior changed:

    • No.
  • Does this need documentation?

    • No.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

…l column

### What problem does this PR solve?

Issue Number: close #xxx

Problem Summary:

When MATCH expressions reference alias slots that have lost column metadata
(e.g., `CAST(variant_col['subkey'] AS VARCHAR) AS fn`), and the MATCH is in
a predicate that cannot be pushed below a join (due to OR with join-dependent
conditions like EXISTS mark or LEFT JOIN null checks), ExpressionTranslator's
visitMatch() crashes with "SlotReference in Match failed to get Column".

Root cause: `Alias.toSlot()` only preserves originalColumn/originalTable when
its child is a direct SlotReference. When wrapped in Cast/ElementAt, all
metadata is lost. Combined with OR preventing filter pushdown, the MATCH is
stuck at the join layer referencing a metadata-less slot.

This PR fixes the issue with two changes:

1. **Graceful fallback in visitMatch()**: When the slot has lost column/table
   metadata, fall back to `invertedIndex = null` instead of throwing. The BE
   evaluates MATCH correctly via slow-path expression evaluation, or the
   virtual column mechanism (below) provides fast-path index evaluation.

2. **New rewrite rule PushDownMatchPredicateAsVirtualColumn**: Extracts MATCH
   from join/filter predicates, traces the alias slot back through the Project
   to find the original column expression, and creates a virtual column on
   OlapScan. The BE evaluates the virtual column via inverted index using
   fast_execute(), and the join layer references the boolean result.

### Release note

Fix MATCH expressions crashing when used with CTE aliases involving type casts
combined with EXISTS/LEFT JOIN and OR conditions. Also enables inverted index
evaluation for such MATCH expressions via virtual column pushdown.

### Check List (For Author)

- Test: Manual test
- Behavior changed: No
- Does this need documentation: No
@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@airborne12
Copy link
Member Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 26656 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit af5d65e4163b5a7421255075462c5ca690c45ff2, data reload: false

------ Round 1 ----------------------------------
orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	17635	4406	4289	4289
q2	q3	10648	765	522	522
q4	4671	360	244	244
q5	7562	1201	1029	1029
q6	176	171	153	153
q7	783	849	671	671
q8	9871	1452	1290	1290
q9	5324	4635	4670	4635
q10	6327	1926	1635	1635
q11	463	251	245	245
q12	734	584	465	465
q13	18062	2915	2141	2141
q14	226	240	226	226
q15	q16	734	735	669	669
q17	722	847	448	448
q18	5877	5406	5193	5193
q19	1308	973	617	617
q20	540	464	370	370
q21	4502	1826	1529	1529
q22	456	363	285	285
Total cold run time: 96621 ms
Total hot run time: 26656 ms

----- Round 2, with runtime_filter_mode=off -----
orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	4782	4673	4533	4533
q2	q3	3899	4322	3861	3861
q4	896	1211	786	786
q5	4043	4383	4361	4361
q6	198	186	145	145
q7	1775	1681	1518	1518
q8	2457	2752	2557	2557
q9	7693	7325	7321	7321
q10	3740	4055	3607	3607
q11	511	422	416	416
q12	492	596	444	444
q13	2647	3301	2582	2582
q14	295	302	286	286
q15	q16	750	793	724	724
q17	1159	1440	1390	1390
q18	7306	6929	6494	6494
q19	939	857	877	857
q20	2086	2119	1986	1986
q21	4151	3499	3396	3396
q22	471	439	387	387
Total cold run time: 50290 ms
Total hot run time: 47651 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 168112 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit af5d65e4163b5a7421255075462c5ca690c45ff2, data reload: false

query5	4353	619	510	510
query6	343	240	210	210
query7	4210	473	269	269
query8	347	244	224	224
query9	8720	2672	2673	2672
query10	486	394	346	346
query11	6999	5085	4883	4883
query12	189	128	126	126
query13	1281	454	343	343
query14	5775	3697	3388	3388
query14_1	2858	2861	2810	2810
query15	209	191	175	175
query16	995	486	445	445
query17	1127	740	629	629
query18	2456	463	356	356
query19	218	212	189	189
query20	136	130	124	124
query21	210	138	110	110
query22	13262	13907	15007	13907
query23	16354	15858	15550	15550
query23_1	15737	15863	15595	15595
query24	7112	1610	1201	1201
query24_1	1234	1229	1234	1229
query25	626	465	400	400
query26	1237	257	148	148
query27	2769	474	291	291
query28	4450	1831	1879	1831
query29	989	564	477	477
query30	301	216	189	189
query31	1007	962	871	871
query32	85	69	69	69
query33	539	343	313	313
query34	917	926	556	556
query35	686	720	606	606
query36	1118	1190	1110	1110
query37	144	102	83	83
query38	3002	2983	2913	2913
query39	846	828	804	804
query39_1	850	812	809	809
query40	232	149	140	140
query41	62	59	59	59
query42	264	258	258	258
query43	255	253	230	230
query44	
query45	200	192	184	184
query46	888	1015	646	646
query47	3039	2151	2068	2068
query48	315	321	226	226
query49	644	465	393	393
query50	682	284	216	216
query51	4089	3986	3994	3986
query52	260	265	257	257
query53	288	332	283	283
query54	302	267	267	267
query55	97	88	81	81
query56	324	319	313	313
query57	1921	1623	1761	1623
query58	284	263	262	262
query59	2814	2950	2735	2735
query60	341	341	329	329
query61	174	155	154	154
query62	616	582	551	551
query63	303	278	273	273
query64	5181	1291	1039	1039
query65	
query66	1469	445	346	346
query67	24237	24193	24140	24140
query68	
query69	408	320	284	284
query70	960	919	910	910
query71	336	308	302	302
query72	2764	2748	2531	2531
query73	528	554	320	320
query74	9637	9612	9358	9358
query75	2865	2715	2438	2438
query76	2279	1015	670	670
query77	363	382	316	316
query78	10907	11120	10457	10457
query79	1113	778	563	563
query80	1366	630	559	559
query81	553	259	226	226
query82	1191	151	124	124
query83	330	262	237	237
query84	247	122	104	104
query85	925	500	473	473
query86	422	328	325	325
query87	3098	3104	3004	3004
query88	3537	2658	2639	2639
query89	432	371	345	345
query90	2040	179	175	175
query91	167	167	139	139
query92	82	68	71	68
query93	898	831	494	494
query94	645	334	289	289
query95	590	340	378	340
query96	652	515	225	225
query97	2479	2490	2417	2417
query98	245	223	228	223
query99	963	1019	919	919
Total cold run time: 250928 ms
Total hot run time: 168112 ms

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants