[BUG] [MTL] set pipeline state returned error when running RTC AEC on secondary core

[yongzhi1]

Describe the bug

With exact same test environment as described in #8022, now update topology to run AEC capture on secondary core (using core 1 as an example):

To Reproduce

1. Start spk playback:

aplay -Dhw:0,1 -c2 -r48000 -fS16_LE /home/LR.wav -vv

2. Start spk feedback capture:

arecord -Dhw:0,29 /tmp/tmp.wav -c2 -r48000 -fs32_le -vv

3. Start AEC capture:

arecord -Dhw:0,27 /tmp/mock_aec.wav -c4 -r48000 -fs32_le -vv -d20

It failed at step 3 for IPC time out:

7.445,208] <err> ipc: failed to wait schedule thread
[00:00:07.445,223] <err> ipc: ipc4: failed to send delayed reply
[00:00:07.491,650] <inf> ipc: ipc_cmd: rx	: 0x13000002|0x1
[00:00:07.491,705] <err> ipc: ipc: invalid status 3 for RESET
[00:00:07.491,716] <err> ipc: failed to process msg 19 status 7
[00:00:07.491,728] <err> ipc: ipc4: FW_GEN_MSG failed with err 7
[00:00:07.521,030] <inf> ipc: ipc_cmd: rx	: 0x46060004|0x70004
[00:00:07.521,546] <inf> ipc: ipc_cmd: rx	: 0x12040000|0x0
[00:00:07.521,595] <err> ipc: ipc_comp_free(): comp id: 262149 state is 3 cannot be freed
[00:00:07.521,608] <err> ipc: ipc_pipeline_free(): module free () failed
[00:00:07.521,620] <err> ipc: ipc4: FW_GEN_MSG failed with err 12

Reproduction Rate
100%

Expected behavior
Step 3 should work as good as Step 1 & 2.

Impact
Same results are observed with Google AEC lib instead of CONFIG_GOOGLE_RTC_AUDIO_PROCESSING_MOCK.

Environment

1. Branch name and commit hash of the 2 repositories: sof (firmware/topology) and linux (kernel driver).
   • Kernel: {[https://chromium.googlesource.com/chromiumos/third_party/kernel/+/refs/heads/chromeos-6.1}]
   • SOF: {MTL-005.0.3 release} https://github.com/thesofproject/sof/releases/tag/releases%2Fmtl%2Fv5.0.3
2. Name of the topology file
   • Topology: {sof-mtl-max98357a-rt5682-ssp2-ssp0.tplg} plus
     [TEST]: topology2: cavs-rt5682: enable AEC on secondary core #8050
3. Name of the platform(s) on which the bug is observed.
   • Platform: {MTL/Rex}

mtrace-core1.txt
dmesg-mock-aec-core1.txt

[yongzhi1]

Cherry-picked #8000 to get new mtrace for more insight:
mtrace-echo-core1.txt

The issue is not reproducible if google-rtc-aec.18.1 being removed from the connection:

cc @abonislawski @lgirdwood @RanderWang

[RanderWang]

We have observed various IPC timeout with multi-core even with normal playback, especially if we disable mtrace output.
#7774 (comment)

[yongzhi1]

The error is slightly different than #7774 though, in this case:

1. The 300ms timed out happens in ipc_wait_for_compound_msg(), driver side reported IPC4_FAILURE for SOF_IPC4_GLB_SET_PIPELINE_STATE:

[   82.420025] sof-audio-pci-intel-mtl 0000:00:1f.3: FW reported error: 6 - Unknown error while processing the request
[   82.431704] sof-audio-pci-intel-mtl 0000:00:1f.3: ipc error for msg 0x13000004|0x1

2. Only two cores are involved for the test.

[yongzhi1]

cc @serhiy-katsyuba-intel

[mengdonglin]

This is one kinds of IPC timed out tracked by #7774 with multi-core , seems to be a duplicate .

[yongzhi1]

This is one kinds of IPC timed out tracked by #7774 with multi-core , seems to be a duplicate .

If I remove google-rtc-aec.18.1 from the pipeline, then the issue is not reproducible, otherwise it's 100%, so there seems to be a very specific reason.