Remove unnecessary cuda sync for better perf

[Gasoonjia]

Stack from ghstack (oldest at bottom):

• -> Remove unnecessary cuda sync for better perf #17315

Right now we always do cudasync before existing cudabackend.execution(). However we only need that when copying data from gpu to cpu; any actions happen inside a same stream do not need explicit sync.

Differential Revision: D92193164

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/17315

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 11 New Failures, 1 Pending, 2 Unrelated Failures

As of commit d3b9ad7 with merge base 6ebaa05 ():

NEW FAILURES - The following jobs have failed:

• Lint / lintrunner / linux-job (gh)
  >>> Lint for backends/cuda/runtime/cuda_backend.cpp:
• pull / test-moshi-linux / linux-job (gh)
  RuntimeError: Command docker exec -t e9e893ef3c8cfc4ff021cd98642a28e222cebd0fa4d735cf681bb4afddb935c2 /exec failed with exit code 1
• pull / test-samsung-models-linux / linux-job (gh)
  RuntimeError: Command docker exec -t eff45eddbf4e993092f882485277e29ef7cbcd0170c3bdcdadb6ca7187894a3c /exec failed with exit code 1
• Test CUDA Builds / check-all-cuda-builds (gh)
  Process completed with exit code 1.
• Test CUDA Builds / test-executorch-cuda-build-13.0 / linux-job (gh)
  RuntimeError: Command docker exec -t 2e527d1c7ddc4b0f6341a1d0fce8ba63240f7a909d63504c3595bd2f6406a607 /exec failed with exit code 1
• Test CUDA Builds / test-model-cuda-e2e (openai, whisper-large-v3-turbo, non-quantized) / linux-job (gh)
  RuntimeError: Command docker exec -t 0bdea218f74c3b45f470f09decb6d71ba0c49542a8263fb6d938e911b553626b /exec failed with exit code 1
• Test CUDA Builds / test-model-cuda-e2e (openai, whisper-large-v3-turbo, quantized-int4-tile-packed) / linux-job (gh)
  RuntimeError: Command docker exec -t bd457f6d9df1dab964c85dbc8de40db858663339a9994eb23f6c985c7c67695f /exec failed with exit code 1
• Test CUDA Builds / test-model-cuda-e2e (openai, whisper-large-v3-turbo, quantized-int4-weight-only) / linux-job (gh)
  RuntimeError: Command docker exec -t f73cca16c3501fc8b6a97e56f012a786a5869e6271efec95eca5667cb2f87d2a /exec failed with exit code 1
• Test CUDA Builds / test-model-cuda-e2e (openai, whisper-small, non-quantized) / linux-job (gh)
  RuntimeError: Command docker exec -t aa20367c784d055477536d31aecf07b5c23926aefbb604b57afb3f5a112939c3 /exec failed with exit code 1
• Test CUDA Builds / test-model-cuda-e2e (openai, whisper-small, quantized-int4-tile-packed) / linux-job (gh)
  RuntimeError: Command docker exec -t c3d5fec761aa8a602d752eae08757cf3832e706d41776505583b59c3f2bed9e8 /exec failed with exit code 1
• Test CUDA Builds / test-model-cuda-e2e (openai, whisper-small, quantized-int4-weight-only) / linux-job (gh)
  RuntimeError: Command docker exec -t 2fcd476d172a90035fa283395e7b241abf8b5eb0b74b6938d22781d53700bd1b /exec failed with exit code 1

FLAKY - The following job failed but was likely due to flakiness present on trunk:

• Test Metal Backend / test-model-metal-e2e (openai, whisper-large-v3-turbo, non-quantized) / macos-job (gh) (similar failure)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1

BROKEN TRUNK - The following job failed but was present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• Test Metal Backend / test-model-metal-e2e (openai, whisper-small, non-quantized) / macos-job (gh) (trunk failure)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[github-actions]

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

[larryliu0820]

Just to be clear this is turning a warning into a fatal error. Just making sure this is your intention.

[Gasoonjia]

Yes that's my plan. We should guarantee that each tensor should get the correct cuda_stream

[larryliu0820]

I think it's better to set handle->cuda_stream to the only cuda stream.

[Gasoonjia]

based on offline sync we tried to create a new CudaHandle class to inherit current aoti_handle, puting a shared_ptr<cuda_stream> inside cuda_handle, to make sure there's only one cuda stream in the whole pipeline.