Skip to content

fix(call): improve virtual background performance#17450

Draft
Antreesy wants to merge 3 commits intomainfrom
fix/noid/virtual-bg-performance
Draft

fix(call): improve virtual background performance#17450
Antreesy wants to merge 3 commits intomainfrom
fix/noid/virtual-bg-performance

Conversation

@Antreesy
Copy link
Copy Markdown
Contributor

@Antreesy Antreesy commented Mar 20, 2026

☑️ Resolves

Reduce GPU load while maintaining similar blur/mask quality and responsiveness

  • After isolation tests, several items were marked as having an impact on GPU load, from low to high:
    • DEFAULT_BLUR_PASSES - Larger value - stronger blur effect. Low GPU impact.
    • SIGMA_SPACE - Larger value - smoother mask over a wider area, softer edges. High GPU impact.
    • MAX_SEGMENTATION_FRAME_RATE - Larger value - more often mask is segmented from video. High GPU impact.
    • SIGMA_COLOR - Larger value: more smoothing across edges. Low GPU impact.
    • SPARSITY_FACTOR - Larger value - faster processing, rougher mask refinement. High GPU impact.
  • All values are configurable in source code, so we need to find appropriate resulting picture, that doesn't consume much memory

As refence - Zoom allows to login -> Meet -> Host a meeting -> Check your camera with virtual BG there.

AI (if applicable)

  • The content of this PR was partly or fully generated using AI

🖌️ UI Checklist

🖼️ Screenshots / Screencasts

Tested on Windows with Intel(R) UHD Graphincs

🏚️ Before 🏡 After 🔧 Tuned Zoom
2026-03-20_18h14_52 2026-03-20_18h15_37 2026-03-20_18h16_14 image

🚧 Tasks

  • Find appropriate values that looks fine on different devices /cameras -> splitted
  • Check for possible mask freezing in _runInference with active video frame and _postProcessing
  • Uncaught (in promise) RuntimeError: divide by zero
  • Other errors

🏁 Checklist

  • 🌏 Tested with different browsers / clients:
    • Chromium (Chrome / Edge / Opera / Brave)
    • Firefox
    • Safari
    • Talk Desktop
    • Integrations with Files sidebar and other apps
    • Not risky to browser differences / client
  • 🖌️ Design was reviewed, approved or inspired by the design team
  • ⛑️ Tests are included or not possible
  • 📗 User documentation in https://github.com/nextcloud/documentation/tree/master/user_manual/talk has been updated or is not required

@Antreesy

This comment was marked as resolved.

@nemphys

This comment was marked as resolved.

@Antreesy Antreesy force-pushed the fix/noid/virtual-bg-performance branch from b75813b to 873612b Compare March 23, 2026 20:52
@Antreesy Antreesy changed the title Fix/noid/virtual bg performance fix(call): improve virtual background performance Mar 23, 2026
Antreesy added 3 commits April 2, 2026 12:52
- setting a size causes canvas 'reset'
- not needed on every frame, only if size actually changes (input constraints)
- low GPU impact

Signed-off-by: Maksim Sukharev <antreesy.web@gmail.com>
- from camera FPS to min(camera, 25) FPS
- mask lagging might be noticeable, but negligible
- 10% less GPU consumed

Signed-off-by: Maksim Sukharev <antreesy.web@gmail.com>
- add controls to tweak the values
- AI-coded, as it's up to deletion. Worth keeping for debugging purposes?

Signed-off-by: Maksim Sukharev <antreesy.web@gmail.com>
@Antreesy Antreesy force-pushed the fix/noid/virtual-bg-performance branch from 873612b to 3291f0a Compare April 2, 2026 10:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants