feat(babel): add parallel processing via worker threads #1956
Conversation
- drops support for rollup 1.x, which allows simplifying the implementation - implements 'plugin hook filters' for improved performance under rolldown - passes a new `code` argument to custom filter functions, so that consumers can add more advanced filtering logic - allows custom filter functions to be async
Add a `parallel` option that processes files concurrently using Node.js worker threads. This reduces build times for large projects where Babel transformation is a bottleneck. This is similar to the existing parallel behavior of `@rollup/plugin-terser`. This required some fairly significant refactoring, because we can only pass serializable objects between the main thread and the worker threads. It also required changes to the plugin's own build config, so that we can generate a dedicated worker entrypoint. Validations are added to ensure that unserializable config (e.g. inline babel plugins) cannot be used alongside the new parallel mode. For people using dedicated babel config files, this isn't a problem, because they are loaded directly by babel in the worker thread itself. The worker threads do have a setup cost, so this only makes sense for large projects. In Discourse, enabling this parallel mode cuts our overall vite (rolldown) build time by about 45% (from ~11s to ~6s) on my machine.
|
@Andarist @shellscape sorry for the ping, but just wanted to check if this is something you'd consider looking at? The performance benefits are pretty massive for large projects using Babel. |
|
I'm really looking forward to this landing! I have a local tarball that I've been copying into my projects to have the benefits of this PR - and it's really good. I can no longer imagine using babel in rollup/vite/rolldown without the parallel mode <3 (usage without parallel mode is painfully slow) |
|
|
||
| const parallelWorkerCount = typeof parallel === 'number' ? parallel : 4; | ||
| workerPool = new WorkerPool( | ||
| new URL('./worker.js', import.meta.url).pathname, |
There was a problem hiding this comment.
is this compatible with repo-wide supported node.js version?
There was a problem hiding this comment.
Worker Threads and import.meta.url are both supported in Node 14 👍
There was a problem hiding this comment.
have you considered using some existing worker pool libraries?
There was a problem hiding this comment.
I aimed to match the implementation of the existing Terser plugin, since I wasn't sure if adding new dependencies would be welcome. I'm certainly happy to explore it though! Did you have any particular libraries in mind?
workerpool looks like it could be a good bet 👀
3 participants
Rollup Plugin Name:
@rollup/plugin-babelThis PR contains:
Are tests included?
Breaking Changes?
If yes, then include "BREAKING CHANGES:" in the first commit message body, followed by a description of what is breaking.
List any relevant issue numbers: n/a
Description
feat(babel): add parallel processing via worker threads
Add a
paralleloption that processes files concurrently using Node.js worker threads. This reduces build times for large projects where Babel transformation is a bottleneck. This is similar to the existing parallel behavior of@rollup/plugin-terser.This required some fairly significant refactoring, because we can only pass serializable objects between the main thread and the worker threads. It also required changes to the plugin's own build config, so that we can generate a dedicated worker entrypoint.
Validations are added to ensure that unserializable config (e.g. inline babel plugins) cannot be used alongside the new parallel mode. For people using dedicated babel config files, this isn't a problem, because they are loaded directly by babel in the worker thread itself.
The worker threads do have a setup cost, so this only makes sense for large projects. In Discourse, enabling this parallel mode cuts our overall vite (rolldown) build time by about 45% (from ~11s to ~6s) on my machine.
This PR is based on #1954, which should ideally be merged first. Happy to rebase this one on
mainif the other PR isn't mergable.For testing, I've pushed a built version of this branch to https://github.com/davidtaylorhq/plugins/tree/babel-parallel-built. In pnpm, it can be installed like: