rustdoc: use webpack to bundle code

[ishitatsuyuki]

webpack is the most popular script bundler for JavaScript and other assets.

With webpack, we can benefit from:

• Transpilation of ES6 or TypeScript
• Code splitting/dynamic imports
• Effortless code minification
• Eliminating the need of resource-suffix with chunk hash

However, this adds the requirement of Node.js + NPM to the build dependencies. We need to care about:

• How to execute webpack
• How to bundle (vendor) node_modules

Below are some implementation notes:

• In rustdoc's build scripts we run webpack and then generate the code for including into the binary. Example:

resources.insert("bundle.ef0d81c.js", include_bytes!(concat!(env!("OUT_DIR"), "bundle.ef0d81c.js")));

• https://github.com/danethurber/webpack-manifest-plugin can be used for determining the file name.

[est31]

related thread for mdbook rust-lang/mdBook#610

[QuietMisdreavus]

cc @rust-lang/rustdoc for changing how we package JavaScript, and @rust-lang/infra and @rust-lang/release for the issue of wanting to add a new build dependency.

As for resource-suffix, that was added so that docs.rs could have a per-release suffix it could use to aggregate rustdoc's static files into a global cache. If you want to remove --resource-suffix, you'll need to work the new build hash into there.

[BatmanAoD]

I'm not sure I understand; is this just a request to add a recommendation to the Rust documentation to use webpack to bundle and deploy WASM code? Or are we talking about changing the behavior of the wasm32-unknown-unknown target, and adding Node.js and NPM as dependencies for using that target?

[QuietMisdreavus]

If i'm reading the request right, it's asking us to bundle our js/css using webpack instead of just packing/minifying the files by hand. However, it seems that to do that properly, we'll need to include nodejs and some npm dependencies to do the packaging, which means that those now become full build dependencies for rustdoc, rather than just dependencies for some of its tests.

[BatmanAoD]

@QuietMisdreavus Oh, I think I understand now; I thought this was changing documentation, but it's about changing the rustdoc tool itself.

[XAMPPRocky]

I wonder if instead of webpack, because webpack is really a lot to manage and maintain even in the simplest of projects, we could use Parcel. This would still add NPM & Node, but we wouldn't have to really worry about build configuration.

Parcel would allow us to use some more advanced stuff in web development such as TypeScript, ES6, Sass, without adding the maintenance of cost of those dependencies. It even has builtin support for Rust to WASM.

[GuillaumeGomez]

I'm against adding any non rust dependency, especially NPM and Node.js.

[GuillaumeGomez]

PS: I'm REALLY against adding JS into the Rust build process. It's very annoying for any rust contributor to add such a useless burden. We spent a very huge amount of time trying to get rid of any non-rust part because it was too difficult to maintain (and fix in case of bugs) and I clearly don't see the gain in here. It'd be awesome to have better tools in order to have a better JS/CSS output, and I think we should work on rust tools.

[ishitatsuyuki]

I'm REALLY against adding JS into the Rust build process. It's very annoying for any rust contributor to add such a useless burden.

I guess you're trying to express the hate for JavaScript, but your words sound exaggerating. JavaScript is not useless, and there are many people that are familiar with NPM. I agree with you that installing Node.js for first comers requires some effort, but it's no harder compared than Rust.

We spent a very huge amount of time trying to get rid of any non-rust part because it was too difficult to maintain (and fix in case of bugs) and I clearly don't see the gain in here.

Certainly, NPM packages are not as well made as Rust partly due to its dynamically typed nature. However, the popular ones are certainly useful for the right job.

You also didn't give any concrete examples regarding maintenance being hard, but in some cases you've got no alternatives. LLVM is the case, where you need C++ compiler and some additional build tools, and that's not getting replaced in near future.

It'd be awesome to have better tools in order to have a better JS/CSS output, and I think we should work on rust tools.

What you are saying is basically "Rewrite it in Rust", which is not the first thing we should do. Tools in the JavaScript ecosystem gets a lot of commits, and achieving the same level in Rust isn't a easy thing.

[GuillaumeGomez]

I'm REALLY against adding JS into the Rust build process. It's very annoying for any rust contributor to add such a useless burden.
I guess you're trying to express the hate for JavaScript, but your words sound exaggerating. JavaScript is not useless, and there are many people that are familiar with NPM.

I have wrote way too much JS for rustdoc to be put in JS-hater case I think. ;)

I agree with you that installing Node.js for first comers requires some effort, but it's no harder compared than Rust.

Clearly, you underestimate how fast JS world is moving. A good example is bower. It got deprecated whereas some time ago it was THE package manager to use. If we can avoid having to go through last JS trends and then moving to the next one, I'd be very happy.

Certainly, NPM packages are not as well made as Rust partly due to its dynamically typed nature. However, the popular ones are certainly useful for the right job.

They are... until they get deprecated because a new more trending one got out.

You also didn't give any concrete examples regarding maintenance being hard, but in some cases you've got no alternatives. LLVM is the case, where you need C++ compiler and some additional build tools, and that's not getting replaced in near future.

Just in case of rustdoc, we had the hoedown case. I made a (few? don't remember...) PR to fix some bugs and it never got merged. Also, maintaining C code into rust was very problematic (and still is).

What you are saying is basically "Rewrite it in Rust", which is not the first thing we should do. Tools in the JavaScript ecosystem gets a lot of commits, and achieving the same level in Rust isn't a easy thing.

And so is any big language, I don't see any proposal for Java tools for example.

In here, my point is: the impact on Rust would be huge for not much gain.