New injections for forked elm packages, and add hot reloading

[lydell]

Summary

This adds https://github.com/lydell/elm-safe-virtual-dom to Lamdera, simplifies the internal implementation, and changes Freeze mode to use elm-watch-style hot reloading (which preserves things like scroll position) when possible during development.

Details

I’ve added hot reloading and stopping of apps directly to elm/core. Read about it here:

• Add hot reloading and make it possible to stop Elm apps elm/core#1155
• https://github.com/lydell/core/blob/hot-reload-stop/javascript-interface.md#appstop

Forks:

• https://github.com/lamdera/elm-core = https://github.com/lydell/core/tree/lamdera (master -> hot-reload-stop -> lamdera)
• https://github.com/lamdera/elm-browser = https://github.com/lydell/browser/tree/lamdera (master -> safe -> hot-reload-stop -> lamdera)
• https://github.com/lamdera/elm-virtual-dom = https://github.com/lydell/virtual-dom/tree/hot-reload-stop (master -> safe -> hot-reload-stop)
• https://github.com/lamdera/elm-html = https://github.com/lydell/html/tree/safe (master -> safe)

As you can see above, lydell/core has a lamdera branch:

• It makes it possible for Lamdera to inject some the things that differ between the four modes (NotLamdera, LamderaBackend, LamderaFrontend, LamderaLive).
• It adds an errorHandler argument to apps. (Used by the new REPL stuff.)
• It includes the lamdera/containers patch directly so we don’t need to inject it.
• lydell/core@hot-reload-stop...lamdera

lydell/browser also has a lamdera branch:

• It changes related to Browser.Navigation.Key, to make it transferable from an old app to a new app. Random note: Browser.Navigation.Key is no longer a special type and can even be sent to the backend and back to the frontend.
• lydell/browser@hot-reload-stop...lamdera

Benefits of this PR:

• Everything is much simpler.
• We get the safe virtual DOM.
• We get real hot reloading in lamdera live.

Testing

• lamdera live hot reloading.
• That things actually work in production (outside lamdera live).
• A production app upgrade.

Installing the forked packages

Here’s the plan: Start out simple and make things more complicated in the future if needed.

1. Add package replacements as submodules.
2. Use TemplateHaskell to generate:
   • A map of packages that are replaced, to the versions that the replacements have. Used for validation.
   • A map of files to be replaced, to ByteString of alternate file content. Used to do the replacements.

Validate versions

In the TemplateHaskell-created map, each elm/* package that we replace specifies a version. That’s the version that the forked package was based on. In elm.json, there will still be lines like "elm/core": "1.0.5". What if the version in elm.json and the “fork version” do not match?

First, let’s go through the scenarios where the fork version is higher than the version specified in elm.json:

• elm.json major < fork major -> 🤷 This can’t happen. Only 1.x versions exists of all elm/* packages we replace. But we could code it to be a hard error.
• elm.json minor < fork minor -> 🤷 This can’t happen. Only 1.0.x versions exists of all elm/* packages we replace. But we could code it to be a hard error.
• elm.json patch < fork patch -> ✅ This is very likely to happen. Lots of people probably have "elm/virtual-dom": "1.0.3" in their elm.json, and haven’t bothered updating to the recent 1.0.4 version. If we made this a hard error, it would be annoying for lots of people. I think it’s better to simply allow the fork patch version to be greater than specified. If we want to, we could inform about this somehow (log message, or maybe the comments in the compiled JS shown below is enough?).

Then, let’s go through the opposite scenarios, where the fork version is lower than the version specified in elm.json:

• elm.json major > fork major -> ❌ It’s very unlikely that Evan will suddenly release 2.x of some package. But if that happens, this should be a hard error.
• elm.json minor > fork minor -> ❌ It’s unlikely that Evan will suddenly release 1.1.x of some package. But if that happens, this should be a hard error.
• elm.json patch > fork patch -> 🚨 This is somewhat likely. As mentioned, elm/virtual-dom 1.0.4 was recently released. If elm/virtual-dom 1.0.5 is released with a security fix, and a user tries to put "elm/virtual-dom": "1.0.5" in their elm.json, I think it should be a hard error, informing them that you can’t go above 1.0.4 with this release of the Lamdera compiler. (Silently using 1.0.4 anyway would be misleading, leading to a false sense of security.) They need to wait for a new Lamdera compiler version that has pulled in the security fix.

⚠️ We should make sure that lamdera init uses the known versions of elm/virtual-dom etc.

In the compiled JS, it might be nice adding comments for debugging:

// Lamdera vX.Y.Z-W: Using the following sources for some packages, rather than the official ones:
// https://github.com/lamdera/elm-core/commit/123
// https://github.com/lamdera/elm-virtual-dom/commit/456
// https://github.com/lamdera/elm-html/commit/789
// https://github.com/lamdera/elm-browser/commit/321

Replacements

When reading files from disk in ELM_HOME, check if the file is in the replacement maps. If so, go with that file content instead of the actual files.

This requires two changes to caches:

• In ELM_HOME there are artifacts.dat files, which Lamdera already call artifacts.x.dat. Now, they’ll be called for example artifacts.lamdera-1.3.3-0.19.1.dat. This is because we can’t trust an artifacts file made by a previous Lamdera version. It might have had different package replacements.
• In elm-stuff/ there’s usually a 0.19.1 folder with artifacts. Now there will be elm-stuff/lamdera-1.3.3-0.19.1/ instead. This is again because cached stuff from a previous Lamdera version cannot be trusted – they might contain the wrong package replacement code. (We don’t use the same approach for ELM_HOME – ~/.elm/lamdera-1.3.3-0.19.1/ – because IDE:s expect packages to be in ~/.elm/0.19.1/.)

Pros and cons

The benefit of having hard-coded (TemplateHaskell) maps of replacements is simplicity. We know that the provided packages are going to work with Lamdera and hot reloading.

The downside is that we need to make a new compiler release if there’s a hotfix in one of the replaced elm/* packages. However, that happens so infrequently that it feels safe to try out this approach.

Hot reloading

In this PR, I tried to change as little as possible. Both in terms of code, and in terms of user experience.

The default is still to refresh the page when an Elm file changes. But in Freeze mode, we now do hot reloading instead of page refresh when possible.

I decided to change as little as possible on the Haskell side, and implemented almost everything needed for hot reloading in Live.elm and live.js.

The web socket server still just sends an r message when an Elm file has changed. When handling that message, the following happens:

1. If Freeze mode isn’t enabled, simply reload the page like before.
2. Fetch / to get the page as HTML, with the inline <script> tag.
3. Cut out the compiled Elm JS from it (and skip the live.js stuff etc) using the new // lamdera-elm-js-start and // lamdera-elm-js-end markers.
4. If those markers aren’t found, we got the compile error page. If so, display that in an iframe that covers the whole page.
5. Eval the cut out Elm JS as documented in https://github.com/lydell/core/blob/hot-reload-stop/javascript-interface.md#elmhotreload
6. Via a port, have Elm check that the backend model can still be Wire-decoded, just like we do in init. If not, refresh the page and let init reset and notify about this.
7. Bonus: Do the same for the frontend model. Previously, the frontend model could go way out of sync with the code and make the app crash until you unfreeze. In the worst case, the devbar wouldn’t even render so you couldn’t unfreeze – then you had to clear local storage manually. Now, that shouldn’t happen thanks to the Wire check.

Previous (maybe future) idea

The last commit of #29 adds two interesting things:

• A "j" WebSocket message, telling the client to hot reload. (It should only be used when the main type hasn’t changed.) It does that by adding a script tag pointing to the URL /_x/js.
• An endpoint for /_x/js.

If the endpoint returns JavaScript like this, hot reload should Just Work:

(function () {
    var scope = {};

    (function () {
        // Put the compiled Elm JavaScript code here. (Only the compiler output, no live.js or anything.)
    }).call(scope);

    Elm.hot.reload(scope);
})();

[supermario]

Tested and can't seem to find any way this breaks things.

Thank you for such a detailed and well thought out contribution, as always ❤️

[lishaduck]

Just FYI, this doesn't work on forks by default :)

[lydell]

I’m not sure I’m following – can you share more details?

[lishaduck]

Sure!

My origin is my fork. GitHub forks, by default, only include the default branch: lamdera-next, there is no origin/master (it's upstream/master).

Does that make sense?

[lydell]

But this command is run in the submodules?

[lishaduck]

Yeah that confused me too. I guess git diff is comparing vs. lamdera's master? (which wouldn't have them... so that doesn't make sense 🤔)

All I can say is that git fetch upstream master:refs/remotes/origin/master in reactor fixed it, so it's about lamdera proper...

Here's the compiler logs if that helps anything

elm> build (exe) with ghc-9.2.8
Preprocessing executable 'lamdera' for elm-0.19.1..
Building executable 'lamdera' for elm-0.19.1..
[130 of 225] Compiling Deps.Website [Elm.Package changed]
[131 of 225] Compiling Lamdera.PackageReplacements
ld: warning: -single_module is obsolete
ld: warning: ignoring duplicate libraries: '-lc++', '-lz'
fatal: ambiguous argument 'origin/master...': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'

/Users/dukese01/Developer/lamdera/extra/Lamdera/PackageReplacements.hs:21:3: error:
    • Exception when trying to run compile-time code:
        readCreateProcess: git "-C" "extra/package-replacements/elm/browser" "diff" "origin/master..." "--name-only" (exit 128): failed
      Code: (PackageReplacementsTH.loadReplacements)
    • In the untyped splice: $(PackageReplacementsTH.loadReplacements)
   |
21 |   $(PackageReplacementsTH.loadReplacements)
   |   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Error: [S-7282]
       Stack failed to execute the build plan.
       
       While executing the build plan, Stack encountered the error:
       
       [S-7011]
       While building package elm-0.19.1 (scroll up to its section to see the error) using:
       /Users/dukese01/.stack/setup-exe-cache/aarch64-osx/Cabal-simple_w2MFVN35_3.6.3.0_ghc-9.2.8 --verbose=1 --builddir=.stack-work/dist/aarch64-osx/ghc-9.2.8 build exe:lamdera --ghc-options " -fdiagnostics-color=always"
       Process exited with code: ExitFailure 1 
                                                                                                                                                                 

[lydell]

I tried the following:

git clone git@github.com:lamdera/compiler.git
cd compiler
git submodule update --init
git "-C" "extra/package-replacements/elm/browser" "diff" "origin/master..." "--name-only"

And it worked. So a fresh clone should work at least. I think.

I haven’t tested a fork, though. But I can’t understand yet how that would make anything different.

[lishaduck]

So a fresh clone should work at least.

Huh, that's weird but it works now. Not sure what's different this time but 🤷🏻‍♂️
Well, sorry for bothering ya and thanks for bearing with me! ❤️