feat(build): added support for nix flake#70
Conversation
|
Hi @ChetanBhasin, thank you for your contribution. I see you build Quarkdown via I'm not familiar with Nix, what's the execution context of |
|
HI @iamgio! Thanks for the quick reply. It's been a few years since I've worked actively with JVM, so I'll have to look into Nix is a few things — a Linux distribution, a hermetic build system (similar to Bazel), and a package manager. In this context, we're using it primarily for package management that could be shipped to either the NixOS (Linux Distro) or any other system that uses Nix package management. Nix allows for separation of dependencies. Having Nix here gives us the opportunity to distribute it via package manager, and so Quarkdown can be installed using This PR specifically adds a bit more than just the ability to install Quarkdown using |
By default, distZip creates a zip with lib (jars) and bin (scripts). Usually executing a shadow jar or an unzipped dist zip as a user shouldn't have any difference. Quarkdown, however, bundles QMD libraries (such as Thank you for the explanation! |
|
Thanks for working on a Flake for this @ChetanBhasin. If nothing else this will make it possible for me to add Quarkdown to Polytype. You might be interested in some discussion in #69 as they apply to the Nix Flake use case as well. |
|
I was able to get the shadowJar build to work just by placing the paper.qmd file at the right place relative to it, but I had to artificially add a segment to the jar path just so that the hard coded Without the extra 'java' segment (which of course I had to account for when making a bin wrapper) the hard coded relative path didn't work. You might try that instead of the distZip dance. |
|
@alerque |
ChetanBhasin
force-pushed
the
main
branch
2 times, most recently
from
80aab8d to
413a68d
Compare
|
@iamgio I've now removed shadowjar in favor of distZip. |
I'm not "tweaking" the shadow jar, just installing the necessary resources along with it. The A more distro-packaging oriented build target that just generates (only) the right stuff in the first place would probably be the way to go in the long run. |
|
@alerque any chance you could test this flake on your end as well if you like it? I've tested on Until it gets merge, one can use |
|
@ChetanBhasin I see the version string is hardcoded, I'd rather avoid that. As you can see in build.gradle.kts, the latest version is fetched from the latest v* git tag. This is just helpful for me as I can just run a 2-command script that generates and pushes new tags to create a new release without worrying about misalignment. You could:
Then it looks good to me, I'd wait for @alerque who definitely knows more about Nix than me. I will just edit the README a bit before merging. |
|
I expected to be able to run the flake straight off the branch, but got an error like so: (Linux x86_64), nix 2.28.3) $ nix run github:ChetanBhasin/quarkdown -- --help
warning: Cannot parse Nix store ''https://nixpkgs-update.cachix.org''
error: builder for '/nix/store/hfxl7696ir73ivwl6n3w6w5cw6gydicc-quarkdown-1.4.0.drv' failed with exit code 1;
last 25 log lines:
> Exception in thread "main" java.net.UnknownHostException: services.gradle.org
> at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:572)
> at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327)
> at java.base/java.net.Socket.connect(Socket.java:633)
> at java.base/sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:304)
> at java.base/sun.net.NetworkClient.doConnect(NetworkClient.java:178)
> at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:533)
> at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:638)
> at java.base/sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:266)
> at java.base/sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:380)
> at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:193)
> at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1257)
> at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1143)
> at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:179)
> at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1702)
> at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1626)
> at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:224)
> at org.gradle.wrapper.Download.downloadInternal(Download.java:100)
> at org.gradle.wrapper.Download.download(Download.java:80)
> at org.gradle.wrapper.Install$1.call(Install.java:82)
> at org.gradle.wrapper.Install$1.call(Install.java:62)
> at org.gradle.wrapper.ExclusiveFileAccessManager.access(ExclusiveFileAccessManager.java:69)
> at org.gradle.wrapper.Install.createDist(Install.java:62)
> at org.gradle.wrapper.WrapperExecutor.execute(WrapperExecutor.java:107)
> at org.gradle.wrapper.GradleWrapperMain.main(GradleWrapperMain.java:63)
For full logs, run:
nix-store -l /nix/store/hfxl7696ir73ivwl6n3w6w5cw6gydicc-quarkdown-1.4.0.drv |
|
@alerque it's |
|
Actually, nevermind. I did make it the default package. So it's not working then. I'll try to figure out why. |
|
Also @ChetanBhasin perhaps it's the case to include npm and its |
Yes, that would be good, it is already usable that way. I've just gotten it working in the AUR recipe, given the system provides puppeteer and a working browser for it, I don't know how that will be easy or hard to setup with the way Nix provides puppeteer, but given what you have coming soon anyway in #71 it might as well go in the flake now. |
|
As of 9199fd6 this still does not build for me, it errors trying to download and setup Gradle. Looking at the Flake you already are trying to supply the Nix packages for Gradle, but somehow the build inputs don't seem to be propagating or something. |
|
I don't think downloading and setting up Gradle on your own is required. Invoking |
That's nice for non-developer end users wanting to directly compile from source, but it does not play nice with distro packaging. One is not always able to have network access at all stages of builds, and side-loading things breaks reproducibility tests, etc. It's particularly antithetical to the way Nix works. |
|
I agree with @alerque. That's kind of the point of nix is to have hermetic builds. It's only truly reproducible if the hashes actually match. |
iamgio
left a comment
iamgio
left a comment
There was a problem hiding this comment.
The dist zip should be generated by gradle distZip, not gradle quarkdown-cli:distZip. Even if cli tools don't directly declare a dependency to a module, that module may still be required for other purposes (e.g. the quarkdoc module for generating documentation).
|
What's the state now? @ChetanBhasin @alerque |
|
As of yet this flake has still not worked for me at all. @ChetanBhasin has something going on taken care of locally that isn't coded up in the Flake yet. I'll be sure and mention when it does work for me, but it shouldn't be too hard to reproduce on any system with $ nix run github:ChetanBhasin/quarkdown -- --help
warning: Cannot parse Nix store ''https://nixpkgs-update.cachix.org''
error: builder for '/nix/store/35kyb4bnb3z39skqwj3llj7clfq7aw9l-quarkdown-1.4.0.drv' failed with exit code 1;
last 25 log lines:
> To honour the JVM settings for this build a single-use Daemon process will be forked. For more on this, please refer to https://docs.gradle.org/8.14.1/userguide/gradle_daemon.html#sec:disabling_the_daemon in the Gradle documentation.
> Daemon will be stopped at the end of the build
>
> FAILURE: Build failed with an exception.
>
> * Where:
> Settings file '/build/source/settings.gradle.kts' line: 1
>
> * What went wrong:
> Plugin [id: 'org.gradle.toolchains.foojay-resolver-convention', version: '0.8.0'] was not found in any of the following sources:
>
> - Gradle Core Plugins (plugin is not in 'org.gradle' namespace)
> - Included Builds (No included builds contain this plugin)
> - Plugin Repositories (could not resolve plugin artifact 'org.gradle.toolchains.foojay-resolver-convention:org.gradle.toolchains.foojay-resolver-convention.gradle.plugin:0.8.0')
> Searched in the following repositories:
> Gradle Central Plugin Repository
>
> * Try:
> > Run with --stacktrace option to get the stack trace.
> > Run with --info or --debug option to get more log output.
> > Run with --scan to get full insights.
> > Get more help at https://help.gradle.org.
>
> BUILD FAILED in 12s
>
For full logs, run:
nix-store -l /nix/store/35kyb4bnb3z39skqwj3llj7clfq7aw9l-quarkdown-1.4.0.drv
$ nix-store -l /nix/store/35kyb4bnb3z39skqwj3llj7clfq7aw9l-quarkdown-1.4.0.drv
Running phase: unpackPhase
@nix { "action": "setPhase", "phase": "unpackPhase" }
unpacking source archive /nix/store/a09lcbr6d9zb7l4gq4vz0m0jgjp9arzr-source
source root is source
Running phase: patchPhase
@nix { "action": "setPhase", "phase": "patchPhase" }
Running phase: updateAutotoolsGnuConfigScriptsPhase
@nix { "action": "setPhase", "phase": "updateAutotoolsGnuConfigScriptsPhase" }
Running phase: configurePhase
@nix { "action": "setPhase", "phase": "configurePhase" }
Running phase: buildPhase
@nix { "action": "setPhase", "phase": "buildPhase" }
Building Quarkdown distribution with Gradle...
Welcome to Gradle 8.14.1!
Here are the highlights of this release:
- Java 24 support
- GraalVM Native Image toolchain selection
- Enhancements to test reporting
- Build Authoring improvements
For more details see https://docs.gradle.org/8.14.1/release-notes.html
To honour the JVM settings for this build a single-use Daemon process will be forked. For more on this, please refer to https://docs.gradle.org/8.14.1/userguide/gradle_daemon.html#sec:disabling_the_daemon in the Gradle documentation.
Daemon will be stopped at the end of the build
FAILURE: Build failed with an exception.
* Where:
Settings file '/build/source/settings.gradle.kts' line: 1
* What went wrong:
Plugin [id: 'org.gradle.toolchains.foojay-resolver-convention', version: '0.8.0'] was not found in any of the following sources:
- Gradle Core Plugins (plugin is not in 'org.gradle' namespace)
- Included Builds (No included builds contain this plugin)
- Plugin Repositories (could not resolve plugin artifact 'org.gradle.toolchains.foojay-resolver-convention:org.gradle.toolchains.foojay-resolver-convention.gradle.plugin:0.8.0')
Searched in the following repositories:
Gradle Central Plugin Repository
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
> Get more help at https://help.gradle.org.
BUILD FAILED in 12sA CI job should probably be setup too so changes to the project build system always get reflected in the Flake any time it requires updates. |
|
Hey guys! Sorry I haven't had much time to look into this with my day job taking most of the attention right now, but I'll try to find time for this soon and also setup CI. |
|
I sent a PR to your form Chetan that should at least get you started on a CI test for this. |
| src = pkgs.fetchFromGitHub { | ||
| owner = "iamgio"; | ||
| repo = "quarkdown"; | ||
| rev = "v${version}"; | ||
| sha256 = "sha256-9WQ3+VjlZmlDcHwSVOWD/ZGJfRCnXdKF0humGq2d8CU="; | ||
| }; |
There was a problem hiding this comment.
| src = pkgs.fetchFromGitHub { | |
| owner = "iamgio"; | |
| repo = "quarkdown"; | |
| rev = "v${version}"; | |
| sha256 = "sha256-9WQ3+VjlZmlDcHwSVOWD/ZGJfRCnXdKF0humGq2d8CU="; | |
| }; | |
| src = ./.; |
I don't think the flake should be re-fetching sources, by the time the flake is running this flake file will already have been located and the sources can be used from the same location (whether that is locally with nix run . or remotely with nix run github:namespace/repo).
| buildPhase = '' | ||
| echo "Building Quarkdown distribution with Gradle..." | ||
| # Use Nix-provided gradle instead of ./gradlew for reproducibility | ||
| gradle --no-daemon --console=plain -Dorg.gradle.java.home=${pkgs.jdk17} distZip |
There was a problem hiding this comment.
| gradle --no-daemon --console=plain -Dorg.gradle.java.home=${pkgs.jdk17} distZip | |
| gradle --no-daemon --console plain distZip |
In my testing $JAVA_HOME as previously exported is available here and should do the trick, gradle is complaining about this syntax and variable.
| buildPhase = '' | ||
| echo "Building Quarkdown distribution with Gradle..." | ||
| # Use Nix-provided gradle instead of ./gradlew for reproducibility | ||
| gradle --no-daemon --console=plain -Dorg.gradle.java.home=${pkgs.jdk17} distZip |
There was a problem hiding this comment.
This project is fetching a plugin remotely from the Gradle Central Plugin Repository (foojay-resolver-convention). This is done when gradle runs here in the buildPhase, but by default Nix builds do not have internet access from this phase at all! I don't know how you are testing this locally but you must have done something to disable sandboxing or defaulting to impure builds or something.
This plugin source needs to be downloaded and gotten ready for the build to use via an input declaration of some kind.
There was a problem hiding this comment.
I'm not even sure if that plugin is needed anymore. I will check later and remove the dependency if possible.
There was a problem hiding this comment.
It isn't just that plugin, that was just the first one I bumped into. There are more, e.g. this block:
id("org.jetbrains.dokka") version "2.0.0"
id("org.jlleitschuh.gradle.ktlint") version "12.2.0"
id("com.gradleup.shadow") version "8.3.6"
id("com.github.ben-manes.versions") version "0.52.0"We need to work out in the Nix flake how these are fetched and hashed so the flake inputs are reproducible.
@ChetanBhasin did you by chance look into gradle2nix? I've never used it but it looks like it is used for this use case.
|
I can see a possible problem though: Gradle fetches the latest version from git tags for reasons I explained in a previous comment. It might be problematic to run offline, as the result version would be |
We can have the Flake derive the version from the Git context, whether local or remote. It shouldn't have to be hard coded in the flake at all. If there is a Gradle command for outputting the version the same way you are deriving it and/or a place we can look in the built sources to extract it we can use that, otherwise some iteration of |
Two options:
The first option is supposed to be more robust. |
|
I've just added the |
|
@ChetanBhasin with 1.6.0, Quarkdown becomes package-manager-friendlier as it's not attempting root-only operations such as installing and linking npm packages at runtime. If you're still interested in the nix flake, you might find a good build example in the Homebrew formula. |
|
I have the hello-world example at least spun up and working on Polytype via local testing, but the PR for it that will update the website via CI is waiting for a working Flake. |
|
@ChetanBhasin if you're not interested in this PR anymore and nobody else is willing to get in charge, I'm going to close it in 3 days. |
3 participants
I don't know if nix installation is desired, but it was very helpful. I would like to leave this here so the maintainers can decide if they would like to have it.