Skip to content

Use http_archive for Git hosted dependencies#3882

Open
jesses-canva wants to merge 3 commits intobazelbuild:mainfrom
jesses-canva:jesses/use-http-archive-for-git-repos
Open

Use http_archive for Git hosted dependencies#3882
jesses-canva wants to merge 3 commits intobazelbuild:mainfrom
jesses-canva:jesses/use-http-archive-for-git-repos

Conversation

@jesses-canva
Copy link

@jesses-canva jesses-canva commented Mar 3, 2026

We've found the use of git_repository to download Git-hosted dependencies in Cargo.toml poses some issues for reliability and performance, specifically:

  1. It doesn't allow use of the local repository cache (--repository_cache), causing excessive traffic to upstream Git servers when switching branches with differences in Git hosted dependencies. At scale this means upstream Git servers are likely to rate limit requests.
  2. It doesn't permit use of the Remote Asset API.
  3. It doesn't permit use of Bazel's downloader rewrite rules, which can be used to route downloads to a mirror or pull-through cache. This means if an upstream Git server is unavailable (eg because of rate limiting due to 1.) an in-house mirror configured using Bazel's downloader cannot be used.

For this reason we would prefer to use http_archive where possible, even for git URLs in Cargo.toml.

This PR adds a step in cargo-bazel where, for Git hosted dependencies, it first attempts to derive a HTTP URL for an equivalent .tar.gz file that can be given to http_archive rule/Http source annotation instead, and otherwise fall back to the previous git_repository/Git source annotation.

Currently the logic is only implemented for GitHub, but more entries can be added to GIT_REPO_TO_HTTP_TAR_GZ_MAPPING.

The example examples/all_crate_deps has been modified to use GitHub to download the anyhow crate, to validate that the expected http_archive referencing the .tar.gz archive appears in examples/all_crate_deps/MODULE.bazel.lock.

@jesses-canva jesses-canva changed the title Use http_archive for git dependencies with Git URLs Use http_archive for Git hosted dependencies Mar 3, 2026
@google-cla
Copy link

google-cla bot commented Mar 3, 2026

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

@jesses-canva jesses-canva force-pushed the jesses/use-http-archive-for-git-repos branch from 1cc65ef to bb17e1e Compare March 3, 2026 19:44
@jesses-canva jesses-canva force-pushed the jesses/use-http-archive-for-git-repos branch from bb17e1e to fbc1ac2 Compare March 3, 2026 19:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant