Add parallel API calls via furrr/future#734
Closed
maciekbanas wants to merge 22 commits intomasterfrom
Closed
Conversation
Replace sequential purrr::map calls with parallel-aware gitstats_map wrappers in all org-level and repo-level API iteration loops. When a non-sequential future plan is active (via set_parallel()), API requests run concurrently across repos/orgs. Falls back to purrr::map when no parallel plan is set, so existing behavior is unchanged by default. Changes: - Add furrr and future to Imports - Add gitstats_map, gitstats_map2, gitstats_map_chr helpers in utils.R - Add set_parallel() exported function for users - Replace purrr::map with gitstats_map in ~35 API-bound loops across GitHost, GitHostGitHub, GitHostGitLab, EngineGraphQL*, EngineRest* Co-authored-by: Ona <no-reply@ona.com>
furrr::future_map requires .progress to be a single logical, but set_progress_bar() returns a character label (e.g. 'GitHub') when progress is enabled. Use !isFALSE(.progress) to coerce: character or TRUE -> TRUE, FALSE -> FALSE. Co-authored-by: Ona <no-reply@ona.com>
furrr spawns separate R sessions that don't have access to package internals like url_decode(), show_message(), etc. Adding packages = 'GitStats' to furrr_options ensures each worker loads the package namespace. Co-authored-by: Ona <no-reply@ona.com>
When GitStats is installed, use packages='GitStats' to load it on workers. When running via devtools::load_all (package not installed), export the namespace contents as explicit globals so workers can access internal functions like url_decode, show_message, cli_icons. Co-authored-by: Ona <no-reply@ona.com>
Replace sequential purrr::map calls with parallel-aware gitstats_map wrappers in all org-level and repo-level API iteration loops. When a non-sequential future plan is active (via set_parallel()), API requests run concurrently across repos/orgs. Falls back to purrr::map when no parallel plan is set, so existing behavior is unchanged by default. Changes: - Add furrr and future to Imports - Add gitstats_map, gitstats_map2, gitstats_map_chr helpers in utils.R - Add set_parallel() exported function for users - Replace purrr::map with gitstats_map in ~35 API-bound loops across GitHost, GitHostGitHub, GitHostGitLab, EngineGraphQL*, EngineRest* Co-authored-by: Ona <no-reply@ona.com>
furrr::future_map requires .progress to be a single logical, but set_progress_bar() returns a character label (e.g. 'GitHub') when progress is enabled. Use !isFALSE(.progress) to coerce: character or TRUE -> TRUE, FALSE -> FALSE. Co-authored-by: Ona <no-reply@ona.com>
furrr spawns separate R sessions that don't have access to package internals like url_decode(), show_message(), etc. Adding packages = 'GitStats' to furrr_options ensures each worker loads the package namespace. Co-authored-by: Ona <no-reply@ona.com>
When GitStats is installed, use packages='GitStats' to load it on workers. When running via devtools::load_all (package not installed), export the namespace contents as explicit globals so workers can access internal functions like url_decode, show_message, cli_icons. Co-authored-by: Ona <no-reply@ona.com>
…rld-devs/gitstats into mb/furrr-parallel-api-calls # Conflicts: # R/EngineRestGitLab.R
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #734 +/- ##
==========================================
- Coverage 91.82% 91.37% -0.45%
==========================================
Files 26 26
Lines 4600 4628 +28
==========================================
+ Hits 4224 4229 +5
- Misses 376 399 +23 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Replace sequential purrr::map calls with parallel-aware gitstats_map wrappers in all org-level and repo-level API iteration loops. When a non-sequential future plan is active (via set_parallel()), API requests run concurrently across repos/orgs. Falls back to purrr::map when no parallel plan is set, so existing behavior is unchanged by default. Changes: - Add furrr and future to Imports - Add gitstats_map, gitstats_map2, gitstats_map_chr helpers in utils.R - Add set_parallel() exported function for users - Replace purrr::map with gitstats_map in ~35 API-bound loops across GitHost, GitHostGitHub, GitHostGitLab, EngineGraphQL*, EngineRest* Co-authored-by: Ona <no-reply@ona.com>
furrr::future_map requires .progress to be a single logical, but set_progress_bar() returns a character label (e.g. 'GitHub') when progress is enabled. Use !isFALSE(.progress) to coerce: character or TRUE -> TRUE, FALSE -> FALSE. Co-authored-by: Ona <no-reply@ona.com>
furrr spawns separate R sessions that don't have access to package internals like url_decode(), show_message(), etc. Adding packages = 'GitStats' to furrr_options ensures each worker loads the package namespace. Co-authored-by: Ona <no-reply@ona.com>
When GitStats is installed, use packages='GitStats' to load it on workers. When running via devtools::load_all (package not installed), export the namespace contents as explicit globals so workers can access internal functions like url_decode, show_message, cli_icons. Co-authored-by: Ona <no-reply@ona.com>
…rld-devs/gitstats into mb/furrr-parallel-api-calls
maciekbanas
added a commit
that referenced
this pull request
Mar 18, 2026
Add parallel API calls via mirai (alternative to #734)
Member
Author
|
Closing, as solution with |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
API calls across repos and orgs are currently sequential — each
purrr::maploop waits for one HTTP response before starting the next. For setups with many repos/orgs, this is dominated by network latency.This PR adds optional parallel execution using
furrr/future. When a user enables parallelism viaset_parallel(), API requests run concurrently across repos/orgs. When no parallel plan is set, behavior is identical to before (falls back topurrr::map).Related Issue(s)
Fixes #736
Key changes
gitstats_map(),gitstats_map2(),gitstats_map_chr()— drop-in wrappers inutils.Rthat checkfuture::plan()and dispatch to eitherpurrr::map(sequential) orfurrr::future_map(parallel)set_parallel(workers)— exported convenience function to configure parallelism. AcceptsTRUE(auto-detect cores), an integer, orFALSE/0to disableGitHost.R,GitHostGitHub.R,GitHostGitLab.R,EngineGraphQL.R,EngineGraphQLGitHub.R,EngineGraphQLGitLab.R,EngineRestGitHub.R,EngineRestGitLab.Rpurrr::map(no benefit from parallelism)furrrandfutureadded toImportsUsage
Design decisions
set_parallel()to enable.<<-in parallel: Loops using<<-for side effects (e.g.,get_orgs_from_orgs_and_reposin GitLab) are left sequential.seed = NULL: Passed tofurrr_optionssince these are deterministic API calls, not stochastic computations.get_furrr_options()detects whether GitStats is installed or loaded viadevtools::load_all()and adjusts how internal functions are exported to workers.How to test
devtools::install()(requiresfurrrandfuture)