Skip to content

more reslient downloads:#3222

Merged
ikreymer merged 1 commit intomainfrom
issue-3221-more-resilient-downloads
Mar 16, 2026
Merged

more reslient downloads:#3222
ikreymer merged 1 commit intomainfrom
issue-3221-more-resilient-downloads

Conversation

@ikreymer
Copy link
Member

@ikreymer ikreymer commented Mar 14, 2026

  • wrap content iterator from requests.get() in get_file() and track how many bytes downloaded, downloading from S3 storage here.
  • attempt to restart with range if iterator fails
  • retry unless 5 consecutive failures, wait for 2 seconds between each retry
  • fixes [Task]: Make streaming downloads more resilient #3221

Local Testing:

  • Create a collection of a decent size, ideally 200MB+ with multiple items
  • Start downloading collection
  • Immediately kill the local minio pod eg with kubectl delete pod local-minio-<pod-name> --force
  • Observe in api logs that there is a failure and the download is retried and should succeed.

- wrap content iterator from requests.get() and track how many bytes downloaded
- attempt to restart with range if iterator fails
- retry unless 5 consecutive failures, wait for 2 seconds between each retry
- fixes #3221
@ikreymer ikreymer requested a review from tw4l March 14, 2026 00:05
Copy link
Member

@tw4l tw4l left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested, nice change

@ikreymer ikreymer merged commit b3ac1b3 into main Mar 16, 2026
24 checks passed
@ikreymer ikreymer deleted the issue-3221-more-resilient-downloads branch March 16, 2026 19:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Task]: Make streaming downloads more resilient

2 participants