archive-index-caches: just flush instead of sync_all, retry on corrupted files

[syphar]

Thinking about the cache / boot problem, this is the next thing I want to try. it looks like we gain up t

Perhaps I was too pessimistic/conservative about the corrupted files we saw.

From what I see:

• the flush will flush into disc caches
• because of the rename, the download is atomic
• so the server will either see the full download, or nothing.

This will also hold true in case of the server process crashes.

Of course it won't in case of kernel crashes, but IMO that's fine for this use-case (locally cached files, the source of truth is on S3)

I added the rust-toolchain change because it lead to CI errors, somehow I think CI isn't using rust-toolchain.toml, need to check that separately.

[rustbot]

This PR was rebased onto a different main commit. Here's a range-diff highlighting what actually changed.

Rebasing is a normal part of keeping PRs up to date, so no action is needed—this note is just to help reviewers.