shorten time we block db connections in requests

[syphar]

Now where the new infra migration comes more and more near, I did another test of how the webserver reacts when it sees an empty archive index directory, which is what will happen after each deploy and web server container restart in the future.

It again took the server down, but now I had a look at the logs and saw error messages that requests couldn't get a database connection (or getting the connection took long).

This lead me to the actual issue with the empty archive index:

In most handlers we just fetch a database connection from the pool for the whole duration of the request. But: in typical rustdoc requests, most of the time is spend in the request to S3. And when the archive index cache is empty, the time to fetch the index comes on top (and some internal locking).

This PR updates all handlers that use storage, trying to hand back the connection earlier, so it gets freed for other requests.

So in the situation where S3 is slow / we don't have a cache, the DB pool won't be a problem any more.

Having this fixed also opens up the possibility to clean up index files when the disk is full, which is helpful for the web containers.

[rustbot]

This PR was rebased onto a different main commit. Here's a range-diff highlighting what actually changed.

Rebasing is a normal part of keeping PRs up to date, so no action is needed—this note is just to help reviewers.

[GuillaumeGomez]

Makes sense, thanks!