Disable chunk uploads for the sync rpm upload endpoint to improve performance on console

[YasenT]

Disable chunk uploads
And retrigger tests to see if it fails at same place. As I see that the nighties run just fine

[YasenT]

@mdellweg can you re-run the tests? It's failing again on a random place. I can't do it myself
And i guess we need them to pass before merging?

[jobselko]

@YasenT I just re-ran the tests. Could you please add a changelog entry, since this changes the functionality?

[mdellweg]

Yes, we should have the tests pass. But it's only necessary once we have established there's no pending change anymore. We are working on the random test failures.

[YasenT]

@YasenT I just re-ran the tests. Could you please add a changelog entry, since this changes the functionality?

Added a changelog

[YasenT]

Yes, we should have the tests pass. But it's only necessary once we have established there's no pending change anymore. We are working on the random test failures.

Anything else you would like changed? :)

[jobselko]

@YasenT Any reason why Gerrod is marked as the commit author? I am guessing this was by accident.

[mdellweg]

Before I will comment more about the excessive comments still left, let me try to understand what this change is all about. When you have a huge file, it always used to be chunked up. Now with this change it will always not be chunked up, thereby running into the exact server limits the chunking was invented for in the first place. And for that you take away the user control over the chunk size. Couldn't you instead just tell the user to increase the chunk size? Isn't this a docs issue in the first place? Do we need a globally configurable chunk_size default?

[YasenT]

@mdellweg
Actually I like having comments, I believe that they help, not sure if this changes anything in code, it's more of a personal viewpoint. But can we move forward with this? Not sure if every comment needs to be reviewed, but if it needs be, can I get the review so that we can complete these?

As for the chunking -> the default behavior actually doesn't change. It will still chunk at default 1MB, and people can set the chunk size if they want to.
There's a new synchronous endpoint in pulp_rpm, which is being called only when someone is trying to upload an RPM without providing a repository. I introduced the use of this endpoint in the cli a few months ago for this specific case. And we don't want chunking for that specific endpoint, to reduce the overhead coming from it.
And this is what the commit is targeting in fact.

[mdellweg]

My last comment wasn't even about the comments, but about the fact that the command has a chunk_size (probably with a badly chosen default) that your changes just silently start to ignore. My point is that even in the face of the upload endpoint, there is a limit to the size of file you can upload in one go.
So my specific question here is, should we make the chunk_size globally configurable?

[YasenT]

My last comment wasn't even about the comments, but about the fact that the command has a chunk_size (probably with a badly chosen default) that your changes just silently start to ignore. My point is that even in the face of the upload endpoint, there is a limit to the size of file you can upload in one go. So my specific question here is, should we make the chunk_size globally configurable?

On the chunk size, I think that the default value should be dynamic and based off the size of the file being uploaded.

But my changes aren't affecting the default behavior/flow that users are used to with the async endpoint. The changes affect just the synchronous one, which wasn't available through the pulp-cli till recently. And there the target is to reduce the tasking/load on the server side.
We increased the timeouts (on console) so that they won't be a limiting factor for the duration of an upload, which was the main factor limiting how big of a file you can upload without chunking it
And it is faster to upload 1x big file than a bunch of chunks, on tcp/ip level, as you don't force it (tcp) to go into ramp-up(slow-start) phase for each chunk. Our chunks upload sequentially, so they actually slow you down.