seqdb fetch lags servers

[Rytisgit]

Using the seqdb fetch command causes lags/degraded experience for players, it might be good to throttle the downloads or create another command to only download a small part of the data for running sequell locally

Another idea is to host a prefilled postgres database for people to import instead of downloading everything from each server as the main path.

[rawlins]

Can you give some more information on what you are seeing and what your exact configuration is? Do you mean lags a server on a local device, or lags a remote server? This seems very dependent on server / config details, since what sequell is pulling isn't even served by the webtiles server (usually it's apache or some kind of more involved reverse proxy / caching setup). Also, in general, I'm not sure running sequell on a play server is recommended if that's what is happening.

Also, downloading a small part of the data doesn't need another command -- just edit your sources to drop most of the files. You can also cache data locally once you've fetched it by changing the sources (this is typically what I have done for local testing of sequell or cao scoring). The sources file in the sequell repository is the production config, so may just not be what you want to use.

[Rytisgit]

What I mean, is that the sequell readme https://github.com/crawl/sequell instructs the user to run the seqdb fetch which currently downloads around 50GB of data combined from all the live servers. This is a pretty good way to lag the servers, I checked CXC monitor(http://crawl.xtahua.com:19999/#menu_system_submenu_cpu;theme) and the cpu usage went up from around 20% to 50%+ along with net outbound speeds being around 60mb/s, this of course depends on the internet connection of the computer running the seqdb fetch script. There was also a person in #DCSS saying that CAO is unplayable around the time I ran the script.

So I'm just cautioning since it is the default in instructions and anyone who might try to set up sequell is likely going to use the lag-causing command because changing anything in the config before even running it once and knowing if it works at all is a recipe for unknown errors.

[rawlins]

Thanks, did you run this script on 1/10?

[Rytisgit]

Yes, I actually ended up running it a second time on 1/11 as well due to having issues getting sequell running the first time, which is when I noticed the impact it has. Since I managed to set it up, I wont need to run it again but someone else might.