High memory usage / Memory leak from thumbnails with emblems - 16GB+ Memory Leak

[mindinsomnia]

Distribution

Linux Mint 21.2 Cinnamon

Package version

5.8.4

Frequency

Always

Bug description

I've identified what seems to be an issue with the way that Nemo renders emblems on file icons.

Here's some background on this. I use the Nemo Dropbox plugin that adds emblems for sync status on files, such as this one:

I've been having a lot of system hangs with Nemo lately so I've been investigating why and I think I've narrowed down the cause. It's the emblems that Nemo is rendering on icons. There seems to be a memory leak somewhere related to these.

Here's a screenshot of Nemo consuming 16.6GBs of memory. I achieved this by scrolling through a folder with photos in it, while using the Dropbox Nemo plugin to display the sync status of files, with thumbnail size was set to the largest setting.

The folder had 10,000 files in it, but I was able to achieve this level of memory usage before I even reached halfway through the files, if I had to guess, about 4,500 files.

(Please note, before anyone says, "split up the files", this issue can not be avoided by splitting up the images across multiple folders, the memory leak persists even after navigating away from folders! Nor is the issue specific to this plugin as I'll explain below.)

I left the folder, and spent around 30 minutes browsing other folders on my system, went away, did some stuff, came back... Nemo never unloaded the data in that time, no matter what else I did, as long as the process was open that data was never unloaded from memory.

Eventually I came back to the same folder full of photos and tested memory usage as I scrolled through it. My assumption was, the loaded data was still in memory, so memory usage shouldn't climb any higher. My assumption was wrong, it took only a minute or two for memory usage to skyrocket as it seemed to reload already generated thumbnails.

See above, Nemo now up to 28GBs. Again navigating away from the folder results in no reduction of memory consumption.

I wanted to narrow down the cause and see if I could eliminate the possibility that the issue here is the Nemo Dropbox plugin, so I turned that plugin off and closed Dropbox.

I then and took two images, gave them emblems using the Nemo Emblems plugin, and put them in a temporary folder in my Downloads folder, away from Dropbox (which is on another drive) - So Dropbox / The Nemo Dropbox plugin has no impact on the following results.

Then copy/pasted those couple of images many times until I had a folder full of thumbnails with emblems and tested scrolling through that as well.

See above, it didn't take long, after scrolling through the folder, going up a folder, back into it, scrolling through it again, and repeating that a few times, already I was up to a level of memory usage that would cause a system hang on my laptop.

What about when there's no emblems?
With no emblems, there's no issues, when browsing through folders of images without emblems I see normal memory usage (almost always under 1GB, even with folders with 10,000 images), and memory drops instantly when I navigate away from folders. The key factor here seems to be the emblems.

There seems to be a couple of issues here:

1. This memory usage seems very high. Even for this number of thumbnails, this seems like a very high memory usage for what is just some small emblems.
2. Nemo is not unloading something from memory. Even when I navigated to a different folder, the memory usage stayed high.
3. Nemo doesn't even re-use what is stuck in memory. It doesn't seem like Nemo is keeping track of something that is already loaded, so it's just consuming more and more space to load the same data.
4. This will cause a near irrecoverable system hang eventually. On the PC I used for testing, I have 64GBs of RAM, but on my laptop I "only" have 16GBs, and on my laptop I regularly experience system hangs from the combination of Nemo + Nemo Dropbox consuming so much memory that my laptop runs out of RAM.
5. This would be probably even worse for owners of 4K resolution monitors. Assuming everything in Nemo scales with screen resolution, owners of 4K resolution monitors should have thumbnails and emblems twice the size of users of 1080p screen owners right? I only have a 1080p screen so I can't test that myself but it seems logical.

Steps to reproduce

Grab a couple of files, put emblems on them using the Nemo Emblems plugin, put them into a folder. Then copy and paste them repeatedly until you have around 2000-3000 files.

Start scrolling through the files while monitoring RAM usage with the system monitor.

If you run out of files, just go up a folder, then come back and start scrolling again.

Repeat until you see the memory leak.

Expected behavior

I expected that once a thumbnail is unloaded from memory, any emblems associated with it would be unloaded as well.

Additional information

No response

[Jeremy7701]

It seems likely that nemo is retaining some file metadata.
In fact, if you had 10,000 thumbnails with emblems, occupying 20GB this would mean that your average thumbnail/emblem occupies 2MB.

I'd be interested to know how its possible to select an emblem other than doing tediously one file at a time - without copying a file multiple times!

BTW Do you have any swap configured, so that nemo could dump some of this wasted memory?

[mindinsomnia]

It seems likely that nemo is retaining some file metadata. In fact, if you had 10,000 thumbnails with emblems, occupying 20GB this would mean that your average thumbnail/emblem occupies 2MB.

I'd be interested to know how its possible to select an emblem other than doing tediously one file at a time - without copying a file multiple times!

BTW Do you have any swap configured, so that nemo could dump some of this wasted memory?

I do yup, just the defaults for Linux Mint, so 2GBs on my laptop and desktop according to the system monitor.

[ret2src]

I can confirm this issue on Arch Linux with Nemo 6.0.0 in a Nextcloud folder containing many images and thumbnail preview enabled. I don't know if it's related to the emblems, but there's definitely a huge memory leak somewhere. It made my whole system unresponsive today, had to force reboot because I couldn't even open a terminal emulator window or switch to another TTY.

[mindinsomnia]

Does the nextcloud folder show sync status emblems on the files?

I can confirm this issue on Arch Linux with Nemo 6.0.0 in a Nextcloud folder containing many images and thumbnail preview enabled. I don't know if it's related to the emblems, but there's definitely a huge memory leak somewhere. It made my whole system unresponsive today, had to force reboot because I couldn't even open a terminal emulator window or switch to another TTY.

[ret2src]

Does the nextcloud folder show sync status emblems on the files?

Yes, it does.

This is the memory usage with three Nemo windows open after browsing some folders with a few images:

Yesterday it ate up my whole 32 GBs (!)

[mindinsomnia]

Does the nextcloud folder show sync status emblems on the files?

Yes, it does.

This is the memory usage with three Nemo windows open after browsing some folders with a few images:

Yesterday it ate up my whole 32 GBs (!)

Yeah that does sound like the same thing I'm seeing. It seems to happen with anything that adds emblems to icons. Perhaps a temporary image is being generated to composite the thumbnails with the emblems and isn't unloaded from memory?

[mtwebster]

I honestly can't tell the difference between emblems and no emblems here, but I can reproduce a leak.

[u2n]

Did the 12/15 commit fix this issue? (Thanks BTW, @mtwebster.) LMDE 6.1.76-1 (2/1/24), Nemo 6.0.2, with same symptoms when even one emblem is used, though file count is less than 50 (much lower than reported above). Eliminating all emblems returns behavior to normal.

[mindinsomnia]

From my testing it seems the issue still persists, but this issue isn't closed yet so perhaps there's more changes planned for how to address it.

[mtwebster]

Correct, the change I did above was a safe fix for something related that I discovered while looking at the primary issue.

The real issue isn't going to be as simple to address (image caching problems). I'm hoping to have it fixed before the next release though.

[juliuspopp]

I have the same issue, 11 GB of memory usage just for nemo while browsing picture folders and having nextcloud active.
The issue does not arise when the owncloud module of nemo, SyncStateExtension_ownCloud+NemoPython, is deactivated.
When I activate it (Nextcloud is running in the background) and navigate to a folder with <100 images, the RAM usage keeps increasing to 9 GB within 20 seconds.