Large base64 string - Out of Memory

[c-emin]

If the image file has between 21,039,264 and 45,805,076 bytes of data.. It throws an Out of Memory Error.
when writing to the DB using

private IMagicQuery<ImagesTable>? ImagesTableQ{ get; set; }

ImagesTableQ= await _MagicDb.Query<ImagesTable>();

await ImagesTableQ.AddAsync(img);

Is there a limit that can be stored in a single field?

[yueyinqiu]

I suspect this limit isn't fixed. It might be related to the interaction between the CSharp and the JavaScript layer, and even be related to the browser. Analyzing the source of this limitation isn't straightforward. I think there are at least three of below will exist at the same time:

1. original data in csharp
2. serializated data in csharp
3. serializated data in js
4. de-serialized data in js

[magiccodingman]

Usually Blazor WebAssembly the default heap memory cap is ~250 MB. You can increase it to 2 GB I believe though you'll run into issues going post 250 MB on safari.

But for example, I have a Blazor WebAssembly app in production that I just checked and it tends to hover ~200 to 228 MB for the heap size. So, if I moved in a file that required over ~30 MB through that app, it'd blow up with an out of memory error all the same.

Chromium/Firefox browsers will very happily provide you ~2 GB of memory, which you can increase.

But as for Magic IndexedDB, this also does bring about the question if the way we transfer from Blazor to JS layer is as efficient as it can be. Because in theory, there's ways that a 30 MB file could become 100+ MB temporarily due to how it works when transferring.

Which on another production Blazor project of mine, it's much leaner and hovers ~126 MB for the heap. So that'd mean a normal 30-50 MB item moving through wouldn't hurt me unlessss.... Unless the heap exploded temporarily from the transfer process.

But as @yueyinqiu brought up as well with how the process works. That's 100% correct in every aspect. Which is how a 30 MB file would become 100 MB temporarily but it doesn't matter if it's a fraction of a section or 10 minutes, it's all the same for an explosion.

---

I'm in the middle of both a lot of work, but I also have my wedding in about a month from now. So I've been incredibly busy, but I do think this is a great topic to look into. My main suggestion @c-emin is to bump up your wasm max heap size to roughly 2 GB to completely eliminate this process for now as long as safari won't be a concern. But when I'm not running around frantically doing wedding stuff lol, I am going to come back to this and properly track what Magic IndexedDB's heap usage is, what can or can't be improved, and document this as well.

But I remember building a lot of really lean code, so if I did it properly, the project at most should be doubling the size of a file temporarily in heap. And as far as I'm aware, a 2X peak during a handoff is as perfect as it can get with ease. So maximum performance in the current system would be 2X during peak handoff.

But the code could be engineered to grab a chunk of the data, send a piece, then delete the old piece, then send the next piece, and so on. Now this would have major risks as any issue during the transfer would equate to unrecoverable loss since we were deleting chunks as we sent to maximize performance. Though in theory we could achieve a 1.2X to 1.5X memory heap jump during transfer. But that'd be hard to pull off without letting the con's from outweighing the pro's.

---

I remember running into this issue due to Excel sheets getting too large in my app. It wasn't fun, but my team basically told everyone, "we're bumping this up to 2 GB, use chrome, if you use Safari, we'll tell you to kick rocks." hahaha. But also, now that you're aware of heap size, always clean up your items instantly. If you have an image in C# or even a list of them, then you add to IndexedDB, be sure to clean it out of Blazor ASAP to drop that heap size if it didn't already explode.

[magiccodingman]

@c-emin as a future reference/note. This library is actually 2 separate projects as 1. The JS layer that I call the universal layer is truly universal. I've been building a JS library wrapper around it, just as what you're using is the Blazor C# wrapper around the universal layer. Any language can utilize Magic IndexedDB with a wrapper.

Very large file support like your case would not have heap issues if you utilized the raw JS version for example. You could plug into the raw JS layer right now as well if you wanted, meaning you'd need to make sure the image goes and stays on the JS layer the entire time. But that interop layer can be a real pain.

This link here if you're feeling adventurous:
https://sayou.biz/Magic-IndexedDB-How-To-Connect-To-Universal-Translation-Layer

That's how you can connect to the universal layer and work with Magic IndexedDB in a much more raw format if you wanted to bypass the interop limitations. Not saying that's what you want to do (nor is it easy), but was just throwing an option at you.

[c-emin]

@yueyinqiu @magiccodingman Thank you for the detailed response!!

I did a little more testing that might help determine what is going on....

Before running the process that encounters the "Out of Memory" error
1022 MB Total JS heap size

After running the process that encounters the "Out of Memory" error
2496 MB Total JS heap size

I suspect that once the "Out of Memory" error has occured, It does not release/dispose the objects from Memory properly.
The memory remains high even after the process is complete.
I am utilizing "Using" statements in my C# code.

Could it be something in the "interaction between the CSharp and the JavaScript layer" ?

I will investigate if the raw JS method would work in my situation.

[magiccodingman]

@c-emin That test you did going from 1022 MB to 2496 MB. Just curious but was a digit added to that accidentally or is it correct that you're going from ~1 GB to 2.5 GB heap? And what size file were you doing for that test?

For example, a simple example would be your heap is at 100 MB and you're working with a 50 MB file. Now combined you're at 150 MB, but during the transfer/serialization process you have another version of it being copied and so you've now got another 50 MB in memory, plus it will double temporarily during the transfer process. Therefore, you could in theory hit 250+ MB in memory during a short burst.

So I'd be curious what your file size was as it'd be insightful to see the efficiency of Magic IndexedDB.

---

Also, while sleeping on the issue. This morning while sipping coffee, I thought of a way to potentially improve this significantly within Magic IndexedDB. In my previous comment I brought up a potential chunking strategy. Plus I looked into IndexedDB's API capabilities for streaming, which is possible.

Therefore, there could be a feature added to Magic IndexedDB whether through automated detection (potentially) or a separate method you would call for large files. And effectively, large files could be truly streamed to IndexedDB.

---

I think I'm going to label this out of memory issue as a feature request. There's a few things I'd have to consider like how it'd properly stream back out, but this library handles streaming really really well due to the custom interop level, so it shouldn't be too much of an issue. But, this also begs the question of how the feature should work.

Should the feature stream from file upload to IndexedDB? That way there's never any instance of needing to load the item in memory ever? Do you ever need to load such large files in memory or is it just for storage and future downloads? If you're rendering it later after storage, then can the system be made to handle both full streaming mode and a hybrid streaming mode? Aka, a way to stream straight to indexedDB without ever putting it in heap plus a way to also allow it in heap to render a file/image?

I'm totally open to suggestions or ideas from anyone on this @yueyinqiu if you have any. And if you agree, I do think this is a good ticket to classify as a feature.

---

Oh and just a note @c-emin I am not saying this will relate or help you at all. You obviously have reasons for such large files. But, I was on a quest of my own recently looking into client side libraries/code that allows extreme levels of image compression without human eye noticeable quality loss. There's tons of sites that help with such things or software, but most are SSR and have limits or cost money.

But I ran into Squoosh.app which is a client side WebAssembly showcase of Google's well known image compression library, but all packed into the front end and fully client side. You'd have to reconfigure it yourself rom the Squoosh GitHub, but I took a liking to the library.

Just thought I'd mention it if image compression was ever a subject of interest for you. I was literally researching this last week which is why it was on my mind and I thought I'd share.

[c-emin]

@magiccodingman

I took 3 snapshots

First Snapshot - Loaded without thumbnails and without trying to download and store large image
2nd Snapshot - Loaded with thumbnails only
3rd Snapthot - Loaded with thumbnails, trying to download and store large image

The Byte data I am trying to store is a base64 string whose .Count() returns 45,889,652

I am already using stream for the image download. So using stream straight into IndexedDB might be the best approach.

The reason for storing large images client side, caching them locally for several reasons... speed, reduce network traffic, still be able to load if network is offline.

SystemTextJson.JsonSerializer.Deserialize<Json_Class_Image>(stream).payload

I'll take a look at Squoosh.

[magiccodingman]

@c-emin Thanks for the images and detailed response. This helps a lot, but it also has me scratching my head a lot. Usually base 64 versions of an image are ~1.3X larger than the original image from my experience. But in your first snapshot your heap is already 537 MB, and if in the 2nd snapshot you have the thumbnails only loaded without Magic IndexedDB, you're still sitting at 972 MB (which is massive). But if you have an image in memory that's ~50 MB, then you turn it to a base 64 version, so there's another 50 MB, then you can have it duplicate twice again in theory, that would mean ~200+ MB in theory could be put into memory during the transfer.

But in the 3rd image you skyrocket to ~2.3 GB. Which is crazy outside of what could happen in Magic IndexedDB. So that'd point to a really bad inefficiency in the project potentially, but then we'd be seeing significantly more explosions on way lower levels if that was the case.

You said you're utilizing the using statement, but that doesn't always dispose from lists, memory, or interop. After a manual addition, are you manually nullifying anywhere else that the images could still be in memory? Because this looks to me there's something not being disposed on the Blazor side.

From these numbers, this actually shows that in image 2, you have ~500 MB in memory when all images are loaded. And the ~3X size you'd get during a transfer process does indeed roughly match up to the 3rd image's 2.3 GB size during the process.

Because I don't think a single 50 MB file will increase your total heap by a whopping 1.3 GB as that'd require this project to take a file and increase the size by 26X the size. I don't think it's that inefficient.

I've got to head out for the day, but I'd check your debugging to validate your using statement and any other tactic is in fact removing images you're done with from memory. Very interesting though!

[c-emin]

@magiccodingman

I stripped down more code and isolated it to this section.
Before running this .. Memory was: 157 MB JS Heap Size

try
{
    var membefore = await JsRuntime.InvokeAsync<string>("MemoryUsage");
    MD.Log("memory before: " + membefore);
    
    ImagesTable lt = new ImagesTable();

    lt.Id = item.Id;
    lt.ImageName = payload.ImageName;
    lt.ImageBytes = @"data:image/png;base64," + payload.ImageBytes;
    lt.ImageLabel = item.Label;
    lt.createdOn = item.CreatedOn;
    lt.modifiedOn = item.ModifiedOn;
    lt.sortOrder = item.SortOrder;
    
     MD.Log("WRITE IT");


    await ImagesTableQ.AddAsync(lt);    //    <----------    This is where the error is thrown
    
     MD.Log("WRITE IT Finished");    // Never gets here!

    lt = null;   

    var memafter = await JsRuntime.InvokeAsync<string>("MemoryUsage");
    MD.Log("memory after: " + memafter);

    ltk = null;
    payload = null;

}
catch (Exception e3)
{
    MD.Log("DB WRITE ERROR: " + e3.Message + "  " + payload.ImageName + " [" + payload.ImageBytes.Count() + "]");
    MD.Log("DB ImageBytes: " + payload.ImageBytes.Count());
}

/*==============================================================================*/
function MemoryUsage() {
    /*==============================================================================*/
    var heap = performance.memory.jsHeapSizeLimit; 
    var used = performance.memory.usedJSHeapSize;
    return used+ " / "+heap;
}

After the Exception Memory is:
2,174 MB JS Heap Size

Console Output:
(0.002 secs)memory before: 494,917,639 / 4,294,705,152 (44.057 totsecs)
(0.054 secs)WRITE IT (45.407 totsecs)
blazor.webassembly.js:1 (59.48 secs)DB WRITE ERROR: Out of memory 05F4FFEB-C044-4C02-EBB9-08DDC7E70FA5 [45889652] (103.537 totsecs)
blazor.webassembly.js:1 (5.48 secs)DB letterBytes: 45,889,652 (109.018 totsecs)

[yueyinqiu]

Hmm... I'm not sure. I currently can't think of a way to bypass memory and pass the stream directly to IndexedDB.

And about the issue itself, a 45,889,652-long string is about 100M, so I don't see how it got to 1000M+. I think there might be some other problems here. Perhaps we need to reproduce it and have a more comprehensive test.

And as you mentioned chunking, perhaps we can could use JsonSerializer.SerializeAsync(Stream, ...) and System.IO.Pipelines, rather than an extra MemoryStream or temporary strings. This could potentially save us a copy in csharp level. But I'm not sure what will happen in the JavaScript layer if an exception occurs during the write process. Need to check it in advance.

---

I have checked System.IO.Pipelines... Looks like the pipe reader knows nothing about the writer... I'm afraid this will cause problmes... (I mean the writer may partially written something but finally failed, and the reader won't know that, which just save the partially written data into indexeddb. This won't be what we want... Hmm... since json requires symmetrical brackets, perhaps the reader will not successfully deserialize it, and therefore will not save it incorrectly?)

[c-emin]

I was able to reproduce it here using the TestWasm Project by adding these two changes.

Memory Before

Memory After

[yueyinqiu]

My laptop can't debug with such large data... I tried to reproduced it with TestServer. I changed Zack's name to "Zack" + new string('A', 10000000). It should be about 20MB.

When it's paused before await personQuery.AddRangeAsync(persons);, the memory usage is about 10MB (browser) + 100MB (server). And after that it's about 50MB + 300MB. (I haven't find a way to monitor the browser memory usage in real time, so it may not be the peak value. It always switches me to other interfaces and I can't see the situation in time. And 300MB is the peak value provided by Visual Studio.) (Visual Studio also shows that GC occurs in the server part, but sometimes the memory does not decrease. This seems to be another issue.)

In contrast, if Zack's name is not changed, it will be 10MB + 100MB to 10MB + 100MB, no change basically. This means that the previous delta 40MB + 200MB (from 10+100 to 50+300) is indeed occupied due to the string ('A', 10000000). By dividing 20MB, we could infer that there are 2 copies in JavaScript and ... 10 copies in CSharp (including the origin data)? Also note that these 15 copies should be in different formats. Identical strings will share the same memory under the hood.

I also tried new string('A', 20000000). It became 50MB (The peak value was 100MB. I observed this peak because the run took a long time.) + 600MB. I'm not sure about the results of the JavaScript layer, but anyway, the CSharp layer does keep about 10 copies. This is incredible.

(My statement of "10 copies" may be misleading. According to the above results, not only does it grow as the string itself grows, but it grows faster. This means that there might be just one copy, but the length of the copy is not linearly related to the length of the original value ($n$), for example it may be $n^{2}$. It just happens at 20MB that the difference is about 10 times.)

[magiccodingman]

@c-emin @yueyinqiu I'm in the midst still of running around and now dealing with a PC that decided to give out on me lol. But as I'm getting my setup fixed up, @Ard2025 is currently on top of this and heading the diagnosis. This seems to be a memory leak of some kind.

The numbers lined up with the serialization and serialization process originally, but this may not be the issue at all. Ard will likely have more to follow up with unless I can fix things and catch up. But from what I was shown. This smells now like a memory leak on the JS layer.

Reason being is there's potential evidence found by Ard that there's tests with != vs == that do or do not have a memory leak. Which heavily hints there's an issue on the JS layer itself as it's hard to imagine syntax statements would have any kind of impact on the serialization process.

But this is now going to be treated as a top priority bug as it's seemingly a potential memory leak.

Just a note, your showing of the code as well really helped @c-emin Thank you. Hope I never come across as pushy that there is or isn't an issue with the library. I never take offense to anything. It just smelled so strongly like a serializer/deserializer issue, but you both have done great work pointing out there's an issue.