Memory leak (?) in iggy server and consumer offsets skip

[lukaszzborek]

Bug description

I've observed the iggy server's memory usage constantly growing.
It's not a huge increase, only 10-15 MB for every 2-3 million messages.

I checked it on multiple last edge builds

When the Iggy server hits its memory limit, it uses almost 100% of all CPUs. Usually, I can't log in to the server to restart the service, so I have to do it via the Proxmox UI.

After restarting, memory usage returns to normal. However, two things started happening:

1. cleaners did not respect the message expiration. Segments were deleted just after the segment rotation (the first run of the next cleaner).
2. consumers (pull next offset) are beginning to lose their offsets. Randomly skip random numbers of messages

I have 10 message expiry, with 128mb segment size
Offset lose is only on topic which was on during cpu drain. On new topics created after restart, everythink looks good.

Also i forgot to check memory usaged in system. I only see some chart in Proxmox UI and logs from iggy stats.

I can send local_data, which is saved after a server restart or CPU drain. I cannot attach it here because it's over 100 MB. You can write messages to the topic, and then try to pull the next one. It should skip messages sometimes.
Feel free to ask if you have any questions, or want additional data or something 😅

Deployment

Pre-built binary / release artifact

Versions

0.7.2-edge.1, hash: c151006, built at: 2026-03-10T21:13:24.620646481Z

Hardware / environment

proxmox, lxc, linux

Sample code

nuget verions: 0.7.0

var iggy = IggyClientFactory.CreateClient(new IggyClientConfigurator()
{
    BaseAddress = "localhost:8090",
    Protocol = Protocol.Tcp,
    AutoLoginSettings = new AutoLoginSettings()
    {
        Enabled = true,
        Username = "otherapp",
        Password = "xxxxxxx"
    }
});

await iggy.ConnectAsync();

var stream = Identifier.String("main");
var topic = Identifier.String("orderbook-topic");
var cg = "cg-orderbook";

await iggy.JoinConsumerGroupAsync(stream, topic, Identifier.String(cg));
var offset = await iggy.GetOffsetAsync(Consumer.Group(cg), stream, topic, 0);
ulong lastOffset = 0L;

var count = 10u;
while (true)
{
    var messages = await iggy.PollMessagesAsync(stream, topic, 0, Consumer.Group(cg), PollingStrategy.Next(), count, true);
    if (messages.Messages.Count == 0)
    {
        await Task.Delay(100);
        continue;
    }

    foreach (var message in messages.Messages)
    {
        if (message.Header.Offset - lastOffset != 1)
        {
            Console.WriteLine(
                $"Last offset: {lastOffset}, Current offset: {message.Header.Offset}, Difference: {message.Header.Offset - lastOffset}");
        }

        lastOffset = message.Header.Offset;
        //await Task.Delay(11);
    }
}

Logs

No response

Iggy server config

current_config.txt

Reproduction

No response

[atharvalade]

(offset skipping): The time-based message cleaner (delete_expired_segments_for_partition) doesn't check min_committed_offset before deleting segments, unlike delete_oldest_segments which does. So when segments expire, the cleaner just nukes them even if consumers haven't read those messages yet. Next poll with PollingStrategy::next() then jumps over the gap to the next surviving segment, which explains the random offset skips.

iggy/core/server/src/shard/system/segments.rs

Lines 128 to 133 in 261d255

	segments
	.iter()
	.enumerate()
	.filter(|(idx, seg)| *idx != last_idx && seg.is_expired(now, expiry))
	.map(|(_, seg)| seg.start_offset)
	.collect()

segments
    .iter()
    .enumerate()
    .filter(|(idx, seg)| *idx != last_idx && seg.is_expired(now, expiry))
    // ^ no min_committed_offset check here, unlike delete_oldest_segments (L294-L308)
    .map(|(_, seg)| seg.start_offset)
    .collect()

(premature deletion after restart): During bootstrap, if a segment's index file is empty (which can happen after a crash when enforce_fsync = false), start_timestamp and end_timestamp both get set to 0. Then is_expired() evaluates 0 + 10min <= now as true, so the cleaner deletes those segments on the very first run after restart, even though the messages aren't actually expired.

iggy/core/server/src/bootstrap.rs

Lines 333 to 340 in 261d255

	let (start_timestamp, end_timestamp) = if loaded_indexes.count() == 0 {
	(0, 0)
	} else {
	(
	loaded_indexes.get(0).unwrap().timestamp(),
	loaded_indexes.last().unwrap().timestamp(),
	)
	};

let (start_timestamp, end_timestamp) = if loaded_indexes.count() == 0 {
    (0, 0)  // <-- segments with empty indexes get timestamp 0, instantly "expired"
} else {
    (
        loaded_indexes.get(0).unwrap().timestamp(),
        loaded_indexes.last().unwrap().timestamp(),
    )
};

Still digging deeper, especially around the memory leak side of things, but wanted to share this early since it explains the post-restart behavior pretty clearly.

[lukaszzborek]

@atharvalade For the first one, I'm almost certain the offset skips were in single segments that grew to about 128mb

[atharvalade]

@atharvalade For the first one, I'm almost certain the offset skips were in single segments that grew to about 128mb

hmm that changes things...my initial finding was about the cleaner deleting whole segments without checking consumer offsets, which would cause skips at segment boundaries. but if the skips are within a single 128mb segment, that points to something else, maybe a race in the consumer group offset tracking or auto-commit path.

[hubcio]

hey @lukaszzborek, can you test your scenario on iggy branch skipping-offset? this should fix the skipping offset part of problem you reported.

[lukaszzborek]

After tests, I saw the same result after iggy restart.
But today, I saw OOM error in systemd logs (very low memory in vm to faster hit max ram usage). I think something in files structure goes wrong (again consumer offset skipping, removing segemnts which should not be expire) due to unexpected restart.
Before OOM there was tcp client errors which tried to reconnect but server suddenly stopped, logs before oom:

logs.txt

[lukaszzborek]

@hubcio
New scenario. Just restart the server and the offset starts jumping after server start. On fresh server everything works great. Build from skipping-offset branch.
Normal shutdown Received shutdown signal (SIGTERM/SIGINT), initiating graceful shutdown...