Memory leak

[enernico]

Summary

We spotted what seems to be a memory leak in the gem, when sending lots of data to InfluxDB.

Steps to reproduce

1. Leave the code running in production, sending lots of data through the gem
2. Observe that memory usage keeps increasing

This is about the code that we use to setup the gem:

module Influxdb
  CLIENT = InfluxDB2::Client.new(url, token, precision: InfluxDB2::WritePrecision::MILLISECOND)

  WRITE_OPTIONS = InfluxDB2::WriteOptions.new(
    write_type: InfluxDB2::WriteType::BATCHING,
    batch_size: 10,
    flush_interval: 5_000,

    max_retries: 25,
    retry_interval: 1_000,
    max_retry_delay: 30_000,
    max_retry_time: 300_000,
  )

  WRITE_API = CLIENT.create_write_api(write_options: WRITE_OPTIONS)

  # @param [String] bucket
  # @param [String] name
  # @param [Hash] values
  # @param [Hash] tags
  def self.write(bucket:, name:, tags: {}, values:)
    WRITE_API.write(
      org: "my-org",
      bucket:,
      data: {
        name: name,
        tags: tags,
        fields: values,
        time: Time.zone.now,
      },
    )
  end
end

Then data is sent with calls to Influxdb.write, like so:

Influxdb.write(bucket: "some-bucket", name: "some-operation", values: { duration: }, tags: { extra: "info" })

Test locally

We ran some tests with out regular rails app:

rails console

# before doing anything, our rails app consumes 250MB

(1..1_000_000).each do |i|
   puts i
   Influxdb.write(bucket: "test", name: "blah", values: {duration: rand(1000)})
end

## RAM slowly increases to 700MB RAM

Note on batch_size

Increasing batch_size to 1_000 as suggested in the defaults has a drastic impact.

Memory usage is increasing much slower with batch_size: 1_000 instead of 10, but it does keep increasing.

Expected behavior

We would expect the RAM to be released once the data is sent.

Actual behavior

RAM is never released.

Do we have to close the client regularly, or something similar, so that it flushes the memory?

Specifications

• Client Version: gem "influxdb-client", "~> 3.2"
• InfluxDB Version: InfluxDB v2.7.12
• Platform: Ubuntu 22.04 LTS