[Feature] Easier setup

[Lure5134]

Feature Description

I would appreciate if the setup would be a bit easier.
The currently recommended docker compose (https://raw.githubusercontent.com/logtide-dev/logtide/main/docker/docker-compose.yml) contains 9 services and 4 volumes.
If I am running just a simple service on a low budget device (e. g. a raspberry) and don't have that much logs it would be nice, if i can start it without to much overload.
For example if instead of redis, mongodb, clickhouse and postgres a simple sqlite could be used if nothing is configured. (The same for the some other services)
This would drastically help in a very simple setup.

Problem/Use Case

The setup seems a bit complicated and a bit overloaded for a simple service.

Priority

• Critical - Blocking my usage of LogTide
• High - Would significantly improve my workflow
• Medium - Nice to have
• Low - Minor enhancement

Target Users

• DevOps Engineers
• Developers
• Security/SIEM Users
• System Administrators
• All Users

Contribution

• I would like to work on implementing this feature

[Polliog]

Hi @Lure5134 thank you for the feedback, and also a point that i need write better inside the documentation.

A couple of things worth noting:

The default docker-compose.yml is already pretty lean. Without any --profile flags, it only starts 5 services: PostgreSQL (TimescaleDB), Redis, the backend, the worker, and the frontend. MongoDB, ClickHouse, and the two Fluent Bit containers are all opt-in via profiles and won't run unless explicitly requested.

If you want to go even lighter, there's docker-compose.simple.yml which removes Redis entirely. The backend falls back to graphile-worker (PostgreSQL-based job queue) and LISTEN/NOTIFY for live tail so you're down to just 3 containers: PostgreSQL, backend, and frontend.

docker compose -f docker-compose.simple.yml up -d

For a Raspberry Pi with low log volume this should work fine. TimescaleDB is just PostgreSQL under the hood and runs well on ARM with modest resources.

Regarding SQLite: this isn't something we're planning to add. Logtide already supports multiple storage backends via@logtide/reservoir (TimescaleDB, ClickHouse, and MongoDB), but SQLite doesn't fit the profile it lacks the time-series capabilities, concurrent write handling, and query patterns that Logtide depends on for log ingestion and aggregation. TimescaleDB on a Raspberry Pi with low volume is already a very lightweight setup.

That said, you've highlighted a real documentation gap the simple compose and the profile system aren't prominent enough. I'll make sure these are better surfaced in the docs and README. Thanks for raising it.

side note: I've just updated it because some new env were not included

[Polliog]

Closed with #183

[Lure5134]

@Polliog Thanks for you help. I am currently setting it up on my server.
I think it would help to add a separate env file for the simple docker compose because not all fields are required.
Nonetheless the simple setup includes 4 services. You missed the worker service.

I will report back, when the services are running. Thank you for all your hard work.

[Lure5134]

I don't know what i am doing wrong but the open telemetry logs aren't coming to logtide.
My setup looks like this (c#)

loggerConfiguration = loggerConfiguration.WriteTo.OpenTelemetry(options =>
        {
            options.Endpoint = "https://[HIDDEN].app";
            options.LogsEndpoint = "/v1/otlp/logs";
            options.Protocol = OtlpProtocol.HttpProtobuf;
            options.Headers = new Dictionary<string, string>
            {
                ["X-API-Key"] = "[HIDDEN]"
            };
        });

If i change the LogsEndpoint to something else logtide logs a 404. With /v1/otlp/logs nothing get's logged.
What I noticed too is that when I stop the container, delete the volume and restart it I am still logged in. Even if there are no users anymore.

[Polliog]

The /v1/otlp/logs endpoint is fully implemented and supports both protobuf and JSON. A couple of things to check:

1. URL construction. In Serilog's OpenTelemetry sink, LogsEndpoint expects a full URL, not a relative path. Try setting the full endpoint directly:

  options.Endpoint = "https://your-domain.app/v1/otlp/logs";
  // Don't set LogsEndpoint separately
  options.Protocol = OtlpProtocol.HttpProtobuf;
  options.Headers = new Dictionary<string, string>
  {
      ["X-API-Key"] = "your-api-key"
  };

You can verify the endpoint works independently:

  curl -X POST https://your-domain.app/v1/otlp/logs \
    -H "Content-Type: application/json" \
    -H "X-API-Key: your-api-key" \
    -d '{"resourceLogs":[{"resource":{"attributes":[{"key":"service.name","value":{"stringValue":"test-service"}}]},"scopeLogs":[{"logRecords":[{"timeUnixNano":"1700000000000000000","body":{"stringValue":"Hello from
  curl"},"severityNumber":9,"severityText":"INFO"}]}]}]}'

If this works but Serilog doesn't, the issue is in how the sink constructs the request.

For the session persistence issue: after wiping the volume, either clear your browser data / localStorage for the LogTide domain, or simply click "Logout". The next API call with the stale token should return 401 and redirect you to login.

[Lure5134]

EDIT:

I feel so dumb now...
If I set the LogsEndpoint property instead of the Endpoint Property the logs are coming to logtide...

Here are some logs from the worker process:

[SigmaDetection] Processing 1 logs for org 4b7b9b30-aa59-480c-bd09-51b721952744
[SigmaDetection] No matches found
[worker(worker-438c179b5cd2aa0eb2)] INFO: Completed task 968 (sigma-detection, 2.17ms) with success

What does it mean, that no matches are found? Is this an error or just a debug log?

Also when there are no logs, there is a example how to send logs (Quick Start | Send your first log with one of these examples)

The example doesn't work but throws a validation error:

{"error":"Validation error","details":{}}

[Polliog]

@Lure5134 No worries, glad you got it working! Both Endpoint and LogsEndpoint should ideally work depending on how the Serilog sink resolves the path. I'll add a clearer note in the docs about this. Also sorry for the delay, I was on vacation.

  About [SigmaDetection] No matches found:                                                                                                                                                                                       

That's just a debug log, not an error. Every incoming log goes through the Sigma detection worker, which checks it against your enabled detection rules. "No matches found" simply means none of the rules fired for that batch the log is still ingested and stored normally, you'll see it in the search/live tail. If you don't have any Sigma rules enabled (or none of them target the kind of logs you're sending), this is the expected behavior.

About the Quick Start example throwing Validation error:

Good catch, that's a real bug. The cURL snippet in the empty state shows:

  {
    "logs": [{
      "level": "info",
      "service": "my-app",
      "message": "Hello from LogTide!"
    }]
  }

But the logSchema requires a time field (ISO 8601 datetime), and the standard {logs: [...]} payload path doesn't auto-fill it (only the array-format path runs normalizeLogData). So the example fails validation as soon as you copy-paste it.

I'll go with that. Tracking it now, will push a fix shortly

[Lure5134]

That is nice to hear, I hope you had a nice vacation.

The endpoint property doesn't require a path.

If I specify https://example.com the logs are send to https://example.com/v1/logs but the path actually should be https://example.com/v1/otlp/logs. (Missing otlp)
If I specify https://example.com/v1/otlp/logs the logs are send to https://example.com/v1/otlp/logs/v1/logs.
The endpoint part is automatically added by the open telemetry sink from serilog.

To get around it the property LogsEndpoint should be set directly.

I have seen, that logtide supports open telemetry for traces too.
What is the endpoint for the traces, or is it the same? The sink has a separate endpoint property for it (TracesEndpoint).

[Polliog]

Good spot, you're right that Serilog.Sinks.OpenTelemetry auto-appends /v1/logs and /v1/traces to whatever you pass as Endpoint, which doesn't line up with LogTide's OTLP routes.

LogTide's OTLP paths are different for logs and traces:

• Logs: /v1/otlp/logs
• Traces: /v1/otlp/traces

So with Serilog.Sinks.OpenTelemetry you need to set LogsEndpoint and TracesEndpoint explicitly to the full URL (including the path), and leave Endpoint unset, otherwise the sink will either use the wrong suffix or double it up (.../v1/otlp/logs/v1/logs):

Log.Logger = new LoggerConfiguration()
    .WriteTo.OpenTelemetry(options =>
    {
        options.LogsEndpoint   = "https://your-logtide-host/v1/otlp/logs";
        options.TracesEndpoint = "https://your-logtide-host/v1/otlp/traces";
        options.Protocol = OtlpProtocol.HttpProtobuf;
        options.Headers = new Dictionary<string, string>
        {
            ["X-API-Key"] = "lp_your_key"
        };
    })
    .CreateLogger();

I've just added a section to the logtide-dotnet README documenting exactly this, with the warning about the auto-appended path, so nobody else gets bitten by it. Thanks for flagging!

(As a side note: if you only need logs, the LogTide.SDK.Serilog package ships a dedicated sink that talks to LogTide's native ingest endpoint directly, so this whole URL-construction gotcha doesn't apply. Serilog.Sinks.OpenTelemetry is the right choice if you want to export traces from the same pipeline.)

[Lure5134]

@Polliog When i try to log in php via the logtide/logtide package I get a 400 too.
Can it be the same problem with the datetime?