GO Telemetry Server

A high-performance telemetry data ingestion server written in Go that supports multiple protocols for collecting metrics, logs, and traces from distributed systems.

✨ Features

• Multi-Protocol Support

  • TCP (raw and custom binary protocols)

  • UDP (for high-volume metric collection)

  • HTTP/HTTPS (REST API and JSON payloads)

  • WebSocket (real-time telemetry streaming)

• High Performance

  • Async processing with worker pools

  • Connection pooling and reuse

  • Zero-copy parsing where possible

  • Configurable rate limiting

• Data Processing

  • Schema validation and data normalization

  • Real-time aggregation

  • Plugin-based processing pipeline

  • Support for custom data formats

• Monitoring & Observability

  • Built-in Prometheus metrics

  • Structured logging with slog

  • Health checks and circuit breakers

  • Deployment Ready

  • Docker & Kubernetes manifests

  • Graceful shutdown

  • Configuration via environment variables

  • Health check endpoints

🚀 Quick Start

Prerequisites

• Go 1.25 or higher

• Docker (optional)

Installation

# Clone the repository
git clone https://github.com/Uter88/telemetry-server-go.git
cd telemetry-server-go

# Install dependencies
go mod download

# Build the server
go build -o telemetry-server-go ./cmd/main.go

# Run with default configuration
./telemetry-server-go

Using Docker

# Pull the image
docker pull uter88/telemetry-server-go:latest

# Run the container
docker run -p 8080:8080 -p 9090:9090 \
  -e STORAGES=tdengine \
  -e PUBLISHERS=kafka \
  uter88/telemetry-server-go

📡 Protocol Endpoints

HTTP/HTTPS

POST /api/v1/telemetry
Content-Type: application/json

{
  "timestamp": "2024-01-15T10:30:00Z",
  "device_id": "device-123",
  "metrics": {
    "cpu_usage": 45.2,
    "memory_used": 2048,
    "temperature": 65.5
  },
  "tags": {
    "region": "us-west",
    "environment": "production"
  }
}

Response: 202 Accepted

WebSocket

// Client-side WebSocket connection
const ws = new WebSocket('ws://localhost:8080/ws/telemetry');
const message = {
    stream: 'realtive-metrics',
    data: { /* ... */ }
};
ws.send(JSON.stringify(message));

TCP Console command protocol

# Send auhtorization data of current user
echo -n "*auth:username:password" | nc localhost 9000

# Request information about telemetry device
echo -n "*status:<device_id>"

# Disconnect connected device
echo -n "*drop:<device_id>"

# Request help information
echo -n "*?"

# Hot reload config
echo -n "*reload"

# Shutdown telemetry server
echo -n "*shutdown"

⚙️ Configuration

The server can be configured via environment variables or a config file:

db:
  driver: postgres
  user: uter
  password: 28031988Uter
  host: localhost
  port: 5431
  dbname: main
  ssl_mode: disable
kafka:
  topic: events
  group: accounter
  brokers:
    - localhost:9092
  read_timeout: 10s
  write_timeout: 10s
  auto_commit: false
publishers: [kafka]
storages: [tdengine]
protocols:
  - {
      name: console,
      version: 1.0.0,
      is_enabled: true,
      listener_type: tcp,
      client_type: user,
      port: 9000,
      connection_deadline: 5m,
      rate_limit_number: 100,
      rate_limit_duration: 1m
  }

Environment Variables

# Postgres Configuration
export DBUSER=user
export DBPASSWORD=secret
export DBHOST=127.0.0.1
export DBPORT=5432
export DBNAME=rcv

# Kafka configuration
export QBROKERS=127.0.0.1:9092
export QTOPIC=rcv
export QGROUP=rcv-1

# Storages and publishers for packets
export STORAGES=tdengine
export PUBLISHERS=kafka

📊 Monitoring

Built-in Metrics (Prometheus)

# Server metrics
GET http://localhost:8080/metrics

# Health check
GET http://localhost:8080/health

# Ready check
GET http://localhost:8080/ready

Example Prometheus Dashboard

The server exposes the following metrics:

• telemetry_requests_total - Total number of requests

• telemetry_request_duration_seconds - Request duration histogram

• telemetry_active_connections - Current active connections

• telemetry_queue_size - Processing queue size

• telemetry_errors_total - Error count by type

🔧 Development

Project Structure

receiver/
├── cmd/
│   └── main.go          # Main application entry point
├── internal/
│   ├── core/            # Application core
│   ├── domain/          # Domain layer
│   │   ├── client/      # Client entity
│   │   ├── packet/      # Telemetry packet entity
│   │   ├── protocol/    # Base protocol entity
│   ├── infrastructure/  # Infrastructure layer
│   │   ├── listeners/   # TCP, HTTP, Websocket listeners
│   │   ├── metrics/     # Metrics collectors
│   │   ├── protocols/   # Data processing protocols
│   │   ├── publishers/  # Telemetry packets publishers
│   │   ├── repositories/# Data repositories
│   │   ├── storages/    # Telemetry packets storages
│   ├── ports/           # Common ports
├── pkg/
│   ├── worker/          # Worker pool
│   └── utils/           # Utility functions
├── api/                 # API specifications
├── deployments/         # Docker & K8s manifests
└── tests/               # Test files
└── config/              # Application config

Running Tests

# Run all tests
go test ./...

# Run tests with coverage
go test -coverprofile=coverage.out ./...
go tool cover -html=coverage.out

# Run integration tests
make test-integration

# Run benchmark tests
go test -bench=. ./...

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

TODO:

• 📖 Documentation
• Tests
• Telemetry protocols
  • EGTS
  • FLEX
  • ADM
  • Arnavi
  • Galileo
  • Concox
  • Wialon IPS
  • Wialon retranslator
  • Wialon combine
  • Mayak
  • Teltonika

🙏 Acknowledgments

• Go Standard Library for excellent networking packages

• Prometheus for metrics collection

• TDEngine for time-series storage