Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 50 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,8 @@ Each skill is a self-contained module with its own model, parameters, and [commu
| Category | Skill | What It Does | Status |
|----------|-------|--------------|:------:|
| **Detection** | [`yolo-detection-2026`](skills/detection/yolo-detection-2026/) | Real-time 80+ class detection — auto-accelerated via TensorRT / CoreML / OpenVINO / ONNX | ✅|
| | [`yolo-detection-2026-coral-tpu`](skills/detection/yolo-detection-2026-coral-tpu/) | Google Coral Edge TPU — ~4ms inference via USB accelerator ([Docker-based](#detection--segmentation-skills)) | 🧪 |
| | [`yolo-detection-2026-openvino`](skills/detection/yolo-detection-2026-openvino/) | Intel NCS2 USB / Intel GPU / CPU — multi-device via OpenVINO ([Docker-based](#detection--segmentation-skills)) | 🧪 |
| **Analysis** | [`home-security-benchmark`](skills/analysis/home-security-benchmark/) | [143-test evaluation suite](#-homesec-bench--how-secure-is-your-local-ai) for LLM & VLM security performance | ✅ |
| **Privacy** | [`depth-estimation`](skills/transformation/depth-estimation/) | [Real-time depth-map privacy transform](#-privacy--depth-map-anonymization) — anonymize camera feeds while preserving activity | ✅ |
| **Segmentation** | [`sam2-segmentation`](skills/segmentation/sam2-segmentation/) | Interactive click-to-segment with Segment Anything 2 — pixel-perfect masks, point/box prompts, video tracking | ✅ |
Expand All @@ -70,6 +72,54 @@ Each skill is a self-contained module with its own model, parameters, and [commu

> **Registry:** All skills are indexed in [`skills.json`](skills.json) for programmatic discovery.

### Detection & Segmentation Skills

Detection and segmentation skills process visual data from camera feeds — detecting objects, segmenting regions, or analyzing scenes. All skills use the same **JSONL stdin/stdout protocol**: Aegis writes a frame to a shared volume, sends a `frame` event on stdin, and reads `detections` from stdout. This means every detection skill — whether running natively or inside Docker — is interchangeable from Aegis's perspective.

```mermaid
graph TB
CAM["📷 Camera Feed"] --> GOV["Frame Governor (5 FPS)"]
GOV --> |"frame.jpg → shared volume"| PROTO["JSONL stdin/stdout Protocol"]

PROTO --> NATIVE["🖥️ Native: yolo-detection-2026"]
PROTO --> DOCKER["🐳 Docker: Coral TPU / OpenVINO"]

subgraph Native["Native Skill (runs on host)"]
NATIVE --> ENV["env_config.py auto-detect"]
ENV --> TRT["NVIDIA → TensorRT"]
ENV --> CML["Apple Silicon → CoreML"]
ENV --> OV["Intel → OpenVINO IR"]
ENV --> ONNX["AMD / CPU → ONNX"]
end

subgraph Container["Docker Container"]
DOCKER --> CORAL["Coral TPU → pycoral"]
DOCKER --> OVIR["OpenVINO → Ultralytics OV"]
DOCKER --> CPU["CPU fallback"]
CORAL -.-> USB["USB/IP passthrough"]
OVIR -.-> DRI["/dev/dri · /dev/bus/usb"]
end

NATIVE --> |"stdout: detections"| AEGIS["Aegis IPC → Live Overlay + Alerts"]
DOCKER --> |"stdout: detections"| AEGIS
```

- **Native skills** run directly on the host — [`env_config.py`](skills/lib/env_config.py) auto-detects the GPU and converts models to the fastest format (TensorRT, CoreML, OpenVINO IR, ONNX)
- **Docker skills** wrap hardware-specific runtimes in a container — cross-platform USB/device access without native driver installation
- **Same output** — Aegis sees identical JSONL from all skills, so detection overlays, alerts, and forensic analysis work with any backend

#### LLM-Assisted Skill Installation

Skills are installed by an **autonomous LLM deployment agent** — not by brittle shell scripts. When you click "Install" in Aegis, a focused mini-agent session reads the skill's `SKILL.md` manifest and figures out what to do:

1. **Probe** — reads `SKILL.md`, `requirements.txt`, and `package.json` to understand what the skill needs
2. **Detect hardware** — checks for NVIDIA (CUDA), AMD (ROCm), Apple Silicon (MPS), Intel (OpenVINO), or CPU-only
3. **Install** — runs the right commands (`pip install`, `npm install`, `docker build`) with the correct backend-specific dependencies
4. **Verify** — runs a smoke test to confirm the skill loads before marking it complete
5. **Determine launch command** — figures out the exact `run_command` to start the skill and saves it to the registry

This means community-contributed skills don't need a bespoke installer — the LLM reads the manifest and adapts to whatever hardware you have. If something fails, it reads the error output and tries to fix it autonomously.


## 🚀 Getting Started with [SharpAI Aegis](https://www.sharpai.org)

Expand Down
53 changes: 53 additions & 0 deletions skills/detection/yolo-detection-2026-coral-tpu/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
# ─── Coral TPU Detection — Runtime Image ─────────────────────────────────────
# Runs YOLO inference on Google Coral Edge TPU via pycoral/tflite-runtime.
# Built locally on user's machine. No external model registry.
#
# Build: docker build -t aegis-coral-tpu .
# Run: docker run -i --rm --device /dev/bus/usb \
# -v /tmp/aegis_detection:/tmp/aegis_detection \
# aegis-coral-tpu

FROM debian:bullseye-slim

# Avoid interactive prompts
ENV DEBIAN_FRONTEND=noninteractive
ENV PYTHONUNBUFFERED=1

# ─── System dependencies ─────────────────────────────────────────────────────
RUN apt-get update && apt-get install -y --no-install-recommends \
python3 \
python3-pip \
python3-dev \
gnupg2 \
ca-certificates \
curl \
usbutils \
libusb-1.0-0 \
&& rm -rf /var/lib/apt/lists/*

# ─── Add Google Coral package repository ─────────────────────────────────────
RUN echo "deb https://packages.cloud.google.com/apt coral-edgetpu-stable main" \
> /etc/apt/sources.list.d/coral-edgetpu.list \
&& curl -fsSL https://packages.cloud.google.com/apt/doc/apt-key.gpg \
| apt-key add - \
&& apt-get update \
&& apt-get install -y --no-install-recommends \
libedgetpu1-std \
&& rm -rf /var/lib/apt/lists/*

# ─── Python dependencies ─────────────────────────────────────────────────────
COPY requirements.txt /app/requirements.txt
RUN pip3 install --no-cache-dir -r /app/requirements.txt

# ─── Application code ────────────────────────────────────────────────────────
COPY scripts/ /app/scripts/
COPY models/ /app/models/

WORKDIR /app

# ─── Shared volume for frame exchange ────────────────────────────────────────
VOLUME ["/tmp/aegis_detection"]

# ─── Entry point ─────────────────────────────────────────────────────────────
# stdin/stdout JSONL protocol — same as yolo-detection-2026
ENTRYPOINT ["python3", "scripts/detect.py"]
175 changes: 175 additions & 0 deletions skills/detection/yolo-detection-2026-coral-tpu/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,175 @@
---
name: yolo-detection-2026-coral-tpu
description: "Google Coral Edge TPU — real-time object detection via Docker"
version: 1.0.0
icon: assets/icon.png
entry: scripts/detect.py
deploy: deploy.sh
runtime: docker

requirements:
docker: ">=20.10"
platforms: ["linux", "macos", "windows"]

parameters:
- name: auto_start
label: "Auto Start"
type: boolean
default: false
description: "Start this skill automatically when Aegis launches"
group: Lifecycle

- name: confidence
label: "Confidence Threshold"
type: number
min: 0.1
max: 1.0
default: 0.5
description: "Minimum detection confidence — lower than GPU models due to INT8 quantization"
group: Model

- name: classes
label: "Detect Classes"
type: string
default: "person,car,dog,cat"
description: "Comma-separated COCO class names (80 classes available)"
group: Model

- name: fps
label: "Processing FPS"
type: select
options: [0.2, 0.5, 1, 3, 5, 15]
default: 5
description: "Frames per second — Edge TPU handles 15+ FPS easily"
group: Performance

- name: input_size
label: "Input Resolution"
type: select
options: [320, 640]
default: 320
description: "320 fits fully on TPU (~4ms), 640 partially on CPU (~20ms)"
group: Performance

- name: tpu_device
label: "TPU Device"
type: select
options: ["auto", "0", "1", "2", "3"]
default: "auto"
description: "Which Edge TPU to use — auto selects first available"
group: Performance

- name: clock_speed
label: "TPU Clock Speed"
type: select
options: ["standard", "max"]
default: "standard"
description: "Max is faster but runs hotter — needs active cooling for sustained use"
group: Performance

capabilities:
live_detection:
script: scripts/detect.py
description: "Real-time object detection on live camera frames via Edge TPU"

category: detection
mutex: detection
---

# Coral TPU Object Detection

Real-time object detection using Google Coral Edge TPU accelerator. Runs inside Docker for cross-platform support. Detects 80 COCO classes (person, car, dog, cat, etc.) with ~4ms inference on 320×320 input.

## Requirements

- **Google Coral USB Accelerator** (USB 3.0 port recommended)
- **Docker Desktop 4.35+** (all platforms — Linux, macOS, Windows)
- USB 3.0 cable (the included cable is recommended)
- Adequate cooling for sustained inference

## How It Works

```
┌─────────────────────────────────────────────────────┐
│ Host (Aegis-AI) │
│ frame.jpg → /tmp/aegis_detection/ │
│ stdin ──→ ┌──────────────────────────────┐ │
│ │ Docker Container │ │
│ │ detect.py │ │
│ │ ├─ loads _edgetpu.tflite │ │
│ │ ├─ reads frame from volume │ │
│ │ └─ runs inference on TPU │ │
│ stdout ←── │ → JSONL detections │ │
│ └──────────────────────────────┘ │
│ USB ──→ /dev/bus/usb (Linux) or USB/IP (Mac/Win) │
└─────────────────────────────────────────────────────┘
```

1. Aegis writes camera frame JPEG to shared `/tmp/aegis_detection/` volume
2. Sends `frame` event via stdin JSONL to Docker container
3. `detect.py` reads frame, runs inference on Edge TPU
4. Returns `detections` event via stdout JSONL
5. Same protocol as `yolo-detection-2026` — Aegis sees no difference

## Platform Setup

### Linux
```bash
# USB Coral should be auto-detected
# Docker uses --device /dev/bus/usb for direct access
./deploy.sh
```

### macOS (Docker Desktop 4.35+)
```bash
# Docker Desktop USB/IP handles USB passthrough
# ARM64 Docker image builds natively on Apple Silicon
./deploy.sh
```

### Windows
```powershell
# Docker Desktop 4.35+ with USB/IP support
# Or WSL2 backend with usbipd-win
.\deploy.bat
```

## Model

Ships with pre-compiled `yolo26n_edgetpu.tflite` (YOLO 2026 nano, INT8 quantized, 320×320). To compile custom models:

```bash
# Requires x86_64 Linux or Docker --platform linux/amd64
python scripts/compile_model.py --model yolo26s --size 320
```

## Performance

| Input Size | Inference | On-chip | Notes |
|-----------|-----------|---------|-------|
| 320×320 | ~4ms | 100% | Fully on TPU, best for real-time |
| 640×640 | ~20ms | Partial | Some layers on CPU (model segmented) |

> **Cooling**: The USB Accelerator aluminum case acts as a heatsink. If too hot to touch during continuous inference, it will thermal-throttle. Consider active cooling or `clock_speed: standard`.
## Protocol

Same JSONL as `yolo-detection-2026`:

### Skill → Aegis (stdout)
```jsonl
{"event": "ready", "model": "yolo26n_edgetpu", "device": "coral", "format": "edgetpu_tflite", "tpu_count": 1, "classes": 80}
{"event": "detections", "frame_id": 42, "camera_id": "front_door", "objects": [{"class": "person", "confidence": 0.85, "bbox": [100, 50, 300, 400]}]}
{"event": "perf_stats", "total_frames": 50, "timings_ms": {"inference": {"avg": 4.1, "p50": 3.9, "p95": 5.2}}}
```

### Bounding Box Format
`[x_min, y_min, x_max, y_max]` — pixel coordinates (xyxy).

## Installation

```bash
./deploy.sh
```

The deployer builds the Docker image locally, probes for TPU devices, and sets the runtime command. No packages pulled from external registries beyond Docker base images and Coral apt repo.
65 changes: 65 additions & 0 deletions skills/detection/yolo-detection-2026-coral-tpu/config.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
# Coral TPU Detection Skill — Configuration Schema
# Parsed by Aegis skill-registry-service.cjs → parseConfigYaml()
# Format: params[] with key, type, label, default, description, options

params:
- key: auto_start
label: Auto Start
type: boolean
default: false
description: "Start this skill automatically when Aegis launches"

- key: confidence
label: Confidence Threshold
type: number
default: 0.5
description: "Minimum detection confidence (0.1–1.0). Lower than GPU YOLO — Edge TPU INT8 quantization produces softer scores"

- key: fps
label: Frame Rate
type: select
default: 5
description: "Detection processing rate — Edge TPU is fast enough for real-time"
options:
- { value: 0.2, label: "Ultra Low (0.2 FPS)" }
- { value: 0.5, label: "Low (0.5 FPS)" }
- { value: 1, label: "Normal (1 FPS)" }
- { value: 3, label: "Active (3 FPS)" }
- { value: 5, label: "High (5 FPS)" }
- { value: 15, label: "Real-time (15 FPS)" }

- key: classes
label: Detection Classes
type: string
default: "person,car,dog,cat"
description: "Comma-separated COCO class names to detect"

- key: input_size
label: Input Resolution
type: select
default: 320
description: "Image size for inference — 320 fits fully on TPU (~4ms), 640 partially runs on CPU (~20ms)"
options:
- { value: 320, label: "320×320 (fastest, fully on-chip)" }
- { value: 640, label: "640×640 (more accurate, partially CPU)" }

- key: tpu_device
label: TPU Device
type: select
default: auto
description: "Which Edge TPU to use — 'auto' selects the first available device"
options:
- { value: auto, label: "Auto-detect (first available)" }
- { value: "0", label: "TPU 0" }
- { value: "1", label: "TPU 1" }
- { value: "2", label: "TPU 2" }
- { value: "3", label: "TPU 3" }

- key: clock_speed
label: TPU Clock Speed
type: select
default: standard
description: "Edge TPU operating frequency — 'max' is faster but runs hotter and needs cooling"
options:
- { value: standard, label: "Standard (lower power, cooler)" }
- { value: max, label: "Maximum (faster inference, needs cooling)" }
Loading