Skip to content

AGENTS

Jump to bottom
Jayson Knight edited this page Feb 9, 2026 · 1 revision

AGENTS.md

This file provides guidance to WARP (warp.dev) when working with code in this repository.

Repository Overview

This is the documentation wiki for adblock-compiler - a Compiler-as-a-Service for adblock filter lists. The wiki contains comprehensive documentation including API guides, deployment instructions, architecture details, and extensibility guides.

Main Project Location: ../adblock-compiler/

Quick Reference

Project Links

Main Project Commands

# Development
deno task dev              # Development with watch mode
deno task compile          # Run compiler CLI

# Testing (always use deno task, not deno test directly)
deno task test             # Run all tests
deno task test:watch       # Tests in watch mode  
deno task test:coverage    # Generate coverage reports

# Run specific test file
deno test --allow-read --allow-write --allow-net --allow-env src/cli/ArgumentParser.test.ts

# Code Quality
deno task lint             # Lint code
deno task fmt              # Format code
deno task check            # Type check

# Build & Deploy
deno task build            # Build standalone executable
npm run dev                # Run wrangler dev server (port 8787)
npm run deploy             # Deploy to Cloudflare Workers

# Benchmarks
deno task bench            # Run performance benchmarks

Wiki Structure

adblock-compiler.wiki/
├── Home.md                    # Wiki homepage with navigation
├── README.md                  # Wiki README with quick links
├── api/
│   └── README.md              # REST API reference and examples
├── deployment/
│   ├── docker.md              # Docker deployment guide
│   └── cloudflare-containers.md  # Cloudflare edge deployment
├── guides/
│   ├── quick-start.md         # Getting started guide
│   └── clients.md             # Client library examples (Python, TypeScript, Go)
├── benchmarks.md              # Performance benchmarking guide
├── claude.md                  # AI assistant guide (detailed)
├── CLOUDFLARE_D1.md           # D1 database integration
├── DIAGNOSTICS.md             # Debugging and diagnostic tools
├── EXTENSIBILITY.md           # Custom transformations and extensions
├── IMPLEMENTATION_SUMMARY.md  # Queue implementation details
├── MIGRATION.md               # Migration from @adguard/hostlist-compiler
├── PRISMA_EVALUATION.md       # Prisma ORM evaluation notes
├── QUEUE_ARCHITECTURE.md      # Queue system architecture (with Mermaid diagrams)
├── QUEUE_SUPPORT.md           # Async compilation with Cloudflare Queues
├── testing.md                 # Testing guide
└── TROUBLESHOOTING.md         # Common issues and solutions

Architecture Overview

Two Compiler Classes

  1. FilterCompiler (src/compiler/) - File system-based, for Deno/Node.js CLI
  2. WorkerCompiler (src/platform/) - Platform-agnostic, for Workers/browsers

11 Transformations (applied in this order)

  1. ConvertToAscii - Non-ASCII to Punycode
  2. RemoveComments - Remove comment lines
  3. Compress - Hosts to adblock syntax conversion
  4. RemoveModifiers - Strip unsupported modifiers
  5. Validate - Remove dangerous/incompatible rules
  6. ValidateAllowIp - Like Validate but keeps IPs
  7. Deduplicate - Remove duplicate rules
  8. InvertAllow - Convert blocks to allow rules
  9. RemoveEmptyLines - Remove blank lines
  10. TrimLines - Remove whitespace
  11. InsertFinalNewLine - Add final newline

API Endpoints (Worker)

  • POST /compile - JSON compilation API
  • POST /compile/stream - Streaming with SSE
  • POST /compile/batch - Batch up to 10 lists
  • POST /compile/async - Queue-based async compilation
  • POST /compile/batch/async - Queue-based batch compilation
  • GET /metrics - Performance metrics
  • GET / - Admin Dashboard

Queue System (Cloudflare Queues)

  • Standard Queue: adblock-compiler-worker-queue (batch size 10, timeout 5s)
  • High Priority Queue: adblock-compiler-worker-queue-high-priority (batch size 5, timeout 2s)
  • Automatic retry with exponential backoff
  • Results cached in KV with gzip compression (70-80% reduction)

Code Conventions

Naming

  • Classes: PascalCase (FilterCompiler, RemoveCommentsTransformation)
  • Functions/methods: camelCase (executeSync, validate)
  • Constants: UPPER_SNAKE_CASE (CACHE_TTL, RATE_LIMIT_MAX_REQUESTS)
  • Interfaces: I-prefixed (IConfiguration, ILogger, ISource)

TypeScript

  • Strict mode enabled
  • No implicit any
  • Use import type for type-only imports
  • Explicit return types on public methods

File Organization

  • Tests co-located as *.test.ts next to source files
  • Each module in its own directory with index.ts exports

Key Configuration Files

Main Project

File Purpose
deno.json Deno tasks and configuration
wrangler.toml Cloudflare Workers config
package.json npm scripts for wrangler
src/types/index.ts Core type definitions
worker/worker.ts Cloudflare Worker API handler

Configuration Schema

interface IConfiguration {
    name: string;           // Required
    description?: string;
    homepage?: string;
    license?: string;
    version?: string;
    sources: ISource[];     // Required, non-empty
    transformations?: TransformationType[];
    exclusions?: string[];  // Patterns to exclude
    inclusions?: string[];  // Patterns to include
}

interface ISource {
    source: string;         // URL or file path
    name?: string;
    type?: 'adblock' | 'hosts';
    transformations?: TransformationType[];
    exclusions?: string[];
    inclusions?: string[];
}

Documentation Guidelines

When Editing Wiki Pages

  • Use GitHub Wiki markdown syntax
  • Internal links use [[Page Name|path/to/page]] format
  • Code examples should use appropriate language identifiers
  • Include Mermaid diagrams for complex flows (see QUEUE_ARCHITECTURE.md)

Documentation Sections to Maintain

  • API documentation in api/ directory
  • Deployment guides in deployment/ directory
  • User guides in guides/ directory
  • Technical deep-dives at root level

Environment Variables

See .env.example in the main project:

  • PORT - Server port (default: 8787)
  • DENO_DIR - Deno cache directory
  • Cloudflare bindings configured in wrangler.toml

CI/CD Pipeline

GitHub Actions workflow (.github/workflows/ci.yml):

  1. Test: Run all tests with coverage
  2. Type Check: Full TypeScript validation
  3. Security: Trivy vulnerability scanning
  4. JSR Publish: Auto-publish on master push
  5. Worker Deploy: Deploy to Cloudflare Workers
  6. Pages Deploy: Deploy static assets

Testing Guidelines

  • Use Deno's native testing framework
  • Co-locate tests with source files
  • Use Deno.test() with descriptive names
  • Mock external dependencies (network, file system)
  • Run tests with required permissions: 
    deno test --allow-read --allow-write --allow-net --allow-env

Security Notes

  • No new Function() or eval() - Use safe parsers instead
  • Validate all user inputs and configurations
  • Pre-fetch content server-side to avoid CORS issues
  • Security scans run automatically in CI via Trivy

Related Documentation

Clone this wiki locally