Add API-only mode and admin CLI support #408

msaroufim · 2026-02-01T03:41:35Z

This PR needs to be reviewed and tested in conjunction with gpu-mode/popcorn-cli#29

It's now possible to pass an --api-only flag to start the FastAPI server without Discord credentials. Which means admins are now determined by a hardcoded token, locally you can set this to whatever but we'll have to hardcode a specific one in heroku

I also added a skills.md for future AI so they can now fully iterate and manage e2e bot tests fully in the terminal

Also more of a drive by contribution but I changed the default behavior of not setting a gpu in a leaderboard to error, since default to H100 just seemed confusing to me

This PR is quite short, it's only the updates to md files for the AIs that are long, I don't expect a thorough review there

Overall this has been a substantial qol improvement for me because I just pass AIs a concrete task and they can finally test everything end to end without clicking buttons in discord

github-actions · 2026-02-01T03:42:34Z

Coverage report

Click to see where and how coverage changed

File	Statements	Missing	Coverage	Coverage (new stmts)	Lines missing
src/libkernelbot
problem_sync.py					72-101, 121-206, 234-300
task.py					165
utils.py					64-69, 89-91
Project Total

_{This report was generated by python-coverage-comment-action}

Copilot

Pull request overview

This PR adds an API-only run mode and HTTP admin surface for kernelbot, along with local dev/test tooling and documentation for using popcorn-cli against the API.

Changes:

Introduces --api-only mode in kernelbot.main and refactors environment initialization via init_environment(skip_discord=...).
Extends kernelbot.env and kernelbot.api.main to support admin-token–protected endpoints for job control and leaderboard/submission management.
Adds helper scripts (dev_local.sh, test_local_admin.sh) and a detailed local testing guide (SKILLS/test_bot.md) for running and validating the API/admin flow locally and against production.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
`src/kernelbot/main.py`	Adds `start_api_only()` and an `--api-only` CLI flag, and wires environment initialization to support running only the FastAPI server without the Discord bot.
`src/kernelbot/env.py`	Refactors environment loading to be explicit and idempotent, and extends the shared `env` namespace with admin-related and problem-dev-directory settings.
`src/kernelbot/api/main.py`	Imports `env`, adds helpers for deadlines and problem directory resolution, introduces `require_admin` and a suite of `/admin/*` endpoints for job control, leaderboards, submissions, and stats.
`scripts/test_local_admin.sh`	New script to spin up kernelbot in API-only mode, build popcorn-cli, and exercise the admin endpoints (start/stop/stats) including an unauthorized-access check.
`scripts/dev_local.sh`	New helper script to start kernelbot in API-only mode for local development with default port and admin token environment setup.
`SKILLS/test_bot.md`	New documentation describing end-to-end local and production testing workflows for kernelbot + popcorn-cli, including DB setup, migrations, API-only mode, admin commands, and troubleshooting.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-02-01T03:47:08Z

SKILLS/test_bot.md

+GITHUB_REPO=owner/kernelbot
+
+# Local PostgreSQL database
+DATABASE_URL=postgresql://$(whoami)@localhost:5432/kernelbot


In the .env example, DATABASE_URL is set using $(whoami) inside the value. .env files loaded via python-dotenv won’t perform shell substitution, so this literal string will be used as-is; it would be clearer and more accurate to show an explicit username (or a placeholder like <your-username>).

Suggested change

DATABASE_URL=postgresql://$(whoami)@localhost:5432/kernelbot

DATABASE_URL=postgresql://<your-username>@localhost:5432/kernelbot

Copilot · 2026-02-01T03:47:08Z

src/kernelbot/api/main.py

+def _parse_deadline(deadline: str) -> datetime.datetime:
+    for fmt in ("%Y-%m-%d %H:%M", "%Y-%m-%d"):
+        try:
+            return datetime.datetime.strptime(deadline, fmt)
+        except ValueError:
+            continue
+    raise HTTPException(
+        status_code=400,
+        detail="Invalid deadline format. Use YYYY-MM-DD or YYYY-MM-DD HH:MM",
+    )
+
+
+def _resolve_problem_directory(directory: str) -> str:
+    root = os.path.abspath(env.PROBLEM_DEV_DIR)
+    target = os.path.abspath(os.path.join(root, directory))
+    if os.path.commonpath([root, target]) != root:
+        raise HTTPException(status_code=400, detail="Invalid problem directory")
+    if not os.path.isdir(target):
+        raise HTTPException(status_code=404, detail="Problem directory not found")
+    return target
+
+


The helper functions _parse_deadline and _resolve_problem_directory are defined twice (lines 98–107 and again 120–129 / 110–117 and 132–139). This duplication is unnecessary and can be confusing; only one definition of each should remain so future edits don’t accidentally diverge between the copies.

Suggested change

def _parse_deadline(deadline: str) -> datetime.datetime:

for fmt in ("%Y-%m-%d %H:%M", "%Y-%m-%d"):

try:

return datetime.datetime.strptime(deadline, fmt)

except ValueError:

continue

raise HTTPException(

status_code=400,

detail="Invalid deadline format. Use YYYY-MM-DD or YYYY-MM-DD HH:MM",

)

def _resolve_problem_directory(directory: str) -> str:

root = os.path.abspath(env.PROBLEM_DEV_DIR)

target = os.path.abspath(os.path.join(root, directory))

if os.path.commonpath([root, target]) != root:

raise HTTPException(status_code=400, detail="Invalid problem directory")

if not os.path.isdir(target):

raise HTTPException(status_code=404, detail="Problem directory not found")

return target

# Duplicate helper definitions removed; using earlier implementations of

# _parse_deadline and _resolve_problem_directory defined above.

Copilot · 2026-02-01T03:47:08Z

src/kernelbot/env.py

+    env.ADMIN_API_TOKEN = os.getenv("ADMIN_API_TOKEN")
+    env.LOCAL_ADMIN_TOKEN = os.getenv(
+        "LOCAL_ADMIN_TOKEN", "79aa14edc807892439a3fafcae9a262f"
+    )


env.LOCAL_ADMIN_TOKEN is given a hard-coded default value, which effectively creates a well-known admin token whenever LOCAL_ADMIN_TOKEN is not explicitly set. This is a security risk, especially for misconfigured staging/production environments; it would be safer to require LOCAL_ADMIN_TOKEN to be set (or generate a random value for local/dev only) and fail fast if it is missing.

Copilot · 2026-02-01T03:47:09Z

scripts/test_local_admin.sh

+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+KERNELBOT_ROOT="$(dirname "$SCRIPT_DIR")"
+POPCORN_CLI_ROOT="/Users/marksaroufim/Dev/popcorn-cli"


POPCORN_CLI_ROOT is set to a user-specific absolute path (/Users/marksaroufim/Dev/popcorn-cli), which makes this script unusable for other developers or CI without editing the file. Consider parameterizing this via an environment variable or deriving it relative to the repository root instead of hard-coding a local path.

Suggested change

POPCORN_CLI_ROOT="/Users/marksaroufim/Dev/popcorn-cli"

POPCORN_CLI_ROOT="${POPCORN_CLI_ROOT:-"$KERNELBOT_ROOT/../popcorn-cli"}"

Copilot · 2026-02-01T03:47:09Z

scripts/test_local_admin.sh

+export POPCORN_ADMIN_TOKEN="${POPCORN_ADMIN_TOKEN:-79aa14edc807892439a3fafcae9a262f}"
+export LOCAL_ADMIN_TOKEN="${LOCAL_ADMIN_TOKEN:-79aa14edc807892439a3fafcae9a262f}"
+export PORT="${PORT:-8000}"
+


The admin-related environment variables are given a fixed default token value (79aa14edc807892439a3fafcae9a262f) for both POPCORN_ADMIN_TOKEN and LOCAL_ADMIN_TOKEN. Using a committed, well-known default for an authentication token is insecure; it would be better to require these to be explicitly set (or only default them in a clearly-scoped local/dev-only workflow) and avoid checking in any real-looking secrets.

Suggested change

export POPCORN_ADMIN_TOKEN="${POPCORN_ADMIN_TOKEN:-79aa14edc807892439a3fafcae9a262f}"

export LOCAL_ADMIN_TOKEN="${LOCAL_ADMIN_TOKEN:-79aa14edc807892439a3fafcae9a262f}"

export PORT="${PORT:-8000}"

export PORT="${PORT:-8000}"

# Require admin tokens to be explicitly set; do not use a committed default.

if [ -z "${POPCORN_ADMIN_TOKEN:-}" ] || [ -z "${LOCAL_ADMIN_TOKEN:-}" ]; then

echo "ERROR: POPCORN_ADMIN_TOKEN and LOCAL_ADMIN_TOKEN must be set in the environment before running this script."

exit 1

fi

export POPCORN_ADMIN_TOKEN

export LOCAL_ADMIN_TOKEN

Copilot · 2026-02-01T03:47:09Z

scripts/dev_local.sh

+# Set default local admin token if not set
+export LOCAL_ADMIN_TOKEN="${LOCAL_ADMIN_TOKEN:-79aa14edc807892439a3fafcae9a262f}"


LOCAL_ADMIN_TOKEN here is also defaulted to the same hard-coded token value, which again creates a well-known admin credential if the variable is not set. To avoid accidentally exposing an insecure default in local or remote environments, this script should either require LOCAL_ADMIN_TOKEN to be set explicitly or generate a per-run random value instead of embedding a constant.

Suggested change

# Set default local admin token if not set

export LOCAL_ADMIN_TOKEN="${LOCAL_ADMIN_TOKEN:-79aa14edc807892439a3fafcae9a262f}"

# Set local admin token: use existing value or generate a secure random one

if [ -z "$LOCAL_ADMIN_TOKEN" ]; then

if command -v openssl >/dev/null 2>&1; then

export LOCAL_ADMIN_TOKEN="$(openssl rand -hex 16)"

else

echo "Error: LOCAL_ADMIN_TOKEN is not set and 'openssl' is not available to generate a secure token." >&2

echo "Please set LOCAL_ADMIN_TOKEN explicitly and re-run this script." >&2

exit 1

fi

fi

CLAUDE.md

src/kernelbot/api/main.py

- Add CLAUDE.md with ruff linting instructions and codebase overview - Point Problem Configuration section to gpu-mode/reference-kernels - Add update-problems command examples to test_bot.md

- Create problem_sync.py with shared logic for downloading repos, parsing competition YAMLs, and creating/updating leaderboards - Provides sync_problems() function usable by both API and Discord bot - Includes ProblemData, CompetitionData, SyncResult data classes

- Add --api-only flag to run FastAPI server without Discord bot - Add admin endpoints: start, stop, stats, submissions, leaderboards - Add POST /admin/update-problems endpoint using shared problem_sync - Rename admin_create_leaderboard to create_dev_leaderboard - Add ADMIN_TOKEN environment variable for API authentication - Extract parse_deadline and resolve_problem_directory to shared utils - Add comprehensive tests for admin API endpoints

S1ro1

lgtm

Add 7 tests covering the admin update-problems API endpoint: - Authorization requirement - Successful sync with created/updated/skipped results - Custom problem_set, force, repository, and branch parameters - ValueError handling (400 response) - Error reporting in response

Copilot AI review requested due to automatic review settings February 1, 2026 03:41

Copilot started reviewing on behalf of msaroufim February 1, 2026 03:41 View session

Copilot AI reviewed Feb 1, 2026

View reviewed changes

msaroufim requested review from S1ro1, alexzhang13 and ngc92 February 1, 2026 07:38

ngc92 reviewed Feb 1, 2026

View reviewed changes

CLAUDE.md Outdated Show resolved Hide resolved

ngc92 reviewed Feb 1, 2026

View reviewed changes

CLAUDE.md Outdated Show resolved Hide resolved

ngc92 reviewed Feb 1, 2026

View reviewed changes

src/kernelbot/api/main.py Outdated Show resolved Hide resolved

msaroufim added 3 commits February 1, 2026 09:16

Add CLAUDE.md and update test_bot.md documentation

78919e8

- Add CLAUDE.md with ruff linting instructions and codebase overview - Point Problem Configuration section to gpu-mode/reference-kernels - Add update-problems command examples to test_bot.md

msaroufim force-pushed the feature/admin-cli-commands branch from 2cd5855 to 7fb7ac0 Compare February 1, 2026 17:17

S1ro1 approved these changes Feb 1, 2026

View reviewed changes

msaroufim merged commit 1eb2687 into main Feb 1, 2026
5 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add API-only mode and admin CLI support #408

Add API-only mode and admin CLI support #408

msaroufim commented Feb 1, 2026 •

edited

Loading

Uh oh!

github-actions bot commented Feb 1, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Feb 1, 2026

Uh oh!

Copilot AI Feb 1, 2026

Uh oh!

Copilot AI Feb 1, 2026

Uh oh!

Copilot AI Feb 1, 2026

Uh oh!

Copilot AI Feb 1, 2026

Uh oh!

Copilot AI Feb 1, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

S1ro1 left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

	DATABASE_URL=postgresql://$(whoami)@localhost:5432/kernelbot
	DATABASE_URL=postgresql://<your-username>@localhost:5432/kernelbot

	POPCORN_CLI_ROOT="/Users/marksaroufim/Dev/popcorn-cli"
	POPCORN_CLI_ROOT="${POPCORN_CLI_ROOT:-"$KERNELBOT_ROOT/../popcorn-cli"}"

		# Set default local admin token if not set
		export LOCAL_ADMIN_TOKEN="${LOCAL_ADMIN_TOKEN:-79aa14edc807892439a3fafcae9a262f}"

-# Set default local admin token if not set
-export LOCAL_ADMIN_TOKEN="${LOCAL_ADMIN_TOKEN:-79aa14edc807892439a3fafcae9a262f}"
+# Set local admin token: use existing value or generate a secure random one
+if [ -z "$LOCAL_ADMIN_TOKEN" ]; then
+    if command -v openssl >/dev/null 2>&1; then
+        export LOCAL_ADMIN_TOKEN="$(openssl rand -hex 16)"
+    else
+        echo "Error: LOCAL_ADMIN_TOKEN is not set and 'openssl' is not available to generate a secure token." >&2
+        echo "Please set LOCAL_ADMIN_TOKEN explicitly and re-run this script." >&2
+        exit 1
+    fi
+fi

Add API-only mode and admin CLI support #408

Add API-only mode and admin CLI support #408

Conversation

msaroufim commented Feb 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Feb 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Coverage report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Feb 1, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 1, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 1, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 1, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 1, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 1, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

S1ro1 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

msaroufim commented Feb 1, 2026 •

edited

Loading

github-actions bot commented Feb 1, 2026 •

edited

Loading