[STG-1232] Speed up PR Github Actions checks, add code coverage, fix e2e bb tests running not running in bb mode, remove tsup, generate unified re-used ESM and CJS builds with sourcemaps and types, switch CI to new Evals CLI

[pirate]

why

• unit tests, e2e integration tests, and evals were taking 10~15 minutes per pr
• finding the specific failing eval or test was tedious and required reading tons of log output
• we weren't leveraging pnpm or turbo caches or github action's native chromium install
• tests were flaky because CDP would randomly fail to connect when github actions runner was overloaded
• e2e bb tests were not actually running against bb, so it was previously just double-running e2e:local and reporting it as bb
• tsup has been deprecated for a while and should be replaced by esbuild

what changed

• parallelized all CI tests and broke out evals and tests into individual matrix jobs
• created unified node+pnpm+turbo setup action that all other actions call
• verified vanilla chromium is able to launch inside all the e2e local jobs before running any of our own code to catch resource contention flakyness
• add proper esm and csj builds with sourcemaps and types for aligned coverage
• remove tsup dependency
• switch to new evals cli
• add coverage reporting
• fix some flaky tests needed to get everything passing reliably

---

Summary by cubic

Speeds up CI and makes tests stable by running Vitest/Playwright/Evals on the built ESM dist with cached artifacts, verified runner Chromium, weighted Browserbase routing, and merged coverage. Publishes a unified ESM+CJS SDK with sourcemaps/types and SEA binaries from ESM; fixes Browserbase e2e, switches to the new Evals CLI, and adds flaky‑test reporting, addressing Linear STG‑1232.

• New Features

  • CI: split pnpm/Turbo caches; composite actions (setup/caches, Chromium launch check, region selection, CTRF/V8 uploads); reuse artifacts; use Actions Chromium via CHROME_PATH with --no-sandbox; unique CTRF reports; V8 coverage normalization/merge with cancellation safety; push triggers enabled.
  • Tests: run from dist/esm; env reporter; true Browserbase runs with weighted/randomized regions; hard timeouts for LLM and session create/CDP connect; faster multi‑click dispatch; higher timeouts and targeted skips; cleanup helpers; fixes for init‑script, wait‑for‑selector, and screenshot reliability.
  • Build/SDK/SEA: export map (ESM import + CJS require) with sourcemaps/types; SEA binaries built from ESM with sourcemaps and plain logs in CI; removed tsup and dotenv; AISdk/CustomOpenAI moved to lib and exported; getAISDKLanguageModel added; replay metrics now include method/parameters/result/timestamps/cost.

• Migration

  • Set STAGEHAND_BROWSER_TARGET=local|browserbase, STAGEHAND_ENV (replaces TEST_ENV), CHROME_PATH; tune LLM_MAX_MS, BROWSERBASE_SESSION_CREATE_MAX_MS, BROWSERBASE_CDP_CONNECT_MAX_MS; optionally set BROWSERBASE_REGION_DISTRIBUTION.
  • Remove dotenv; set env vars directly and import @browserbasehq/stagehand via the export map (avoid dist paths).

^{Written for commit 5c76fec. Summary will update on new commits. Review in cubic}

---

Summary by cubic

Ships an ESM‑first SDK with a CJS entry, unified sourcemaps, and types. CI runs Vitest/Playwright and the new Evals CLI against the built ESM dist using the runner’s Chromium (launch‑verified), with region‑weighted Browserbase routing, normalized V8 coverage, unique CTRF artifacts, and more stable tests (timeouts, Browserbase skips, understudy click/API fixes).

Speed up CI by running Vitest and Playwright tests in parallel matrices, add V8/CTRF coverage, and stabilize Browserbase/LLM flows with hard timeouts and better cleanup. Server integration tests now use the runner’s Chromium for reliability and are folded into the main CI.

• Refactors

  • Parallelized core unit tests via discover-core-tests/core-unit-tests (Vitest matrix, fail-fast).
  • Parallelized E2E via discover-e2e-tests (Playwright local + Browserbase) gated on core changes; weighted Browserbase region selection; verify Chromium/CDP before tests; local uses runner Chromium via CHROME_PATH and --no-sandbox; Playwright can emit JUnit when CTRF_JUNIT_PATH is set.
  • Unified Lint & Build with Turbo caching and shared artifacts; tests/evals download and reuse; prepare script skips build in CI; SEA builds support sourcemaps.
  • Enforced configurable timeouts and parallelism via env (LLM_MAX_MS, BROWSERBASE_CDP_CONNECT_MAX_MS, BROWSERBASE_SESSION_CREATE_MAX_MS, LOCAL_SESSION_LIMIT_PER_E2E_TEST, BROWSERBASE_SESSION_LIMIT_PER_E2E_TEST, EVAL_*); added LLM and Browserbase hard timeouts; improved click event dispatch to reduce remote flakiness.
  • Explicitly close Browserbase sessions in tests and evals to free capacity.
  • Added composite actions: setup-node-pnpm-turbo, verify-chromium-launch, select-browserbase-region, publish-ctrf-report (JUnit→CTRF and evals), upload-v8-coverage.
  • Renamed TEST_ENV to STAGEHAND_ENV across E2E; added an env reporter for CI logs; removed the standalone stagehand-server tests workflow and ran server tests in the unified CI with Node’s test runner and GitHub Actions Chromium.

• New Features

  • Build/Exports: ESM in dist/esm and CJS in dist/cjs via export map; types from ESM; SEA binaries built from dist with sourcemaps; server SEA job targets CJS; new build/test/coverage scripts; Node loader maps package imports to dist/esm; ESM‑safe pathing (fileURLToPath); ESLint/Prettier tweaks; dotenv removed.
  • CI: Composite actions (setup-node-pnpm-turbo, verify-chromium-launch, select-browserbase-region, upload-ctrf-report, upload-v8-coverage); uses runner Chromium via CHROME_PATH with --no-sandbox and launch checks; weighted Browserbase region selection; unique CTRF artifacts; V8 coverage normalized/merged with debug logs and graceful cancellation; Playwright/Vitest discover tests in dist/esm; push/PR triggers enabled.
  • Tests & API: Vitest/Playwright run on dist/esm; new env reporter and closeV3 helper; tighter multi‑click CDP dispatch; higher timeouts and targeted skips (e.g., console on Browserbase) reduce flakiness; local launches honor CHROME_PATH; replay metrics now include parameters/result/timestamps/cost with parser fixes; AISdk/CustomOpenAI clients moved to lib/v3/external_clients and exported; env‑driven hard timeouts for LLM ops and Browserbase session create/CDP connect; understudy API usage fixes.

• Migration

  • Set STAGEHAND_BROWSER_TARGET=local|browserbase and CHROME_PATH; tune LLM_MAX_MS, BROWSERBASE_SESSION_CREATE_MAX_MS, and BROWSERBASE_CDP_CONNECT_MAX_MS; optionally set BROWSERBASE_REGION_DISTRIBUTION.
  • dotenv support is removed; set env vars explicitly in dev and CI.
  • Import @browserbasehq/stagehand via the export map; do not reference dist paths or prior tsup outputs.

^{Written for commit 8119145. Summary will update on new commits. Review in cubic}

[cubic-dev-ai]

1 issue found across 1 file (changes from recent commits).

Prompt for AI agents (all issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/evals/scripts/build-esm.ts">

<violation number="1" location="packages/evals/scripts/build-esm.ts:35">
P2: ~80 lines of specifier-rewriting logic (`resolveRuntimeSpecifier`, `rewriteFileRuntimeSpecifiers`, `rewriteDistRuntimeSpecifiers`, and associated constants) are exact duplicates of `packages/core/scripts/build-esm.ts`. Consider extracting them into a shared build utility (e.g., alongside `test-utils` which is already shared between packages) to avoid divergence and reduce maintenance burden.</violation>
</file>

_{Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.}

[cubic-dev-ai]

P2: ~80 lines of specifier-rewriting logic (resolveRuntimeSpecifier, rewriteFileRuntimeSpecifiers, rewriteDistRuntimeSpecifiers, and associated constants) are exact duplicates of packages/core/scripts/build-esm.ts. Consider extracting them into a shared build utility (e.g., alongside test-utils which is already shared between packages) to avoid divergence and reduce maintenance burden.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At packages/evals/scripts/build-esm.ts, line 35:

<comment>~80 lines of specifier-rewriting logic (`resolveRuntimeSpecifier`, `rewriteFileRuntimeSpecifiers`, `rewriteDistRuntimeSpecifiers`, and associated constants) are exact duplicates of `packages/core/scripts/build-esm.ts`. Consider extracting them into a shared build utility (e.g., alongside `test-utils` which is already shared between packages) to avoid divergence and reduce maintenance burden.</comment>

<file context>
@@ -22,6 +32,83 @@ const run = (args: string[]) => {
   }
 };
 
+const resolveRuntimeSpecifier = (
+  importerPath: string,
+  specifier: string,
</file context>

[pirate]

no, we're getting rid of it all in a fast-follow pr that manually adds .js to all our imports

[cubic-dev-ai]

Thanks for the feedback! I've saved this as a new learning to improve future reviews.