[TRUNK-17697] Document Prometheus metrics endpoint for merge queue by samgutentag · Pull Request #460 · trunk-io/docs

samgutentag · 2026-03-12T16:46:53Z

Summary

• Added new "Prometheus metrics endpoint" section to Metrics and monitoring with metrics table, scrape config, and Grafana query examples
• Added "Metrics Endpoints" section to API reference linking to the Prometheus docs

Context links

trunk-io/trunk2#2859, #2963, #3171, #3095 (all merged)

Files changed

merge-queue/administration/metrics.md — new Prometheus section
merge-queue/reference/merge.md — new Metrics Endpoints section

Open questions

Critical: What is the exact endpoint URL? (Draft assumes /v1/merge-queue/metrics on api.trunk.io)
Critical: What auth mechanism? (Draft assumes bearer token via Trunk API token)
What are the exact metric names and labels? (Draft provides best guesses from PR code)
Are there any rate limits on scraping?
Is this available on all plans or only certain tiers?

Test plan

Check GitBook preview for both pages
Eng review required — metric names, endpoint URL, and auth need verification
Verify Grafana query examples are syntactically valid

🤖 Generated with Claude Code

Add Prometheus scrape endpoint docs with available metrics, example scrape config, and Grafana queries. New section in metrics page and reference from API page. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

trunk-staging-io · 2026-03-12T16:46:58Z

✨ Submitted to Merge by github-actions[bot]. It will be added to the merge queue once all branch protection rules pass and there are no merge conflicts with the target branch. See more details here.

claude · 2026-03-12T16:50:19Z

Claude finished @samgutentag's task in 1m 53s —— View job

Review complete

Read changed files
Check CLAUDE.md for repo guidelines (none found)
Review for typos, grammar, and formatting
Review documentation best practices
Post inline comments
Final summary

Summary

Good foundation — the structure is clear and the cross-reference from the API page is well done. A few issues to address before merging:

Blocking issues:

Broken PromQL query (metrics.md line 157) — The histogram_quantile example is missing the required sum(rate(...)) by (le) wrapper. As written, it will fail or return incorrect results. Fix this →
Auth mechanism mismatch (metrics.md line 139) — The scrape config uses Prometheus bearer_token (sets Authorization: Bearer header), but the rest of the API uses x-api-token header. Needs eng confirmation of which auth the Prometheus endpoint actually accepts before documenting this. Fix this →

Non-blocking:

Heading level inconsistency (metrics.md line 110) — ## Prometheus metrics endpoint uses h2 while all other sections in the file use h3 (###). Should be ###, with subsections bumped to ####. Fix this →

The author's own open questions (endpoint URL, exact metric names, rate limits, plan availability) are the real blockers for this content being accurate — eng review is clearly needed before this merges.

merge-queue/administration/metrics.md

claude · 2026-03-12T16:51:48Z

merge-queue/administration/metrics.md

+
+**P95 wait time:**
+```promql
+histogram_quantile(0.95, trunk_merge_queue_wait_time_seconds_bucket{repo="my-org/my-repo"})


This histogram_quantile query is incomplete — it will produce incorrect results or errors. Prometheus histogram quantile requires wrapping with sum(rate(...)) by (le). The corrected form:

Suggested change

histogram_quantile(0.95, trunk_merge_queue_wait_time_seconds_bucket{repo="my-org/my-repo"})

histogram_quantile(0.95, sum(rate(trunk_merge_queue_wait_time_seconds_bucket{repo="my-org/my-repo"}[5m])) by (le))

The [5m] range vector is also required by rate(). Adjust the window to your scrape interval needs.

@blx can you verify this?

merge-queue/administration/metrics.md

Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>

…ndpoint details Replaces placeholder metric names (trunk_merge_queue_*) with actual implementation (mq_queue_depth_current, mq_pr_conclusions_1h_total, etc.). Adds correct endpoint path, auth mechanism, query params, histogram details, gauge semantics caveat, sample output, and PromQL examples from the Prometheus Metrics Spec. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

claude · 2026-03-13T17:50:51Z

Claude finished @samgutentag's task in 1m 57s —— View job

Review complete

Read changed files
Check CLAUDE.md for repo guidelines (none found)
Review for typos, grammar, and formatting
Review documentation best practices
Final summary

Summary

The two blocking issues from the previous review have been resolved in this update:

Auth mechanism — now correctly uses x-api-token header (metrics.md lines 188–189) ✅
PromQL queries — no longer apply rate() to gauge-semantics metrics, consistent with the inline warning ✅

Non-blocking issues remaining:

Heading hierarchy (metrics.md lines 118, 135, 173, 194, 237) — The parent section is ### Prometheus metrics endpoint (h3), but all subsections (Endpoint, Available metrics, Scrape configuration, Example queries, Sample output) also use ###. They should be ####, and #### Point-in-time gauges / #### Rolling 1-hour window metrics (lines 145, 155) should become #####. Fix this →
Histogram PromQL missing by (le) (metrics.md lines 225, 231) — histogram_quantile queries work as-is for single-series cases, but best practice is to add sum(...) by (le) to correctly aggregate across label combinations (different repos/branches). Given the gauge semantics, rate() should be omitted — the corrected form would be histogram_quantile(0.90, sum(mq_pr_wait_duration_1h_seconds_bucket) by (le)). Fix this →

The author's open questions about exact metric names, rate limits, and plan availability remain — eng sign-off is still needed before merging.

github-actions · 2026-03-13T17:53:04Z

/trunk merge

github-actions

Auto-approved: Claude code review passed.

blx

a few notes

blx · 2026-03-13T18:03:58Z

merge-queue/administration/metrics.md

+{% hint style="info" %}
+The Prometheus metrics endpoint is available on select plans. Contact your Trunk representative for access.
+{% endhint %}


It is available to all merge queue users currently

blx · 2026-03-13T18:05:57Z

merge-queue/administration/metrics.md

+| Parameter | Required | Description |
+| --- | --- | --- |
+| `repo` | No | Repository in `owner/name` format (e.g., `my-org/my-repo`). If omitted, returns metrics for all repositories in the organization. |
+| `repoHost` | No | Repository host. Defaults to `github.com`. |


repo and repoHost must be provided together, if either is specified. It does not default to github. So either no query params, or both repo and repoHost.

For repoHost, I would keep the default text but make it an example instead ("Example: github.com")

blx · 2026-03-13T18:06:42Z

merge-queue/administration/metrics.md

+| `repo` | No | Repository in `owner/name` format (e.g., `my-org/my-repo`). If omitted, returns metrics for all repositories in the organization. |
+| `repoHost` | No | Repository host. Defaults to `github.com`. |
+
+The response uses content type `text/plain; version=0.0.4; charset=utf-8` (standard Prometheus format). Metrics are computed on-demand with a 60-second cache.


I would omit the computed on-demand cache part -- that is internal and subject to change and not relevant to the API contract exposed.

blx · 2026-03-13T18:08:09Z

merge-queue/administration/metrics.md

+| --- | --- | --- |
+| `repo` | Repository name | `my-org/my-repo` |
+| `branch` | Base branch name | `main`, `develop` |
+| `queue_type` | Queue type | `main`, `bisection` |


to clarify: queue_type will be only main or bisection (it's an enum), whereas the examples for repo and branch are good but ultimately freeform text

blx · 2026-03-13T18:09:02Z

merge-queue/administration/metrics.md

+
+| Metric | Type | Description |
+| --- | --- | --- |
+| `mq_queue_depth_current` | Gauge | Number of PRs currently in the queue (excludes NOT\_READY PRs) |


blx · 2026-03-13T18:11:41Z

merge-queue/administration/metrics.md

+| `mq_pr_restarts_1h_total` | Gauge | — | PR restarts (TESTING to PENDING transitions) in the last hour |
+| `mq_pr_wait_duration_1h_seconds` | Histogram | `le` (bucket boundary) | Distribution of time PRs spent waiting before testing starts |
+| `mq_pr_test_duration_1h_seconds` | Histogram | `le` (bucket boundary) | Distribution of time PRs spent in the testing phase |
+| `mq_pr_time_in_queue_1h_seconds` | Histogram | `conclusion`, `le` | Distribution of total time in queue for PRs that concluded in the last hour |


this is not available yet - please remove for now

blx · 2026-03-13T18:13:39Z

merge-queue/administration/metrics.md

+# P95 total time in queue for merged PRs
+histogram_quantile(0.95, mq_pr_time_in_queue_1h_seconds_bucket{conclusion="merged"})


omit this until time in queue metric is added

blx · 2026-03-13T18:14:03Z

merge-queue/reference/merge.md

+| Parameter | Required | Description |
+| --- | --- | --- |
+| `repo` | No | Repository in `owner/name` format. If omitted, returns metrics for all repositories in the organization. |
+| `repoHost` | No | Repository host. Defaults to `github.com`. |


does not default, and is required if repo is passed - see note above

Document Prometheus metrics endpoint for merge queue

ca912f2

Add Prometheus scrape endpoint docs with available metrics, example scrape config, and Grafana queries. New section in metrics page and reference from API page. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

samgutentag changed the title ~~Document Prometheus metrics endpoint for merge queue~~ [TRUNK-17697] Document Prometheus metrics endpoint for merge queue Mar 12, 2026

claude bot reviewed Mar 12, 2026

View reviewed changes

merge-queue/administration/metrics.md Outdated Show resolved Hide resolved

claude bot reviewed Mar 12, 2026

View reviewed changes

merge-queue/administration/metrics.md Outdated Show resolved Hide resolved

claude bot reviewed Mar 12, 2026

View reviewed changes

merge-queue/administration/metrics.md Outdated Show resolved Hide resolved

Update merge-queue/administration/metrics.md

148ad33

Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>

samgutentag requested review from blx and pv72895 March 13, 2026 17:18

github-actions bot approved these changes Mar 13, 2026

View reviewed changes

blx requested changes Mar 13, 2026

View reviewed changes

	histogram_quantile(0.95, trunk_merge_queue_wait_time_seconds_bucket{repo="my-org/my-repo"})
	histogram_quantile(0.95, sum(rate(trunk_merge_queue_wait_time_seconds_bucket{repo="my-org/my-repo"}[5m])) by (le))

	\| `mq_queue_depth_current` \| Gauge \| Number of PRs currently in the queue (excludes NOT\_READY PRs) \|
	\| `mq_depth_current` \| Gauge \| Number of PRs currently in the queue (excludes PRs that are waiting to be mergeable before being admitted to the queue) \|

		# P95 total time in queue for merged PRs
		histogram_quantile(0.95, mq_pr_time_in_queue_1h_seconds_bucket{conclusion="merged"})

Conversation

samgutentag commented Mar 12, 2026

Summary

Context links

Files changed

Open questions

Test plan

Uh oh!

trunk-staging-io bot commented Mar 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

claude bot commented Mar 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review complete

Summary

Uh oh!

Uh oh!

claude bot Mar 12, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

claude bot commented Mar 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review complete

Summary

Uh oh!

github-actions bot commented Mar 13, 2026

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

blx left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

2 participants

trunk-staging-io bot commented Mar 12, 2026 •

edited

Loading

claude bot commented Mar 12, 2026 •

edited

Loading

claude bot commented Mar 13, 2026 •

edited

Loading