Skip to content

App Hosting rollout can succeed while custom domain/hosted.app still serve stale routing until Cloud Run tag repoint #10040

@karabil

Description

@karabil

After a successful Firebase App Hosting rollout, the App Hosting control plane reported the new build as current, but the public custom domain and hosted.app frontend still behaved as if routing had not fully moved until the underlying Cloud Run tag was manually repointed.

[REQUIRED] Environment info

firebase-tools: 15.8.0

Platform: macOS 26.3

[REQUIRED] Steps to reproduce

  1. Create a Firebase App Hosting backend backed by Cloud Run.
  2. Trigger a rollout with firebase apphosting:rollouts:create.
  3. Wait for the rollout/build to report success.
  4. Confirm App Hosting traffic and rollout APIs both show the new build is current/succeeded.
  5. Probe the public custom domain and App Hosting hosted.app domain.
  6. Inspect Cloud Run request logs for those public requests.

[REQUIRED] Expected behavior

Once the rollout succeeds and App Hosting traffic shows the new build as the 100% current split, the custom domain and hosted.app domain should consistently route to the new build without any extra manual intervention.

[REQUIRED] Actual behavior

The App Hosting control plane showed the rollout/build as current/succeeded, but public routing still appeared stale until the App Hosting-related Cloud Run tag was manually repointed to the new revision.

Concrete observations from the incident:

  • GET /v1beta/projects/.../backends/.../traffic showed current.splits[0].build = build-2026-03-08-026 with percent = 100.
  • GET /v1/projects/.../backends/.../rollouts/build-2026-03-08-026 showed state = SUCCEEDED and reconciling = false.
  • Despite that, the public route did not reflect the expected behavior until Cloud Run tags associated with the backend were repointed to the latest revision.
  • After repointing those tags, public requests started landing on the expected revision.

Additional confusing detail:

  • Cloud Run service metadata can still show older App Hosting tag entries at the same time the public frontend is effectively routing to the newer revision.
  • The App Hosting traffic resource updateTime also appeared stale even though current.splits had moved to the new build.

This makes it hard to know whether the rollout is actually complete from the operator side. The rollout API says success, but the effective public routing can still require manual recovery.

Useful debugging data

What helped us verify the mismatch:

  • App Hosting traffic API
  • App Hosting rollout API
  • Cloud Run request logs for the public custom domain / hosted.app requests
  • Cloud Run tag state on the managed service

If useful, I can add sanitized request-log examples showing the custom domain being rewritten through a specific tagged Cloud Run hostname and then landing on the newer revision only after manual tag repointing.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions