Skip to content

Fix Dapr client startup deadlock in distributed deployment#713

Draft
Copilot wants to merge 2 commits intomasterfrom
copilot/fix-dapr-client-start-issue
Draft

Fix Dapr client startup deadlock in distributed deployment#713
Copilot wants to merge 2 commits intomasterfrom
copilot/fix-dapr-client-start-issue

Conversation

Copy link
Contributor

Copilot AI commented Mar 20, 2026

Service hosts blocked on WaitForUpdatedDatabaseAsync() before calling app.Run(), creating a circular dependency: the Dapr sidecar won't initialize its secret store until the app responds on port 8080, but port 8080 only opens after app.Run(), which requires secrets from Dapr.

Root Cause

The startup sequence was: get DB secrets → wait for DB ready → app.Run(). Dapr sidecar health-checks port 8080 to initialize, so secrets were never retrievable on first start.

Fix

Leverage IHostedLifecycleService.StartedAsync (.NET 8+), which fires after the HTTP server is listening. This decouples "HTTP API available" from "game servers ready":

app.Run()  →  HTTP port 8080 up  →  Dapr initializes  →  StartedAsync: get secrets, wait for DB, start game servers

Key Changes

  • Extensions.cs — Adds WaitForDatabaseInitializationAsync(IServiceProvider, CancellationToken) for use inside StartedAsync, awaiting both the secrets Initialization task and PersistenceContextProvider.WaitForUpdatedDatabaseAsync.

  • ConnectServerHostedServiceWrapper, GameServerHostedServiceWrapper — Converted from IHostedService to IHostedLifecycleService. Constructors now accept IServiceProvider (no DB-dependent injection at construction time); actual initialization and server start deferred to StartedAsync.

  • ManagableServerStatePublisher — Similarly converted; IManageableServer resolved lazily via a dedicated InitializeServerAsync loop within the heartbeat task, preventing duplicate event subscriptions and avoiding DB access at construction.

  • ChatServerHostedServiceWrapper (new) — Replaces the anonymous factory lambda that eagerly resolved ChatServerDefinition at host startup; initialization deferred to StartedAsync.

  • **/Program.cs (ConnectServer, GameServer, ChatServer, FriendServer, GuildServer) — Replaces the blocking WaitForUpdatedDatabaseAsync() call before app.Run() with the non-blocking WaitForDatabaseConnectionInitializationAsync() (which merely starts the background secrets-fetch task and sets up ConnectionConfigurator).

Original prompt

This section details on the original issue you should resolve

<issue_title>Dapr client fail to start at first chance</issue_title>
<issue_description>Describe the bug
When starting up all services of the distributed deployment, the dapr clients fail to load at first chance. When the corresponding service host was fast enough to retrieve secrets, it might start up on a second (manual) attempt.

To Reproduce
Steps to reproduce the behavior:

  1. Set up the distributed docker compose, or soon the new Aspire project.
  2. Start the project

Expected behavior
All components should be running without errors or warnings (all displayed green) after a short initialization time.

Additional context
The reason for the fail is simple:
The dapr clients expect the app http api to work when they start.
However, the service hosts first wait for a database connection and therefore try to retrieve connection strings as secrets from the dapr client before they do fully start up. So we have a little circular reference, which doesn't work out well.

One solution could be, that the API of the service host is immediately available after start. The functionality (e.g. the connect server, the login server etc.) is made available later.
</issue_description>

Comments on the Issue (you are @copilot in this section)


⌨️ Start Copilot coding agent tasks without leaving your editor — available in VS Code, Visual Studio, JetBrains IDEs and Eclipse.

… to StartedAsync

Co-authored-by: sven-n <5238610+sven-n@users.noreply.github.com>
Agent-Logs-Url: https://github.com/MUnique/OpenMU/sessions/b9f6bd43-e155-4f43-ad0d-a5b304a95c1b
Copilot AI changed the title [WIP] Fix Dapr client fail to start at first chance Fix Dapr client startup deadlock in distributed deployment Mar 20, 2026
Copilot AI requested a review from sven-n March 20, 2026 23:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Dapr client fail to start at first chance

2 participants