Unify non-Azure CLI E2E helpers and stabilize localhive#16131
Merged
Unify non-Azure CLI E2E helpers and stabilize localhive#16131
Conversation
Contributor
|
🚀 Dogfood this PR with:
curl -fsSL https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 16131Or
iex "& { $(irm https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 16131" |
Member
|
Let's make sure the deployment end to end tests don't regress. Right now we are down to 4 failing tests. 3 I hope will be fixed by #16125 and AcaCompactNamingUpgradeDeploymentTests, which I don't know why fails yet. 
|
davidfowl
force-pushed
the
davidfowl/unify-e2e-helpers
branch
from
8d78862 to
887df3f
Compare
eerhardt
reviewed
Apr 14, 2026
eerhardt
reviewed
Apr 14, 2026
eerhardt
reviewed
Apr 14, 2026
eerhardt
reviewed
Apr 14, 2026
eerhardt
reviewed
Apr 14, 2026
eerhardt
reviewed
Apr 14, 2026
davidfowl
changed the title
davidfowl
changed the title
davidfowl
force-pushed
the
davidfowl/unify-e2e-helpers
branch
5 times, most recently
from
5e54f1e to
91670f6
Compare
Contributor
|
Re-running the failed jobs in the CI workflow for this pull request because 1 job was identified as retry-safe transient failures in the CI run attempt.
|
davidfowl
force-pushed
the
davidfowl/unify-e2e-helpers
branch
from
490eaa3 to
a84da5b
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
This PR consolidates CLI E2E test infrastructure around CliInstallStrategy, improves localhive/local package resolution behavior, and hardens E2E diagnostics/capture so failures are easier to debug across local and CI runs.
Changes:
- Unified Docker-based E2E suites to use
CliInstallStrategy+ shared automator helpers (including more robustaspire addhandling and improved diagnostics capture). - Stabilized localhive/local package flows by pinning versions from local hive contents and resolving exact local
.nupkgpaths fordotnet new install. - Improved process output draining behavior and added regression tests for stream forwarder drain/idle scenarios.
Show a summary per file
| File | Description |
|---|---|
| tests/Shared/TemporaryRepo.cs | Adds workspace preservation tracking for failed E2E runs. |
| tests/Shared/Hex1bTestHelpers.cs | Updates local recording output location docs/paths. |
| tests/Shared/Docker/Dockerfile.e2e-podman | New privileged container image for Podman-in-Docker E2E runs. |
| tests/Shared/Docker/Dockerfile.e2e | Installs docker buildx/compose plugins for publish/deploy flows. |
| tests/Aspire.Cli.Tests/Utils/AppHostHelperTests.cs | Adds version-compatibility test coverage for AppHost checks. |
| tests/Aspire.Cli.Tests/Packaging/PackagingServiceTests.cs | Adds coverage for local hive pinned-version behavior and synthetic template package results. |
| tests/Aspire.Cli.Tests/DotNet/ProcessExecutionTests.cs | New tests validating forwarder drain behavior around process exit/idle periods. |
| tests/Aspire.Cli.Tests/DotNet/DotNetCliRunnerTests.cs | Verifies local folder sources resolve to a concrete .nupkg path for template install. |
| tests/Aspire.Cli.EndToEnd.Tests/WaitCommandTests.cs | Migrates Docker E2E to CliInstallStrategy install flow. |
| tests/Aspire.Cli.EndToEnd.Tests/TypeScriptStarterTemplateTests.cs | Enables workspace capture on failure. |
| tests/Aspire.Cli.EndToEnd.Tests/TypeScriptSqlServerNativeAssetsBundleTests.cs | Uses CliInstallStrategy and simplifies local channel handling. |
| tests/Aspire.Cli.EndToEnd.Tests/TypeScriptReusablePackageTests.cs | Migrates Docker E2E to CliInstallStrategy install flow. |
| tests/Aspire.Cli.EndToEnd.Tests/TypeScriptPublishTests.cs | Migrates to CliInstallStrategy; updates GA-vs-PR skip logic. |
| tests/Aspire.Cli.EndToEnd.Tests/TypeScriptPolyglotTests.cs | Migrates to CliInstallStrategy and removes legacy mode branching. |
| tests/Aspire.Cli.EndToEnd.Tests/TypeScriptEmptyAppHostTemplateTests.cs | Migrates to CliInstallStrategy; enables workspace capture on failure. |
| tests/Aspire.Cli.EndToEnd.Tests/TypeScriptCodegenValidationTests.cs | Removes local helper terminal logic; standardizes on CliInstallStrategy. |
| tests/Aspire.Cli.EndToEnd.Tests/StopNonInteractiveTests.cs | Migrates Docker E2E to CliInstallStrategy install flow. |
| tests/Aspire.Cli.EndToEnd.Tests/StartStopTests.cs | Adds explicit wait for “AppHost stopped” success text. |
| tests/Aspire.Cli.EndToEnd.Tests/StagingChannelTests.cs | Migrates Docker E2E to CliInstallStrategy install flow. |
| tests/Aspire.Cli.EndToEnd.Tests/SmokeTests.cs | Enables workspace capture on failure. |
| tests/Aspire.Cli.EndToEnd.Tests/SecretTypeScriptAppHostTests.cs | Migrates Docker E2E to CliInstallStrategy install flow. |
| tests/Aspire.Cli.EndToEnd.Tests/SecretDotNetAppHostTests.cs | Migrates Docker E2E to CliInstallStrategy install flow. |
| tests/Aspire.Cli.EndToEnd.Tests/PythonReactTemplateTests.cs | Migrates Docker E2E to CliInstallStrategy install flow. |
| tests/Aspire.Cli.EndToEnd.Tests/PsCommandTests.cs | Migrates Docker E2E to CliInstallStrategy install flow. |
| tests/Aspire.Cli.EndToEnd.Tests/ProjectReferenceTests.cs | Migrates Docker E2E to CliInstallStrategy install flow. |
| tests/Aspire.Cli.EndToEnd.Tests/PodmanDeploymentTests.cs | Runs Podman scenario in privileged helper container; updates validation curl. |
| tests/Aspire.Cli.EndToEnd.Tests/PlaywrightCliInstallTests.cs | Migrates Docker E2E to CliInstallStrategy install flow. |
| tests/Aspire.Cli.EndToEnd.Tests/OtelLogsTests.cs | Migrates Docker E2E to CliInstallStrategy install flow. |
| tests/Aspire.Cli.EndToEnd.Tests/NewWithAgentInitTests.cs | Migrates Docker E2E to CliInstallStrategy; removes a prompt step. |
| tests/Aspire.Cli.EndToEnd.Tests/LogsCommandTests.cs | Migrates Docker E2E to CliInstallStrategy install flow. |
| tests/Aspire.Cli.EndToEnd.Tests/LocalConfigMigrationTests.cs | Migrates Docker E2E to CliInstallStrategy install flow. |
| tests/Aspire.Cli.EndToEnd.Tests/ListStepsTests.cs | Migrates Docker E2E to CliInstallStrategy install flow. |
| tests/Aspire.Cli.EndToEnd.Tests/KubernetesPublishTests.cs | Moves Kubernetes E2E into Docker terminal + CliInstallStrategy. |
| tests/Aspire.Cli.EndToEnd.Tests/KubernetesDeployWithValkeyTests.cs | Migrates Kubernetes E2E to Docker terminal + CliInstallStrategy. |
| tests/Aspire.Cli.EndToEnd.Tests/KubernetesDeployWithSqlServerTests.cs | Migrates Kubernetes E2E to Docker terminal + CliInstallStrategy. |
| tests/Aspire.Cli.EndToEnd.Tests/KubernetesDeployWithRedisTests.cs | Migrates Kubernetes E2E to Docker terminal + CliInstallStrategy. |
| tests/Aspire.Cli.EndToEnd.Tests/KubernetesDeployWithRabbitMQTests.cs | Migrates Kubernetes E2E to Docker terminal + CliInstallStrategy. |
| tests/Aspire.Cli.EndToEnd.Tests/KubernetesDeployWithPostgresTests.cs | Migrates Kubernetes E2E to Docker terminal + CliInstallStrategy. |
| tests/Aspire.Cli.EndToEnd.Tests/KubernetesDeployWithNatsTests.cs | Migrates Kubernetes E2E to Docker terminal + CliInstallStrategy. |
| tests/Aspire.Cli.EndToEnd.Tests/KubernetesDeployWithMySqlTests.cs | Migrates Kubernetes E2E to Docker terminal + CliInstallStrategy. |
| tests/Aspire.Cli.EndToEnd.Tests/KubernetesDeployWithMongoDBTests.cs | Migrates Kubernetes E2E to Docker terminal + CliInstallStrategy. |
| tests/Aspire.Cli.EndToEnd.Tests/KubernetesDeployWithGarnetTests.cs | Migrates Kubernetes E2E to Docker terminal + CliInstallStrategy. |
| tests/Aspire.Cli.EndToEnd.Tests/KubernetesDeployTypeScriptTests.cs | Migrates Kubernetes E2E to Docker terminal + CliInstallStrategy. |
| tests/Aspire.Cli.EndToEnd.Tests/KubernetesDeployBasicApiServiceTests.cs | Migrates Kubernetes E2E to Docker terminal + CliInstallStrategy. |
| tests/Aspire.Cli.EndToEnd.Tests/JsReactTemplateTests.cs | Enables workspace capture on failure; migrates to CliInstallStrategy. |
| tests/Aspire.Cli.EndToEnd.Tests/JavaScriptPublishTests.cs | Migrates JS publish E2E into Docker terminal + CliInstallStrategy. |
| tests/Aspire.Cli.EndToEnd.Tests/JavaPolyglotTests.cs | Migrates Docker E2E to CliInstallStrategy install flow. |
| tests/Aspire.Cli.EndToEnd.Tests/JavaEmptyAppHostTemplateTests.cs | Enables workspace capture on failure; migrates to CliInstallStrategy. |
| tests/Aspire.Cli.EndToEnd.Tests/JavaCodegenValidationTests.cs | Migrates Docker E2E to CliInstallStrategy install flow. |
| tests/Aspire.Cli.EndToEnd.Tests/Helpers/KubernetesDeployTestHelpers.cs | Improves tool download validation + kind kubeconfig internal endpoint for container runs; standardizes aspire add completion. |
| tests/Aspire.Cli.EndToEnd.Tests/Helpers/CliInstallStrategyTests.cs | Adds tests for PR-install args and workflow-run-id propagation. |
| tests/Aspire.Cli.EndToEnd.Tests/Helpers/CliInstallStrategy.cs | Propagates optional workflow run id to container env for PR installs. |
| tests/Aspire.Cli.EndToEnd.Tests/Helpers/CliE2ETestHelpers.cs | Adds workflow-run-id parsing, capture-file support, Podman terminal builder, and capture helpers returning paths. |
| tests/Aspire.Cli.EndToEnd.Tests/Helpers/CliE2EAutomatorHelpers.cs | Refactors install flows, makes aspire add tolerant to version prompt, improves diagnostics capture and aspire start failure handling. |
| tests/Aspire.Cli.EndToEnd.Tests/Helpers/CaptureWorkspaceOnFailureAttribute.cs | Adds file capture support; prints local capture locations; toggles local workspace preservation behavior. |
| tests/Aspire.Cli.EndToEnd.Tests/Fixtures/JsPublish/verify.sh | Adds published-host resolution + wait helpers for more reliable HTTP verification. |
| tests/Aspire.Cli.EndToEnd.Tests/EmptyAppHostTemplateTests.cs | Enables workspace capture on failure; migrates to CliInstallStrategy. |
| tests/Aspire.Cli.EndToEnd.Tests/DoctorCommandTests.cs | Migrates Docker E2E to CliInstallStrategy install flow. |
| tests/Aspire.Cli.EndToEnd.Tests/DockerDeploymentTests.cs | Runs deployment E2E inside Docker terminal + CliInstallStrategy; improves service reachability verification. |
| tests/Aspire.Cli.EndToEnd.Tests/DescribeCommandTests.cs | Migrates Docker E2E to CliInstallStrategy install flow. |
| tests/Aspire.Cli.EndToEnd.Tests/DashboardOtelTracesTests.cs | Migrates Docker E2E to CliInstallStrategy install flow. |
| tests/Aspire.Cli.EndToEnd.Tests/ConfigMigrationTests.cs | Migrates migration E2E to CliInstallStrategy install flow. |
| tests/Aspire.Cli.EndToEnd.Tests/ConfigHealingTests.cs | Migrates Docker E2E to CliInstallStrategy install flow. |
| tests/Aspire.Cli.EndToEnd.Tests/ConfigDiscoveryTests.cs | Migrates Docker E2E to CliInstallStrategy install flow. |
| tests/Aspire.Cli.EndToEnd.Tests/CertificatesCommandTests.cs | Migrates Docker E2E to CliInstallStrategy install flow. |
| tests/Aspire.Cli.EndToEnd.Tests/CentralPackageManagementTests.cs | Migrates Docker E2E to CliInstallStrategy install flow. |
| tests/Aspire.Cli.EndToEnd.Tests/BundleSmokeTests.cs | Enables workspace capture on failure; switches to shared AspireStartAsync/AspireStopAsync. |
| tests/Aspire.Cli.EndToEnd.Tests/BannerTests.cs | Migrates Docker E2E to CliInstallStrategy install flow. |
| tests/Aspire.Cli.EndToEnd.Tests/AgentCommandTests.cs | Migrates Docker E2E to CliInstallStrategy install flow. |
| src/Shared/BackchannelConstants.cs | Removes trailing whitespace (formatting-only). |
| src/Aspire.ProjectTemplates/Aspire.ProjectTemplates.csproj | Removes a comment line in the templates version replacement target. |
| src/Aspire.Cli/Packaging/PackagingService.cs | Pins local-hive channels based on highest local package semver. |
| src/Aspire.Cli/DotNet/ProcessExecution.cs | Adds idle-based forwarder drain before closing streams to avoid truncating tail output. |
| src/Aspire.Cli/DotNet/DotNetCliRunner.cs | Resolves local template .nupkg paths and avoids --force duplicates by uninstall+reinstall. |
| localhive.sh | Filters AppleDouble artifacts and disables macOS metadata in tar; improves copy counts/logging. |
| .github/workflows/run-tests.yml | Passes workflow run id into tests to enable deterministic PR artifact resolution. |
| .github/skills/cli-e2e-testing/SKILL.md | Updates local recording location documentation. |
Copilot's findings
- Files reviewed: 77/77 changed files
- Comments generated: 2
JamesNK
reviewed
Apr 17, 2026
Member
JamesNK
left a comment
JamesNK
left a comment
There was a problem hiding this comment.
Reviewed the full PR (77 files). Found 2 issues worth flagging — 1 potential container leak in the new Podman test terminal helper, and 1 unconditional assertion that would break local runs of the Docker deployment tests.
davidfowl
force-pushed
the
davidfowl/unify-e2e-helpers
branch
from
97b4695 to
39ba86f
Compare
JamesNK
approved these changes
Apr 18, 2026
- unify the shared non-Azure CLI E2E install helpers around CliInstallStrategy - keep the ProcessExecution timing stabilization - migrate the remaining CLI E2E suites onto the shared helper flow Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
davidfowl
force-pushed
the
davidfowl/unify-e2e-helpers
branch
from
39ba86f to
ec7ad10
Compare
Contributor
|
Re-running the failed jobs in the CI workflow for this pull request because 1 job was identified as retry-safe transient failures in the CI run attempt.
|
mitchdenny
approved these changes
Apr 18, 2026
Contributor
|
🎬 CLI E2E Test Recordings — 72 recordings uploaded (commit View recordings
📹 Recordings uploaded automatically from CI run #24594583887 |
radical
pushed a commit
to radical/aspire
that referenced
this pull request
Apr 18, 2026
) * Unify non-Azure CLI E2E helpers and stabilize localhive - unify the shared non-Azure CLI E2E install helpers around CliInstallStrategy - keep the ProcessExecution timing stabilization - migrate the remaining CLI E2E suites onto the shared helper flow Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix remaining PR review feedback Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Share TemporaryWorkspace delete path Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This draft consolidates the non-Azure CLI E2E helper migration with the localhive reliability fixes that fell out of validating real bundle-backed scenarios.
CliInstallStrategyand the shared terminal/install helpers..nupkgpath, improves process output draining with an idle-based forwarder wait, and filters AppleDouble._*package artifacts from localhive archives on macOS.Validation:
dotnet test tests/Aspire.Cli.Tests/Aspire.Cli.Tests.csproj -- --filter-not-trait "quarantined=true" --filter-not-trait "outerloop=true"dotnet test tests/Aspire.Cli.Tests/Aspire.Cli.Tests.csproj -- --filter-method "*.WaitForExitAsync_AllowsForwardersToDrainBeforeClosingStreams" --filter-method "*.SdkDumpCi_ForHostingProject_DoesNotEmitWarnings" --filter-not-trait "quarantined=true" --filter-not-trait "outerloop=true"ASPIRE_E2E_ARCHIVE=/tmp/aspire-stop-arm64-695e.tar.gz dotnet test tests/Aspire.Cli.EndToEnd.Tests/Aspire.Cli.EndToEnd.Tests.csproj -- --filter-method "*.DeployK8sBasicApiService" --filter-method "*.DeployTypeScriptAppToKubernetes" --filter-method "*.CreateAndDeployToDockerComposeWithPodman" --filter-not-trait "quarantined=true"Fixes # (issue)
Checklist
<remarks />and<code />elements on your triple slash comments?aspire.devissue: