From e0f413fd6d97a03a8cf0267d3fd5690c55cc4d0d Mon Sep 17 00:00:00 2001 From: "Fei Fang (KEVIN)" Date: Tue, 17 Feb 2026 23:03:25 -0800 Subject: [PATCH 1/4] Add new TSG for network infra connection validator failure due to exception --- TSG/EnvironmentValidator/Networking/README.md | 1 + ...Test-Infra-IP-Connection-ExceptionFound.md | 173 ++++++++++++++++++ 2 files changed, 174 insertions(+) create mode 100644 TSG/EnvironmentValidator/Networking/Troubleshoot-Network-Test-Infra-IP-Connection-ExceptionFound.md diff --git a/TSG/EnvironmentValidator/Networking/README.md b/TSG/EnvironmentValidator/Networking/README.md index 98e28fa..ab605f1 100644 --- a/TSG/EnvironmentValidator/Networking/README.md +++ b/TSG/EnvironmentValidator/Networking/README.md @@ -14,6 +14,7 @@ For Azure Local Network Resources not related to Environment Validator, see [TSG - [Troubleshoot: Cluster Storage Intent Exists](Troubleshoot-Network-Test-Cluster-StorageIntent-Exists.md) - [Troubleshoot: Host Network Configuration Readiness](Troubleshoot-Network-Test-HostNetworkConfigurationReadiness.md) - [Troubleshoot: Infrastructure IP Azure Endpoint Connection](Troubleshoot-Network-Test-InfraIP-Azure-Endpoint-Connection.md) +- [Troubleshoot: Infrastructure IP Connection Exception Found](Troubleshoot-Network-Test-Infra-IP-Connection-ExceptionFound.md) - [Troubleshoot: Infrastructure IP DNS Client Readiness](Troubleshoot-Network-Test-InfraIP-DNS-Client-Readiness.md) - [Troubleshoot: Infrastructure IP DNS Port 53 Connection](Troubleshoot-Network-Test-InfraIP-DNS-Port-53.md) - [Troubleshoot: Infrastructure IP Hyper-V Readiness](Troubleshoot-Network-Test-InfraIP-Hyper-V-Readiness.md) diff --git a/TSG/EnvironmentValidator/Networking/Troubleshoot-Network-Test-Infra-IP-Connection-ExceptionFound.md b/TSG/EnvironmentValidator/Networking/Troubleshoot-Network-Test-Infra-IP-Connection-ExceptionFound.md new file mode 100644 index 0000000..d42591e --- /dev/null +++ b/TSG/EnvironmentValidator/Networking/Troubleshoot-Network-Test-Infra-IP-Connection-ExceptionFound.md @@ -0,0 +1,173 @@ +# AzureLocal_NetworkInfraConnection_Test_Infra_IP_Connection_ExceptionFound + + + + + + + + + + + + + + +
NameAzureLocal_NetworkInfraConnection_Test_Infra_IP_Connection_ExceptionFound
SeverityCritical
Applicable ScenariosDeployment (without ArcGateway), Upgrade (without ArcGateway)
+ +## Overview + +This is a **catch-all exception handler** for the infrastructure IP pool connection validation. When the `Test-NwkInfraConnectionValidator_InfraIpPoolConnection` function encounters an unhandled exception during any phase of infrastructure IP connectivity validation, this result is returned. + +Unlike other validators in the `NetworkInfraConnection` family that report specific issues (e.g., DNS failure, vNIC readiness), this validator fires when an **unexpected error** occurs — typically due to environmental issues such as VMSwitch configuration failures, Hyper-V component errors, or other system-level problems that prevent the validator from completing normally. + +The exception message and stack trace are captured in the `AdditionalData.Detail` field. + +## Requirements + +1. Hyper-V role must be installed and functioning +2. VMSwitch must be configurable on the host +3. Network adapters specified in the management intent must be present and operational +4. Infrastructure IP pool must be valid and reachable +5. System must have sufficient resources to create test virtual network adapters + +## Troubleshooting Steps + +### Review Environment Validator Output + +Review the Environment Validator output JSON. The `AdditionalData.Detail` field contains the exception message and stack trace, which is essential for identifying the root cause. + +```json +{ + "Name": "AzureLocal_NetworkInfraConnection_Test_Infra_IP_Connection_ExceptionFound", + "DisplayName": "Exception found during infra IP pool connection validation.", + "Title": "Exception found during infra IP pool connection validation.", + "Status": 1, + "Severity": 2, + "Description": "Experienced exception during infra IP pool readiness validation. Please check information in AdditionalData.Detail section", + "Remediation": "URI", + "TargetResourceID": "Infra_IP_Connection_Exception", + "TargetResourceName": "Infra_IP_Connection_Exception", + "TargetResourceType": "InfraIpPool", + "Timestamp": "", + "AdditionalData": { + "Source": "Infra_IP_Connection_Exception", + "Resource": "Infra_IP_Connection_Exception", + "Detail": "\n\r", + "Status": "FAILURE", + "TimeStamp": "" + } +} +``` + +> **Important:** Copy the full `AdditionalData.Detail` content. The exception message and stack trace are the primary clues for diagnosing the root cause. + +--- + +### Failure: General Exception During Validation + +If the exception message does not match the VMSwitch scenario above, the exception may have been thrown by any of the sub-steps in the infra IP validation process: + +1. **VMSwitch creation/discovery** — Could not find or create a suitable external VMSwitch +2. **vNIC creation** — Could not create the test virtual network adapter on the VMSwitch +3. **IP assignment** — Could not assign infrastructure IP to the test vNIC +4. **Gateway ping** — Error while testing ICMP connectivity to the default gateway +5. **DNS testing** — Error while testing DNS server connectivity on port 53 +6. **Endpoint connectivity** — Error while running curl.exe tests to Azure endpoints + +#### Remediation Steps + +##### 1. Read the Exception Details Carefully + +The `AdditionalData.Detail` field contains both the exception message (first line) and the stack trace (subsequent lines). The stack trace shows which function threw the error. + +**Common patterns in the stack trace:** + +| Stack Trace Contains | Likely Cause | +|----------------------|--------------| +| `EnvValidatorNwkLibConfigureVMSwitchForTesting` | VMSwitch setup failed | +| `New-VMSwitch` | VMSwitch creation failed | +| `Add-VMNetworkAdapter` or `Get-VMNetworkAdapter` | vNIC management failed | +| `New-NetIPAddress` | IP assignment failed | +| `InfraIpCurlTestToEndpoint` | curl endpoint test failed | +| `Get-AzStackHciConnectivityTarget` | Failed to fetch endpoint manifest | + +##### 2. Check System Prerequisites + +```powershell +# Verify Hyper-V feature and management tools +Get-WindowsFeature -Name "Hyper-V", "Hyper-V-PowerShell" | Select-Object Name, Installed + +# Check VMSwitch creation capability +Get-Command Get-VMSwitch -ErrorAction SilentlyContinue | Select-Object Name, Source +``` + +##### 3. Review System Event Logs + +```powershell +# Check for recent errors in System log related to networking/Hyper-V +$startTime = (Get-Date).AddHours(-2) +Get-WinEvent -LogName System -MaxEvents 100 | + Where-Object { $_.TimeCreated -gt $startTime } | + Where-Object { $_.Message -like "*network*" -or $_.Message -like "*Hyper-V*" -or $_.Message -like "*switch*" } | + Select-Object TimeCreated, LevelDisplayName, Message | + Format-Table -Wrap +``` + +##### 4. Re-install Hyper-V +If the exception is complaining something like below: +``` +Failed while adding virtual Ethernet switch connections. Switch port create failed, switch = "ABCDEF00-1234-5678-ABCD-ABCDABCDABCD", port name = "ABCDEF00-1234-5678-ABCD-ABCDABCDABCD", port friendly name = "ConvergedSwitch(MYINTENT)": Class not registered (0x80040154) +``` +It means the Hyper-V is not installed correctly on the machine, you will need to re-install Hyper-V feature on the machine: + +```powershell +Disable-WindowsOptionalFeature -Online -FeatureName Microsoft-Hyper-V +# reboot machine +Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Hyper-V +# reboot machine again +``` +##### 5. Retry the Validation +After addressing the underlying issue, re-run the Environment Validator. + +--- + +## Additional Information + +### How This Validator is Triggered + +This result is produced by the top-level `catch` block in the `Test-NwkInfraConnectionValidator_InfraIpPoolConnection` function. The function performs these steps inside a `try` block: + +1. Retrieves infrastructure IP range from IP pools +2. Validates Hyper-V and VMSwitch prerequisites +3. Finds or creates an external VMSwitch using management intent adapters +4. Creates a test vNIC on the VMSwitch +5. For each infrastructure IP (up to 9): + - Assigns the IP to the test vNIC + - Tests gateway connectivity (ICMP ping) + - Tests DNS connectivity (port 53) + - Tests Azure endpoint connectivity (curl.exe) +6. Cleans up test resources + +If **any** unhandled exception occurs during steps 1–6, the catch block returns this `ExceptionFound` result with the exception details. + +### When This Validator is Skipped + +The infrastructure IP connectivity validator (including all endpoint tests) is **skipped** when **ArcGateway enabled**. ArcGateway provides alternative connectivity. + +### Related Validators + +This exception can mask failures that would normally be reported by these specific validators: + +- `AzureLocal_NetworkInfraConnection_Test_Infra_IP_Connection_Hyper_V_Readiness` +- `AzureLocal_NetworkInfraConnection_Test_Infra_IP_Connection_VMSwitch_Readiness` +- `AzureLocal_NetworkInfraConnection_Test_Infra_IP_Connection_MANAGEMENT_VNIC_Readiness` +- `AzureLocal_NetworkInfraConnection_Test_Infra_IP_Connection_vNIC_Readiness` +- `AzureLocal_NetworkInfraConnection_Test_Infra_IP_Connection_DNSClientServerAddress_Readiness` +- `AzureLocal_NetworkInfraConnection_Test_Infra_IP_Connection_IPReadiness` +- `AzureLocal_NetworkInfraConnection_Test_Infra_IP_Connection_DNS_Server_Port_53` +- `AzureLocal_NetworkInfraConnection_Test_Infra_IP_Connection_{ServiceName}` + +### Related Documentation + +- [Azure Local network requirements](https://learn.microsoft.com/azure/azure-local/concepts/host-network-requirements) From 57fcd0e07cc3af5c76b2f706bd369090b57a969b Mon Sep 17 00:00:00 2001 From: "Fei Fang (KEVIN)" Date: Fri, 20 Feb 2026 09:29:35 -0800 Subject: [PATCH 2/4] Update with new section for Troubleshoot-Network-Test-Cluster-Intent-Status --- ...hoot-Network-Test-Cluster-Intent-Status.md | 72 +++++++++++++++++++ 1 file changed, 72 insertions(+) diff --git a/TSG/EnvironmentValidator/Networking/Troubleshoot-Network-Test-Cluster-Intent-Status.md b/TSG/EnvironmentValidator/Networking/Troubleshoot-Network-Test-Cluster-Intent-Status.md index 975aee7..7e532b0 100644 --- a/TSG/EnvironmentValidator/Networking/Troubleshoot-Network-Test-Cluster-Intent-Status.md +++ b/TSG/EnvironmentValidator/Networking/Troubleshoot-Network-Test-Cluster-Intent-Status.md @@ -171,6 +171,78 @@ If you've resolved the underlying issue, you can retry the intent provisioning: --- +### Failure: Intent ConfigurationStatus Flipping Between Validating and Success + +**Error Message:** +```text +Intent on host in pending state and hasn't stabilized. ConfigurationStatus: Validating ProvisioningStatus: +``` + +**Root Cause:** A known issue causes the local status returned by `Get-NetIntentStatus` to toggle between `Validating` and `Success` after a Global Intent failure. This means the intent never stays in a stable `Success` state long enough for the validator to pass, blocking solution updates. + +You may observe this behavior when running `Get-NetIntentStatus` repeatedly — the `ConfigurationStatus` rapidly flips between `Validating` and `Success` on one or more nodes, while the `ProvisioningStatus` may appear empty during the `Validating` phase. + +#### Remediation Steps + +##### Step 1: Confirm the Flipping Behavior + +1. Run `Get-NetIntentStatus` multiple times in quick succession and observe whether `ConfigurationStatus` alternates between `Validating` and `Success`: + + ```powershell + # Run multiple times to observe the flipping behavior + Get-NetIntentStatus | ft IntentName, Host, ConfigurationStatus, ProvisioningStatus + ``` + +2. If you see the status toggling between `Validating` and `Success` (rather than staying in `Failed` or remaining stuck in `Validating`), proceed with the mitigation below. + +##### Step 2: Reset Global Intent Overrides + +The mitigation involves removing and re-adding the global cluster overrides for Network ATC. This stabilizes the intent status. + +1. Check current global overrides: + + ```powershell + # Check current overrides + $globalIntent = Get-NetIntent -GlobalOverrides + $clusterOverride = $globalIntent.ClusterOverride + $clusterOverride + ``` + +2. Create new cluster overrides and apply previous override values: + + ```powershell + $newClusterOverride = New-NetIntentGlobalClusterOverrides + + # NOTE: + # MAKE SURE YOUR NEW OVERRIDE VALUE MATCHES YOUR PREVIOUS VALUE, unless there is any empty value on the previous data + # Set overrides on object based on the old value: ex) $newClusterOverride. = + # If all the old properties are having empty value, you could put: + # $newClusterOverride.EnableLiveMigrationNetworkSelection = $true + # $newClusterOverride.EnableNetworkNaming= $true + # DO NOT USE OTHER DEFAULT VALUE IF THE OLD PROPERTIES HAVE EMPTY VALUE + ``` + +3. Remove old intent global overrides and re-add them: + + ```powershell + # Remove old intent global overrides + Remove-NetIntent -GlobalOverrides + + # Re-add global cluster overrides + Add-NetIntent -GlobalClusterOverrides $newClusterOverride + ``` + +4. Verify that the intent status is now stable: + + ```powershell + # Verify intent status is stable at Success + Get-NetIntentStatus | ft IntentName, Host, ConfigurationStatus, ProvisioningStatus + ``` + + Confirm that `ConfigurationStatus` remains `Success` and `ProvisioningStatus` is `Completed` across multiple checks. + +--- + ### Transient States The intent transient states include: From 16497f1f702f30da85d9de8f01cf8ff5ad0a3cbd Mon Sep 17 00:00:00 2001 From: fangfei Date: Fri, 20 Feb 2026 09:32:52 -0800 Subject: [PATCH 3/4] Update TSG/EnvironmentValidator/Networking/Troubleshoot-Network-Test-Infra-IP-Connection-ExceptionFound.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- ...ubleshoot-Network-Test-Infra-IP-Connection-ExceptionFound.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/TSG/EnvironmentValidator/Networking/Troubleshoot-Network-Test-Infra-IP-Connection-ExceptionFound.md b/TSG/EnvironmentValidator/Networking/Troubleshoot-Network-Test-Infra-IP-Connection-ExceptionFound.md index d42591e..da5d9a3 100644 --- a/TSG/EnvironmentValidator/Networking/Troubleshoot-Network-Test-Infra-IP-Connection-ExceptionFound.md +++ b/TSG/EnvironmentValidator/Networking/Troubleshoot-Network-Test-Infra-IP-Connection-ExceptionFound.md @@ -115,7 +115,7 @@ Get-WinEvent -LogName System -MaxEvents 100 | ``` ##### 4. Re-install Hyper-V -If the exception is complaining something like below: +If the exception message looks like the following: ``` Failed while adding virtual Ethernet switch connections. Switch port create failed, switch = "ABCDEF00-1234-5678-ABCD-ABCDABCDABCD", port name = "ABCDEF00-1234-5678-ABCD-ABCDABCDABCD", port friendly name = "ConvergedSwitch(MYINTENT)": Class not registered (0x80040154) ``` From 1e2c0423710e47b5d7a7cdcee8ffa17ff8adbf2f Mon Sep 17 00:00:00 2001 From: fangfei Date: Fri, 20 Feb 2026 09:33:08 -0800 Subject: [PATCH 4/4] Update TSG/EnvironmentValidator/Networking/Troubleshoot-Network-Test-Infra-IP-Connection-ExceptionFound.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- ...ubleshoot-Network-Test-Infra-IP-Connection-ExceptionFound.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/TSG/EnvironmentValidator/Networking/Troubleshoot-Network-Test-Infra-IP-Connection-ExceptionFound.md b/TSG/EnvironmentValidator/Networking/Troubleshoot-Network-Test-Infra-IP-Connection-ExceptionFound.md index da5d9a3..d8ca508 100644 --- a/TSG/EnvironmentValidator/Networking/Troubleshoot-Network-Test-Infra-IP-Connection-ExceptionFound.md +++ b/TSG/EnvironmentValidator/Networking/Troubleshoot-Network-Test-Infra-IP-Connection-ExceptionFound.md @@ -153,7 +153,7 @@ If **any** unhandled exception occurs during steps 1–6, the catch block return ### When This Validator is Skipped -The infrastructure IP connectivity validator (including all endpoint tests) is **skipped** when **ArcGateway enabled**. ArcGateway provides alternative connectivity. +The infrastructure IP connectivity validator (including all endpoint tests) is **skipped** when **ArcGateway is enabled**. ArcGateway provides alternative connectivity. ### Related Validators