SystemTrustCertificateWithCustomRootTrust failure in CI

[jakobbotsch]

Example console log: https://helixr18s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-123814-merge-842f5f0e87314049be/System.Security.Cryptography.Tests/1/console.5c9e8028.log?helixlogtype=result

  System.Security.Cryptography.X509Certificates.Tests.ChainTests.SystemTrustCertificateWithCustomRootTrust(addCertificateToCustomRootTrust: True) [FAIL]
      Assert.Equal() Failure: Values differ
      Expected: 3
      Actual:   2
      Stack Trace:
        /_/src/libraries/System.Security.Cryptography/tests/X509Certificates/ChainTests.cs(293,0): at System.Security.Cryptography.X509Certificates.Tests.ChainTests.SystemTrustCertificateWithCustomRootTrust(Boolean addCertificateToCustomRootTrust)
           at System.RuntimeMethodHandle.InvokeMethod(ObjectHandleOnStack target, Void** arguments, ObjectHandleOnStack sig, BOOL isConstructor, ObjectHandleOnStack result)
           at System.RuntimeMethodHandle.InvokeMethod(ObjectHandleOnStack target, Void** arguments, ObjectHandleOnStack sig, BOOL isConstructor, ObjectHandleOnStack result)
        /_/src/coreclr/System.Private.CoreLib/src/System/Reflection/MethodBaseInvoker.CoreCLR.cs(36,0): at System.Reflection.MethodBaseInvoker.InterpretedInvoke_Method(Object obj, IntPtr* args)
        /_/src/libraries/System.Private.CoreLib/src/System/Reflection/MethodBaseInvoker.cs(174,0): at System.Reflection.MethodBaseInvoker.InvokeDirectByRefWithFewArgs(Object obj, Span`1 copyOfArgs, BindingFlags invokeAttr)
    System.Security.Cryptography.X509Certificates.Tests.ChainTests.SystemTrustCertificateWithCustomRootTrust(addCertificateToCustomRootTrust: False) [FAIL]
      Assert.Equal() Failure: Values differ
      Expected: 3
      Actual:   2
      Stack Trace:
        /_/src/libraries/System.Security.Cryptography/tests/X509Certificates/ChainTests.cs(293,0): at System.Security.Cryptography.X509Certificates.Tests.ChainTests.SystemTrustCertificateWithCustomRootTrust(Boolean addCertificateToCustomRootTrust)
           at InvokeStub_ChainTests.SystemTrustCertificateWithCustomRootTrust(Object, Span`1)
        /_/src/libraries/System.Private.CoreLib/src/System/Reflection/MethodBaseInvoker.cs(95,0): at System.Reflection.MethodBaseInvoker.InvokeWithOneArg(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=1274036
Build error leg or test failing: System.Security.Cryptography.X509Certificates.Tests.ChainTests.SystemTrustCertificateWithCustomRootTrust
Pull request: https://github.com/dotnet/runtime.git/pull/123814

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "System.Security.Cryptography.X509Certificates.Tests.ChainTests.SystemTrustCertificateWithCustomRootTrust(addCertificateToCustomRootTrust: True) [FAIL]",
  "ErrorPattern": "",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=1274036
Error message validated: [System.Security.Cryptography.X509Certificates.Tests.ChainTests.SystemTrustCertificateWithCustomRootTrust(addCertificateToCustomRootTrust: True) [FAIL]]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 1/31/2026 6:48:10 PM UTC

Report

Build	Definition	Test	Pull Request
1344071	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution
1343541	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution
1338320	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution
1338630	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution
1328600	dotnet/runtime	System.Security.Cryptography.X509Certificates.Tests.ChainTests.SystemTrustCertificateWithCustomRootTrust	#125389
1328510	dotnet/runtime	System.Security.Cryptography.X509Certificates.Tests.ChainTests.SystemTrustCertificateWithCustomRootTrust	#125390
1319233	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution
1318408	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	4	8

[dotnet-policy-service]

Tagging subscribers to this area: @bartonjs, @vcsjones, @dotnet/area-system-security
See info in area-owners.md if you want to be subscribed.

[MihuBot]

I'm a bot. Here is a possible related and/or duplicate issue (I may be wrong):

• SystemTrustCertificateWithCustomRootTrust fails on macOS 26 #118646

[mangod9]

here is some copilot analysis:

Why It's Intermittent: AIA Certificate Download

  .NET's X509 chain builder on Linux performs AIA (Authority Information Access) online certificate fetching by
  default (X509ChainPolicy.DisableCertificateDownloads = false). When the chain builder can't find "DigiCert Global
  Root G2" in the local Linux Bionic trust store, it looks at the AIA extension in MicrosoftDotComIssuerBytes and
  tries to download the root cert over HTTP (e.g., from http://cacerts.digicert.com/DigiCertGlobalRootG2.crt).

  ┌───────────────┬────────────────────┬────────────────────┬─────────────────────────────┬───────────────────────┐
  │ CI run        │ Network available? │ Download succeeds? │ TrustsMicrosoftDotComRoot   │ Test outcome          │
  ├───────────────┼────────────────────┼────────────────────┼─────────────────────────────┼───────────────────────┤
  │ Build 1342363 │ ✅                 │ ✅                 │ true                        │ PASS (3 elements)     │
  ├───────────────┼────────────────────┼────────────────────┼─────────────────────────────┼───────────────────────┤
  │ Build 1343541 │ ❌ / slow          │ ❌                 │ false                       │ FAIL (2 elements)     │
  └───────────────┴────────────────────┴────────────────────┴─────────────────────────────┴───────────────────────┘

  The test neither sets DisableCertificateDownloads = true nor adds the root cert explicitly, so the result is
  network-dependent. The same applies to BuildInvalidSignatureTwice — if .NET can download "Microsoft Code Signing
  PCA" via MsCertificate's AIA extension, it detects the invalid signature and valid = False. If not, it's a
  PartialChain and valid = True.

  Corrected Root Cause

  The real root cause is that these tests rely on online AIA certificate downloads to complete the chain on Linux
  Bionic, which is inherently non-deterministic. PR #121450 (Nov 2025) switched to a cert hierarchy where the root
  ("DigiCert Global Root G2") is not in the Linux Bionic trust store — so the tests now depend on a network download
  that wasn't needed before.

  Better Fix

  The tests should set chain.ChainPolicy.DisableCertificateDownloads = true to be deterministic, and then either:

   1. Explicitly add the root cert to ExtraStore / CustomTrustStore, or
   2. Adjust assertions for the known-no-root case on Linux Bionic (consistent 2-element chain)
  

[danmoseley]

Note

This analysis was generated by Copilot.

Building on @mangod9's analysis, which correctly identifies the root cause — I agree with the diagnosis and wanted to expand on the fix options.

Root Cause

After PR #121450 switched to a cert hierarchy whose root ("DigiCert Global Root G2") isn't always in the Linux Bionic trust store, SystemTrustCertificateWithCustomRootTrust became network-dependent. The test never adds microsoftDotComRoot to ExtraStore and doesn't set DisableCertificateDownloads = true, so the chain builder relies on AIA downloads to find the root:

AIA download	Chain elements	Result
Succeeds	3 (leaf+intermediate+root)	PASS
Fails (network)	2 (leaf+intermediate)	FAIL

This explains the intermittent 8-hit-per-month pattern in CI.

Fix Recommendation

Of the two approaches @mangod9 outlined, I'd recommend adding the root to ExtraStore and setting DisableCertificateDownloads = true:

• Add microsoftDotComRoot to ExtraStore so the chain builder can discover it locally
• Set DisableCertificateDownloads = true to eliminate the network dependency
• The chain still fails with UntrustedRoot (which is the correct CustomRootTrust behavior being tested)
• Assert 3 elements deterministically on all Unix platforms

This is consistent with how other tests in the same file already work (e.g., BuildChainCustomTrustStore loads all 3 certs into ExtraStore).

The alternative (disable downloads + expect 2 elements) would also work but tests a different scenario (root not discoverable at all) rather than the intended CustomRootTrust validation.

Sketch of the key changes:

using (var microsoftDotComRoot = new X509Certificate2(TestData.MicrosoftDotComRootBytes))
// ...
chain.ChainPolicy.DisableCertificateDownloads = true;
chain.ChainPolicy.ExtraStore.Add(microsoftDotComIssuer);
chain.ChainPolicy.ExtraStore.Add(microsoftDotComRoot);