Skip to content

CimFS snapshotter causes BSOD on Windows Server 2022 (build 20348.4773) under concurrent container workload #633

@doctorpangloss

Description

@doctorpangloss

Describe the bug

Using the CimFS snapshotter (cimfs.sys) with containerd 2.2.1 on Windows Server 2022 causes repeated BSODs when pulling and starting multiple container images concurrently. Two distinct bugchecks occurred within 13 minutes:

  1. 0x00000044 (MULTIPLE_IRP_COMPLETE_REQUESTS)
  2. 0x000001ca (SYNTHETIC_WATCHDOG_TIMEOUT)

Also filed as microsoft/hcsshim#2625 since the CimFS snapshotter/differ implementation lives in hcsshim.

To Reproduce

  1. Install containerd v2.2.1 on Windows Server 2022 (build 20348.4773).
  2. Configure containerd to use the cimfs snapshotter:
    version = 3
    
    [plugins.'io.containerd.cri.v1.images']
      snapshotter = 'cimfs'
    
    [plugins.'io.containerd.transfer.v1.local']
      [[plugins.'io.containerd.transfer.v1.local'.unpack_config]]
        platform = 'windows/amd64'
        snapshotter = 'cimfs'
        differ = 'cimfs'
  3. Start containerd and kubelet.
  4. Pull ~15-20 Windows container images concurrently (Kubernetes daemonsets + workloads).
  5. System BSODs within minutes.

Expected behavior

The CimFS kernel driver should handle concurrent image layer mounts without crashing.

Configuration:

  • Edition: Windows Server 2022 Datacenter (21H2)
  • OS Build: 20348.4773 (KB5075906, February 2026 cumulative update)
  • cimfs.sys driver version: 10.0.20348.3804
  • Container engine: containerd v2.2.1
  • Orchestrator: Kubernetes v1.34.2
  • Hardware: Bare metal, AMD Ryzen 9 7950X, 64GB RAM
  • Base images: mcr.microsoft.com/windows/servercore:ltsc2022 and mcr.microsoft.com/windows/nanoserver:ltsc2022

Additional context

Bugcheck details:

Bugcheck 1 (12:37 PST):
  Code: 0x00000044 (MULTIPLE_IRP_COMPLETE_REQUESTS)
  Parameters: (0xffffb1075846f4c0, 0x000000000000125a, 0x0000000000000000, 0x0000000000000000)

Bugcheck 2 (12:50 PST):
  Code: 0x000001ca (SYNTHETIC_WATCHDOG_TIMEOUT)
  Parameters: (0x0000000019628f83, 0x00000001bf799269, 0x00000001bf7ca01d, 0x0000000000000008)

The system was stable for over 3 years with the default windows (WCIFS) snapshotter. BSODs started immediately after switching to CimFS and stopped after reverting. The crashes appear to be in the cimfs.sys kernel driver under concurrent I/O load.

Minidumps are available if needed.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions