instances with read-only boot disk fail to boot

[jordanhendricks]

Read-only disks were integrated as a control plane feature in oxidecomputer/omicron#9731. Today in dogfood I attempted to boot a variant of a debian image we've been using to test RFD 605 work. The instance (as well as future attempts to boot an instance with a RO disk) failed to boot, dropping into the UEFI shell, e.g.:

UEFI Interactive Shell v2.2
EDK II
UEFI v2.70 (EDK II, 0x00010000)
Mapping table
      FS0: Alias(s):HD0p:;BLK3:
          PciRoot(0x0)/Pci(0x10,0x0)/NVMe(0x1,00-00-00-00-00-00-00-00)/HD(15,GPT,6012ACAE-A51D-C14F-BB91-DBBFB6CF0F77,0x2000,0x3E000)
      FS1: Alias(s):F1:;BLK4:
          PciRoot(0x0)/Pci(0x18,0x0)
     BLK0: Alias(s):
          PciRoot(0x0)/Pci(0x10,0x0)/NVMe(0x1,00-00-00-00-00-00-00-00)
     BLK1: Alias(s):
          PciRoot(0x0)/Pci(0x10,0x0)/NVMe(0x1,00-00-00-00-00-00-00-00)/HD(1,GPT,E6842644-BB59-324A-BFF6-F7D01B8B2CB2,0x40000,0x3BFFDF)
     BLK2: Alias(s):
          PciRoot(0x0)/Pci(0x10,0x0)/NVMe(0x1,00-00-00-00-00-00-00-00)/HD(14,GPT,26E893FC-03C9-6049-B821-276C1F0B03A4,0x800,0x1800)

@hawkw and @jmpesp and I spent some time looking at this today. A collection of observations from today:

• booting from a read-only disk worked about a week ago (and eliza tested with this exact image; known good propolis sha is 2aa7f9d0ee84a1c45e821d6444b1d2f0e69b743e)
• it is possible to boot from the same image as a normal (read-write) disk
• james was able to reproduce this failure trivially in the canada region
• the md5sum of the image, as uploaded to the control plane from james' desktop, matches the md5sum of the disk when it is attached as read-write disk to a working instance (implying crucible has not seen corruption of the disk)
• UEFI does not seem to believe there is a file system on the read-only disk (e.g., ls FS0 shows nothing and no files), but there is valid data in the blocks on disk (e.g., seen via dblk blk0 1 in the UEFI shell), and that data seems to match the image file locally
• the presence of a NIC on the instance did not change the behavior (i.e., I tried booting the image as a RO disk both with and without a NIC on the instance, and they failed the same way)
• this image is bootable as a read-only disk on qemu

[hawkw]

There is also a bug when multiple read-only disks are created from the same image/snapshot, because we don't generate new UUIDs in the VCR so Crucible just kicks off the read-only disk that's currently using the volume with a YouAreNoLongerActive when the second readonly volum is activated, because they have the same UUIDs and stuff. This one is entirely due to me being stupid, and I believe @jmpesp has a fix for it.

[hawkw]

I think what i'm going to do is try to write a PHD test that asserts we can boot from a read-only Crucible volume and then bisect every commit since 2aa7f9d to see if any of them break it.

[jmpesp]

Here's the configuration that I've been testing with propolis-standalone (with generic paths):

[main]
name = "jamesvm"
cpus = 24
bootrom = "/path/to/out/propolis-server/OVMF_CODE.fd"
memory = 4096

[block_dev.disk]
type = "file"
path = "/another/path/to/vm-attest-proto_vm-rootfs.raw"
read_only = true
worker_count = 16
block_size = 512

[dev.disk]
driver = "pci-nvme"
block_dev = "disk"
pci-path = "0.16.0"

vm-attest-... is a debian image configured to be entirely read-only, and should boot just fine, but does not.

One interesting test that was performed was changing the device emulation from pci-nvme to pci-virtio-block, and this does work.

There's a subtle difference between the nvme and virtio device emulation: the read-only-ness of a disk is exposed to guests in the virtio case but not in the nvme case. For virtio, it's part of the features of the device:

    fn features(&self) -> u64 {
        let mut feat = VIRTIO_BLK_F_BLK_SIZE;
        feat |= VIRTIO_BLK_F_SEG_MAX;
        feat |= VIRTIO_BLK_F_FLUSH;

        let info = self.block_attach.info().unwrap_or_else(Default::default);
        if info.read_only {
            feat |= VIRTIO_BLK_F_RO;
        }
        if info.supports_discard {
            feat |= VIRTIO_BLK_F_DISCARD;
        }
        feat
    }

whereas there is nothing the nvme case that guests can query. Looking at the nvme specification for the version that propolis emulates, there isn't even bits for that information that we can set. NVMe version 1.4, figure 198 does have a section where the controller can indicate that "all of the media has been placed in read only mode" through SMART information but this is a critical warning instead of a property of the device.

I don't have any experience in edk2 code, but for virtio there is code that sets some ReadOnly property based on virtio features:

  //
  // Populate the exported interface's attributes; see UEFI spec v2.4, 12.9 EFI
  // Block I/O Protocol.
  //
  Dev->BlockIo.Revision              = 0;
  Dev->BlockIo.Media                 = &Dev->BlockIoMedia;
  Dev->BlockIo.Reset                 = &VirtioBlkReset;
  Dev->BlockIo.ReadBlocks            = &VirtioBlkReadBlocks;
  Dev->BlockIo.WriteBlocks           = &VirtioBlkWriteBlocks;
  Dev->BlockIo.FlushBlocks           = &VirtioBlkFlushBlocks;
  Dev->BlockIoMedia.MediaId          = 0;
  Dev->BlockIoMedia.RemovableMedia   = FALSE;
  Dev->BlockIoMedia.MediaPresent     = TRUE;
  Dev->BlockIoMedia.LogicalPartition = FALSE;
  Dev->BlockIoMedia.ReadOnly         = (BOOLEAN) ((Features & VIRTIO_BLK_F_RO) != 0);
  Dev->BlockIoMedia.WriteCaching     = (BOOLEAN) ((Features & VIRTIO_BLK_F_FLUSH) != 0);
  Dev->BlockIoMedia.BlockSize        = BlockSize;
  Dev->BlockIoMedia.IoAlign          = 0;
  Dev->BlockIoMedia.LastBlock        = DivU64x32 (NumSectors,
                                         BlockSize / 512) - 1;

For nvme, ReadOnly is hard-coded as false.

If the section of propolis code that sets VIRTIO_BLK_F_RO is commented out, then the vm-attest image on a read-only virtio device no longer boots and presents the same symptoms that attempting to boot with nvme shows. Internally OVMF is probably checking ReadOnly and making decisions based on that, but for setting read-only to false to cause FS0 and kin to appear like there aren't any files there, even though data is present when dumping with dblk, seems like an error in our fork.

[jmpesp]

For read-only devices receiving writes, the standard thing to do is to return block::Result::ReadOnly, like:

            if self.info.read_only && req.op.is_write() {
                dreq.complete(block::Result::ReadOnly);
                continue;
            }

For nvme, this is returned to the guest as a specific "you tried to write to a read only range":

            block::Result::ReadOnly => Completion::specific_err(
                bits::StatusCodeType::CmdSpecific,
                bits::STS_WRITE_READ_ONLY_RANGE,
            ),

OVMF does issue some writes, and receives this error type, but it doesn't appear to do any special handling of nvme errors. At https://github.com/oxidecomputer/edk2/blob/907a5fd1763ce5ddd74001261e5b52cd200a25f9/MdeModulePkg/Bus/Pci/NvmExpressDxe/NvmExpressBlockIo.c#L83-L84:

  @retval EFI_SUCCESS            Datum are written into the buffer.
  @retval Others                 Fail to write all the datum.