Deleted three large local disks, got them into a 'FAULTED' state and their names now appear as 'deleted-XXXXXX'

[paudmir]

I was doing some clean up in Dogfood, where I had three ~1.5TB local disks I was using for performance tests.
I tried deleting them through console, but got warning that there had been some error.
When I refreshed the UI, the entries appeared like this:

Tried deleting one of those again, and that retry succeeeded.

There are 2 more in that state here:
https://oxide.sys.rack2.eng.oxide.computer/projects/verdant-df/disks

[david-crespo]

I would be very surprised if this wasn’t an API issue but let’s leave it open for now.

[jmpesp]

Looking into one of the disk delete sagas that failed, we see:

root@oxz_switch0:~# omdb db saga show 6f8351c5-e617-489b-bf6e-8ee0d0d4af59
note: database URL not specified.  Will search DNS.
note: (override with --db-url or OMDB_DB_URL)
note: using DNS from system config (typically /etc/resolv.conf)
note: (if this is not right, use --dns-server to specify an alternate DNS server)
note: using database URL postgresql://root@[fd00:1122:3344:109::3]:32221,[fd00:1122:3344:105::3]:32221,[fd00:1122:3344:10b::3]:32221,[fd00:1122:3344:107::3]:32221,[fd00:1122:3344:108::3]:32221/omicron?sslmode=disable
note: database schema version matches expected (233.0.0)
 id                                   | current_sec                          | time_created             | name        | state 
--------------------------------------+--------------------------------------+--------------------------+-------------+-------
 6f8351c5-e617-489b-bf6e-8ee0d0d4af59 | accf6141-95ce-4d16-be2b-46a40f68b0a2 | 2026-03-06T02:10:23.230Z | disk-delete | Done  

DAG: {"end_node":5,"graph":{"edge_property":"directed","edges":[[0,1,null],[1,2,null],[2,3,null],[4,0,null],[3,5,null]],"node_holes":[],"nodes":[{"Action":{"action_name":"disk_delete.delete_disk_record","label":"DeleteDiskRecord","name":"deleted_disk"}},{"Action":{"action_name":"disk_delete.space_account","label":"SpaceAccount","name":"no_result1"}},{"Action":{"action_name":"disk_delete.delete_local_storage","label":"DeleteLocalStorage","name":"delete_local_storage"}},{"Action":{"action_name":"disk_delete.deallocate_local_storage","label":"DeallocateLocalStorage","name":"deallocate_local_storage"}},{"Start":{"params":{"disk":{"LocalStorage":{"disk":{"block_size":"AdvancedFormat","disk_type":"LocalStorage","identity":{"description":"ML disk with 1500","id":"581e9614-c3c7-4cb9-8afa-b24f09a5620d","name":"ml-disk-1500","time_created":"2026-02-10T20:16:11.719326Z","time_deleted":null,"time_modified":"2026-02-10T20:16:11.719326Z"},"project_id":"290be4ae-3f36-40b0-9fc7-6d49277c9bf4","rcgen":1,"runtime_state":{"attach_instance_id":null,"disk_state":"detached","gen":2,"time_updated":"2026-02-10T20:16:11.814925Z"},"size":1610612736000,"slot":null},"disk_type_local_storage":{"disk_id":"581e9614-c3c7-4cb9-8afa-b24f09a5620d","local_storage_dataset_allocation_id":null,"local_storage_unencrypted_dataset_allocation_id":"5a4fbe95-2bf3-4407-93ef-afe514cd8284","required_dataset_overhead":110100480000},"local_storage_dataset_allocation":{"Unencrypted":{"dataset_size":1720713216000,"id":"5a4fbe95-2bf3-4407-93ef-afe514cd8284","local_storage_unencrypted_dataset_id":"54c15da9-0e5e-4c78-935b-a5cdd8e7a39b","pool_id":"6b9ec5f1-859f-459c-9c06-6a51ba87786f","sled_id":"71def415-55ad-46b4-ba88-3ca55d7fb287","time_created":"2026-02-10T20:16:12.991055Z","time_deleted":null}}}},"project_id":"290be4ae-3f36-40b0-9fc7-6d49277c9bf4","serialized_authn":{"kind":{"Authenticated":[{"actor":{"SiloUser":{"silo_id":"7bd7623a-68ed-4636-8ecb-b59e3b068787","silo_user_id":"9ceeb971-95af-4cbe-8b25-0c9ad57c925b"}},"credential_id":"84bb9266-2c42-4501-b327-c40bbeea7149","device_token_expiration":null},{"mapped_fleet_roles":{"admin":["admin"]}}]}}}}},"End"]},"saga_name":"disk-delete","start_node":4}

              event time | sub saga | node id                               | event type    | data
------------------------ | -------- | ------------------------------------- | ------------- | ---
2026-03-06T02:10:23.239Z |          |                              4: start | started       | 
2026-03-06T02:10:23.243Z |          |                              4: start | succeeded     | 
2026-03-06T02:10:23.246Z |          |     0: disk_delete.delete_disk_record | started       | 
2026-03-06T02:10:23.274Z |          |     0: disk_delete.delete_disk_record | succeeded     | "deleted_disk" => {"block_size":"AdvancedFormat","disk_type":"LocalStorage","identity":{"description":"ML disk with 1500","id":"581e9614-c3c7-4cb9-8afa-b24f09a5620d","name":"ml-disk-1500","time_created":"2026-02-10T20:16:11.719326Z","time_deleted":null,"time_modified":"2026-02-10T20:16:11.719326Z"},"project_id":"290be4ae-3f36-40b0-9fc7-6d49277c9bf4","rcgen":1,"runtime_state":{"attach_instance_id":null,"disk_state":"detached","gen":2,"time_updated":"2026-02-10T20:16:11.814925Z"},"size":1610612736000,"slot":null}
2026-03-06T02:10:23.278Z |          |          1: disk_delete.space_account | started       | 
2026-03-06T02:10:23.295Z |          |          1: disk_delete.space_account | succeeded     | 
2026-03-06T02:10:23.299Z |          |   2: disk_delete.delete_local_storage | started       | 
2026-03-06T02:10:28.226Z |          |   2: disk_delete.delete_local_storage | failed        | "delete_local_storage" => {"ActionFailed":{"source_error":"failed to delete local storage: failed at attempt 1: progenitor API operation failed: Error Response: status: 500 Internal Server Error; headers: {\"content-type\": \"application/json\", \"x-request-id\": \"3e8a9610-50c5-4658-9a33-d96e2c7ce8c3\", \"content-length\": \"124\", \"date\": \"Fri, 06 Mar 2026 02:10:28 GMT\"}; value: Error { error_code: Some(\"Internal\"), message: \"Internal Server Error\", request_id: \"3e8a9610-50c5-4658-9a33-d96e2c7ce8c3\" }"}}
2026-03-06T02:10:28.238Z |          |          1: disk_delete.space_account | undo_started  | 
2026-03-06T02:10:28.259Z |          |          1: disk_delete.space_account | undo_finished | 
2026-03-06T02:10:28.263Z |          |     0: disk_delete.delete_disk_record | undo_started  | 
2026-03-06T02:10:28.275Z |          |     0: disk_delete.delete_disk_record | undo_finished | 
2026-03-06T02:10:28.280Z |          |                              4: start | undo_started  | 
2026-03-06T02:10:28.283Z |          |                              4: start | undo_finished |

Tracking down request 3e8a9610-50c5-4658-9a33-d96e2c7ce8c3 shows that the sled-agent encountered an error during the zfs delete:

02:10:28.225Z INFO SledAgent (dropshot (SledAgent)): request completed
    error_message_external = Internal Server Error
    error_message_internal = Command [/usr/sbin/zfs destroy -r oxp_6b9ec5f1-859f-459c-9c06-6a51ba87786f/local_storage_unencrypted/5a4fbe95-2bf3-4407-93ef-afe514cd8284] executed and failed with status: exit status: 1  stdout:   stderr: cannot destroy 'oxp_6b9ec5f1-859f-459c-9c06-6a51ba87786f/local_storage_unencrypted/5a4fbe95-2bf3-4407-93ef-afe514cd8284': out of space\n
    file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:855
    latency_us = 4910326
    local_addr = [fd00:1122:3344:10b::1]:12345
    method = DELETE
    remote_addr = [fd00:1122:3344:108::66]:34516
    req_id = 3e8a9610-50c5-4658-9a33-d96e2c7ce8c3
    response_code = 500
    uri = /local-storage

We're investigating this on the sled.