Is your feature request related to a problem? Please describe.
I'm always frustrated when I'm thinking of various recovery scenarios, configuration issues and API inconsistencies. We have three components that make up a shard: blobstor, write cache and metabase. We have APIs that allow to set/update IDs in various components independently and we're missing some consistency checks between these components. The primary source of an ID for existing shard is metabase (blobstor is being "updated" to it) which is also wrong given that metabase is just a fancy blobstor index.
Describe the solution you'd like
What we need in the end is:
- sticky IDs, once set they never change unless component is gone completely (drive failure), this means no update APIs, either ID is stored already or it's set once at some open/init stage
- all components that make up a shard must have the same ID
- we should use the same type in APIs dealing with shard IDs (the most effective one)
What should be taken into account is:
- the primary thing shard has is blobstor, it can't function at all without it
- metabase can be lost, but it's easy to restore it (resync)
- write cache is a blobstor of its own, it'd be nice to have some "subtype" stored in its config to separate it from the main blobstor in recovery scenarios
Which to me leads to a design where an ID is generated by blobstor (fstree) and can't be changed there once blobstor is inited. Either it's autogenerated or supplied externally during open (for empty write cache that should refuse to take it if there is a mismatch between stored and passed ID). Then metabase takes this ID and also either accepts as new one (empty DB) or checks against its own stored ID.
Describe alternatives you've considered
Things kinda work as is, but it's not reliable currently.
Additional context
#3849
Is your feature request related to a problem? Please describe.
I'm always frustrated when I'm thinking of various recovery scenarios, configuration issues and API inconsistencies. We have three components that make up a shard: blobstor, write cache and metabase. We have APIs that allow to set/update IDs in various components independently and we're missing some consistency checks between these components. The primary source of an ID for existing shard is metabase (blobstor is being "updated" to it) which is also wrong given that metabase is just a fancy blobstor index.
Describe the solution you'd like
What we need in the end is:
What should be taken into account is:
Which to me leads to a design where an ID is generated by blobstor (fstree) and can't be changed there once blobstor is inited. Either it's autogenerated or supplied externally during open (for empty write cache that should refuse to take it if there is a mismatch between stored and passed ID). Then metabase takes this ID and also either accepts as new one (empty DB) or checks against its own stored ID.
Describe alternatives you've considered
Things kinda work as is, but it's not reliable currently.
Additional context
#3849