File system abstraction [DRAFT]

[cryptoyang]

This document is not complete and can change over time.

File system abstraction

Purpose

Persistent data storage is a common problem that can be solved in many ways. However, each available solution has its own drawbacks, and any of them doesn't meet all our requirements that we can imagine.

Requirements

• Independent of the transport layer.
• Identifies data using predictable id.
• Extensible.
• Control redundancy of files.
• Scalable across physical devices

Layers

The file system abstraction can be represented as a unidirectional(?) layers or graph of cooperating services.

Store

The lowest level of data persistence abstraction. Key-value store for hash-based IDs associated with blob.

Requirements

• Id is predictable from the data which points to. So the same data will always generate the same unique Id.
• Id requirement is implying data immutability. So is not possible to modify existing data without modifying the id.
• The storage interface allows only for basic operations, the domain-specific functions will be delivered on higher layers.

API

typealias Id = String

interface Store<T> {
    suspend operator fun plus(data: T): Id
    operator fun get(id: Id): T
    operator fun minus(id: Id): Boolean
    operator fun contains(id: Id): Boolean
    fun set(): Set<Id>
    fun additions(): Flow<Id>
}

Data

ID

Is a summary result of sha256 and the size of the given data.
id(D) = sha256(D) + size(D); where D is any data

Blob

On this level of abstraction, it can be any kind of data that can be stored on a hard drive.

Graph

The abstraction, responsible for storing and indexing relations between blobs and stories.

Requirements

• Graph is an independent service that knows Store API
• Graph can recognize the special type of blobs called Stories.
• Graph is indexing relations for each recognized story added to the store.
• Graph can serve information about relations between blobs.

API

typealias Type = String

interface Graph {
    fun source(id: Id): Set<Id>
    fun target(id: Id): Set<Id>
    fun type(name: Type): Set<Id>
}

Data

Story

It's a unified kind of blob that can describe another blob.
The story should contain, version id, type name, set of relations plus optional body, the key-value map.

data class Story(
    val ver: Int = 0,
    val type: Type = "",
    val rel: Set<Id> = emptySet(),
    val body: Map<String, Any> = emptyMap()
)

Push service

Gateway to the file system, responsible for handling upcoming messages outside of the file system.
... TODO

[cryptoarashi]

A binary (canonical) ID is a 320-bit value of concat(size, hash), where:

size is a uint64 data length in network byte order.
hash is a sha256 checksum of the data

A text representation of an ID is literal id1 concatenated with zBase32-encoded binary ID with removed leading zeroes (represented as y in zBase32)