Split partitions into blocks/chunks

[albe]

For some use-cases it might be preferable to have partitions split into multiple ordered blocks.
The most important use-case is archiving (or even deletion) of old data. Right now, this would be achieved by creating a new partition (=stream), write a "consolidated event" (or snapshot) and then keep working on that new stream.
Another use-case is scanning a partition for the latest document without an index, e.g. during auto-repair (see #107). If the partition is stored in a single file, the whole partition needs to be scanned because only forward reading is supported by the file format. If the partition is chunked though, only the last chunk needs to be scanned, which is likely orders faster.
It would also potentially help in replication logic, since only a small file would need "hot" replication.

A simple solution would be to just append a starting partition offset to the filename (e.g. partition-foo.0, partition-foo.4096, etc.) and then start a new file whenever either a configured amount of documents or chunk size is reached. Reads can then easily find the correct file to read from by searching for the chunk that has filename-offset < position < filename-offset + filesize (- header) - the list of chunks could also be pre-scanned and indexed on opening the partition.
The drawback is that the file handle would need to be switched multiple times, so there is more file close/open involved, especially when scanning the whole partition (=projection rebuild).

This could be made fully b/c if a missing filename offset is interpreted as 0. Potentially old chunks could then even be merged together to reduce the amount of files per partition again, though an evergrowing partition (stream) is a bad pattern.
Also the chunking should be easily turned off with the configuration.

{
  "chunkSize": 4096,
  "chunkDocuments": 100
}

=> start a new chunk every 4kb or 100 documents, whatever happens first.

{
   "chunkSize": 0,
   "chunkDocuments": 0
}

=> disable chunking (=current behaviour)

[albe]

The amount of file handle switches could be greatly reduced by a logic, that only keeps a fixed amount of (gradually smaller) files per partition. E.g. the first partition chunk (0) will always keep growing, the second chunk (1) is 4MB, the third (2) 2MB and the most current one (3) at most 1MB. Once the current chunk is full, chunk (1) will be appended onto (0), (2) becomes the new (1) and gets (3) appended. The new "current" chunk is now (2), until it reaches 1MB, which will merge it into (1).

The most simple version of that logic is having only exactly two chunks per partition. A "hot" and a "cold" one. Size of the "hot" chunk is configurable like above and will hence start at 0 once it is full and is merged into "cold". A 0 size "hot" chunk also means the last commit finished.

The question then becomes when and how often to actually merge chunks, because that is a disruptive operation (can not read from the chunk to be merged during the time it is appended to the main chunk - there's always a small window where either an event exists in two chunks and could be read twice or reading is blocked).

[albe]

Either one of two problems needs to be solved to make this viable:

• deal with an ever growing amount of files
• deal with the inconsistency of reading during merge

the former could be slightly alleviated by having a folder structure for each storage/partition/, which only puts a limit on the amount of files per partition. A clever sub-folder structure would make this even more scalable, at the cost of making watching a partition more costly.

[albe]

A good solution that allows a solid chunking strategy without lots of complexity and a costly merge job that needs to run asynchronously but still block writes, would be to allow for this:

• provide a function that maps partitionId+sequenceNumber+fileSize to a file subpath, e.g. /ab/cd
• by default the function is a noop which effectively evaluates to empty string, meaning the base directory of the storage/partition
• when writing to a partition the destination filename is resolved through that function
• if file name changed from last, flush write buffer (writing to the previous file still) and append a link to the next file path - in the new file prepend a link to the previous file
• when reading through the partition (backwards) resolve the file name for the current lowest (highest) sequence number in that partition from the index (first or last entry), then scan the file as before and jump to the next file when reaching start/end

However: this means we can no longer use binary search over a single partition easily, unless we do a scan of all partition files on start up and index their positions. But, if we do index the files with start (and end) sequence number, finding a specific sequence doc will end up scanning only a single chunk file.

So the trade off is:

• slower startup due to more file scanning (which becomes worse the more files/partitions there are) and a slightly lower sequential write performance
• quicker random search without an index (i.e. during repair, though that's already merely a log(n) operation as we can already scan backwards and hence do binary search on the partition file) and the option to improve archiving/backup with cold files

Looking at this, I don't think the trade off is (currently) worth it or specifically working towards the goal of this event store implementation. A kind of chunking can already be achieved with the partitioner function. So for very specific needs there is (limited) options to achieve something like that already.

What would be interesting though is allowing the partitioner to specify paths, so that single partition files are spread across a folder structure which allows the system to scale beyond some file system boundaries.