This section is about what is stored where in a Firehose deployment.
Stores used by the Firehose are abstractions on top of Object Storage.
NoteFor production deployments outside of cloud providers, we recommend ceph over its compatible S3 API as the distributed storage system.
Firehose stack uses Protocol Buffers version 3 for serialization, pretty much throughout.
Merged Blocks Files
100-blocks files, or merged blocks files, or merged bundles, these are all used interchangeably.
They are produced by
Extractors, in catch-up mode (set as such with certain flags), or by the
Merger in an HA setup.
In the latter case, the
Extractor contributes one-block files to the
Merger instead, and the
all of those in a single bundle.
100-blocks files can contain more than 100 blocks (because they can include multiple versions
(e.g. fork block) of a given block number), ensuring continuity through the previous block link.
The protocol specific decoded block objects (for Ethereum as an example) are what circulate amongst all processes that work with executed block data.
One Block Files
These are transient files, destined to ensure that the
Merger gathers all visible forks from
Extractor instances, in an HA setup.
They contain one
bstream.Block, as serialized Protobuf (see links above).
Merger will consume them, bundle them in executed blocks files (100-blocks files) and store
dstore storage, for consumption by most other processes.