Firehose Stores are abstractions sitting on top of Object Storage.
Note:Object Storage is a data storage technique that manages data as objects in opposition to other data storage architectures like hierarchical file systems.
For production deployments outside of cloud providers, StreamingFast recommends Ceph as the distributed storage instead of its compatible Amazon S3 API system.
Firehose uses Firehose-enabled node components that have been set with a special flag to work in catch-up mode to create merged blocks.
Highly-available Merged Blocks
In high-availability Firehose configurations, merged blocks will be created by the Merger component. The Firehose-enabled node component will provide the Merger component with one-block files.
Block Bundles
The Merger component will also collate all of the one-block files into a single bundle of blocks.
One Hundred Blocks Files
Up to one hundred blocks can be contained within a single 100-blocks file.
The 100-blocks files can include multiple versions such as a fork block or a given block number, ensuring continuity through the previous block link.
Blocks Files Consumption & Use
Nearly all components in Firehose rely on or utilize 100-blocks files. The bstream library consumes 100-blocks files for example.
Protocol-specific decoded block objects, like Ethereum, are what circulate amongst all processes that work with executed block data in Firehose.
Important: One-block files contain only one bstream.Block as a serialized protocol buffer.
One-block File Consumption & Use
One-block files are consumed by the Merger component, bundled in executed __ 100-blocks files. The one-block files are then stored to dstore storage and consumed by most of the other Firehose processes.
