LogoLogo
  • Firehose
  • Introduction
    • Firehose Overview
    • Prerequisites
  • Architecture
    • Components
      • Firehose-enabled Node
      • Reader
      • Merger
      • Relayer
      • gRPC Server
      • High Availability
    • Data Flow
    • Data Storage
  • Firehose Setup
    • Overview
    • Ethereum
      • Installation
      • Single-Machine Deployment
      • Reprocessing history
      • Synchronization
    • Injective
      • Single-Machine Deployment
    • NEAR
      • Single-Machine Deployment
    • Solana
      • Single-machine Deployment
    • System Requirements
  • Community Integrations
    • Starknet
      • Networks and nodes
      • Local deployment with Docker
      • Local deployment without Docker
  • Integrate New Chains
    • Benefits
    • Integration overview
    • Design Principles
    • Firehose Acme
  • References
    • Supported Protocols
    • Naming Conventions
    • Schemas
    • Indexing
    • FAQ
  • Release Notes
    • Change logs
      • Nov 8th 2023 Polygon Update
Powered by GitBook
On this page
  • Data Storage in Firehose
  • Data Storage in Detail
  • Data Stores
  • Abstraction Library
  • Production Environments
  • Serialization
  • Merged Blocks Files
  • Merged Blocks in Detail
  • Merged Block Creation
  • Highly-available Merged Blocks
  • Block Bundles
  • One Hundred Blocks Files
  • Blocks Files Consumption & Use
  • One Block Files
  • One Block Files in Detail
  • One-block File Consumption & Use

Was this helpful?

Edit on GitHub
  1. Architecture

Data Storage

StreamingFast Firehose data storage

PreviousData FlowNextOverview

Last updated 2 years ago

Was this helpful?

Data Storage in Firehose

Data Storage in Detail

Data and the locations where it is stored are important facets of Firehose deployment and operation.

Key Firehose data storage topics include , , , , and .

Data Stores

Firehose Stores are abstractions sitting on top of Object Storage.

Note: Object Storage is a data storage technique that manages data as objects in opposition to other data storage architectures like hierarchical file systems.

Abstraction Library

Stores utilize the Firehose to provide support for local file systems, , , , and other Amazon S3 API compatible object storage solutions such as or .

Production Environments

For production deployments outside of cloud providers, StreamingFast recommends as the distributed storage instead of its compatible Amazon S3 API system.

Serialization

Merged Blocks Files

Merged Blocks in Detail

Merged blocks files are also referred to as 100-blocks files, and merged bundles. These terms are all used interchangeably within Firehose.

Merged Block Creation

Highly-available Merged Blocks

Block Bundles

One Hundred Blocks Files

Up to one hundred blocks can be contained within a single 100-blocks file.

The 100-blocks files can include multiple versions such as a fork block or a given block number, ensuring continuity through the previous block link.

Blocks Files Consumption & Use

Nearly all components in Firehose rely on or utilize 100-blocks files. The bstream library consumes 100-blocks files for example.

Protocol-specific decoded block objects, like Ethereum, are what circulate amongst all processes that work with executed block data in Firehose.

One Block Files

One Block Files in Detail

Important: One-block files contain only one bstream.Block as a serialized protocol buffer.

One-block File Consumption & Use

One-block files are consumed by the Merger component, bundled in executed __ 100-blocks files. The one-block files are then stored to dstore storage and consumed by most of the other Firehose processes.

Firehose primarily utilizes for serialization.

Merged blocks are binary files that use the packing format to store a series of , serialized as .

Firehose uses components that have been set with a special flag to work in catch-up mode to create merged blocks.

In Firehose configurations, merged blocks will be created by the component. The component will provide the Merger component with one-block files.

The component will also collate all of the one-block files into a single bundle of blocks.

In configurations, one-block files are transient and ensure the component gathers all visible forks from any components.

Protocol Buffers version 3
dbin
bstream block objects
protocol buffers
Firehose-enabled node
high-availability
Merger
Firehose-enabled node
Merger
high availability
Merger
Firehose-enabled Node
dstore abstraction library
Azure
Google Cloud
Amazon S3
MinIO
Ceph
Ceph
Data Stores
Merged blocks files
serialization
one block files
100-blocks files