# Firehose StreamingFast Firehose documentation ## Welcome to Firehose Docs Firehose is a blockchain data streaming technology developed by StreamingFast working with [The Graph Foundation](https://thegraph.com/). The product provides previously unseen capabilities and speeds for indexing such data using a files-based and streaming-first approach. This documentation serves as the main resource for operators on understanding and operating a Firehose Blockchain Indexing stack. ### Where to Start [Firehose Overview](/introduction/firehose-overview) is a great place to begin learning about the overall scope, vision, and capabilities of Firehose. The [Concepts & Architecture](/firehose/architecture/components) provides an in-depth look at the decisions that drove the development team's efforts while designing Firehose. Full source code is available in the [official Firehose Core Github repository](https://github.com/streamingfast/firehose-core). ### Community ***Need help? Reach out!*** Firehose is an open-source community effort, so feel free to suggest new topics, report issues, and provide feedback wherever it could be helpful. Please contribute through GitHub [pull requests](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/about-pull-requests). * [StreamingFast Discord](https://discord.gg/jZwqxJAvRs) * [StreamingFast on Twitter](https://twitter.com/streamingfastio) * [StreamingFast on YouTube](https://www.youtube.com/c/streamingfast) ### Contributing Please first refer to the general [StreamingFast contribution guide](https://github.com/streamingfast/streamingfast/blob/master/CONTRIBUTING.md) for additional information. ### License [Apache 2.0](https://github.com/streamingfast/firehose-docs/blob/master/LICENSE/README.md) # Firehose Overview StreamingFast Firehose overview ## Purpose ### What is it? Firehose is a core component of StreamingFast’s suite of open-source blockchain technologies. ### Streaming-first & Files-based Firehose provides a files-based and streaming-first approach to processing blockchain data. ### Blockchain Data Extraction Firehose is responsible for extracting data from blockchain nodes in a highly efficient manner. Firehose also consumes, processes, and streams blockchain data to consumers of nodes running Firehose-enabled, instrumented blockchain client software. ### Speed Increases Across Blockchains Firehose makes paramount improvements in the speed and performance of data availability for *any blockchain*. The full [StreamingFast software suite](https://github.com/streamingfast) enables low-latency processing of real-time blockchain data in the form of binary data streams. [Substreams](https://substreams.streamingfast.io/) is another application in the suite that works with Firehose to execute massive operations on historical blockchain data, in an *extremely parallelized manner*. ## Benefits Firehose is built using a component-based design. Data extraction is made possible through the family of Firehose components. The Firehose components include Firehose-enabled Blockchain Node, Reader, Merger, Relayer, Firehose gRPC Server, and Substreams. ### Substreams Integration Firehose includes [Substreams](https://substreams.streamingfast.io/), a high-performance parallel data transformation engine. Running Substreams alongside Firehose provides: * **Massively Parallel Processing**: Execute data transformations across historical blockchain data in parallel, achieving processing speeds previously thought impossible. * **Custom Data Pipelines**: Enable users to define their own transformation logic in Rust/WASM, filtering and aggregating data exactly as needed. * **Ecosystem Compatibility**: Immediate support for dozens of data sinks including PostgreSQL, MongoDB, Kafka, ClickHouse, BigQuery, and more. * **Shared Infrastructure**: Substreams reuses the same block storage and data sources as Firehose, adding powerful capabilities with minimal additional overhead. * **Developer Community**: Attract developers who already know Substreams and are ready to build on your infrastructure. ### Cursors The Firehose cursor points to a specific position in the stream of events emitted by ForkDB and the blockchain itself. The ForkDB cursor contains information that is required to reconstruct an equivalent forked or canonical instance. Consumer requests for historical blocks are fetched from persistent Firehose storage. The historical blocks are passed inside a ForkDB and sent with a cursor uniquely identifying the block and its position in the blockchain. ### Low Latency Racing Speed Placing multiple Reader components side by side, and fronted by one or more Relayers, allows for highly available setups. A Relayer connected to multiple Readers will deduplicate incoming streams, and push the first block downstream. Two Reader components will even race to push the data out first. Firehose is designed to leverage this racing Reader feature to the benefit of the end-user by producing the lowest latency possible. ## Motivation ### Why does it exist? Firehose was created to increase the speed and performance of blockchain data extraction from problems encountered in deployed, real-world applications. ### Firehose Prevents Downtime Companies experienced up to three week-long periods of downtime due to the reprocessing of blockchain data in a linear fashion. Firehose was designed for highly efficient parallelized node data processing, at a massively large scale, circumventing these unwanted and problematic downtimes. ### Unrivaled Blockchain Data Processing Speeds Firehose was designed to process blockchain at speeds that were previously unseen and *thought to be impossible*. ### Resolving Slow JSON-RPC Responses Another factor that heavily contributed to the design of Firehose is the brittleness and slow response times of, often inconsistent, JSON-RPC systems. ## Capabilities ### How it works The Firehose instrumentation service is added to a node for efficient capture and simple storage of blockchain data. ### Data Extraction, Storage, & Access Firehose extracts, transforms and saves blockchain data in a highly performant file-based strategy. Blockchain developers can then access data extracted by Firehose through binary data streams. ### Firehose Single Source of Truth (SSOT) Firehose provides a single source of truth for developers looking to utilize blockchain data for their blockchain application development efforts. ### The Graph & Firehose Firehose is intended to stand as a replacement for The Graph’s original blockchain data extraction layer. [The Graph](https://thegraph.com/) is an indexing protocol used for the organization of blockchain data. ### Technical Overview To get started with Firehose, the first step is to learn about its core concepts and technical architecture, beginning with the [component family](https://firehose.streamingfast.io/concepts-and-architeceture/components). ### Existing Firehose Users Experienced node operators can get up and running by using one of the pre-instrumented blockchain node codebases provided by StreamingFast. Look in the [Firehose Setup](https://firehose.streamingfast.io/firehose-setup/setup) section of the Firehose documentation for further information. ### Custom Firehose Setups & New Chains Lastly, developers can learn how to implement custom Firehose nodes in the [New Blockchains](https://firehose.streamingfast.io/integrate-new-chains/new-blockchains) section of the Firehose documentation. # Prerequisites StreamingFast Firehose prerequisites We expect Software Developers and Systems Administrators using Firehose to have good knowledge of standard tools like [shell scripting](https://en.wikipedia.org/wiki/Shell_script) and [command line interface](https://en.wikipedia.org/wiki/Command-line_interface). ## Technologies Used * [Golang](https://go.dev/) * [Google Protocol Buffers](https://developers.google.com/protocol-buffers) * [gRPC](https://grpc.io/) * [Blockchain](https://en.wikipedia.org/wiki/Blockchain) technologies and terminology ## Requirements Full Firehose spins up a full blockchain node (an [Ethereum full node](https://ethereum.org/en/run-a-node/) on Ethereum, a [NEAR full node](https://near-nodes.io/rpc) on NEAR, etc.) Each blockchain has specific hardware and internet connection requirements. Refer to the specific vendor's documentation for the target blockchain. # Architecture Firehose architecture overview and core concepts ## Overview Firehose is a distributed system designed to extract, process, and serve blockchain data at scale. This section covers the core architectural concepts that apply to all blockchain implementations. ## Core Principles ### Chain-Agnostic Design * **90% Universal**: Core components work across all blockchains * **10% Chain-Specific**: Only reader nodes differ between chains * **Consistent Interface**: Same gRPC API regardless of blockchain ### Scalable Architecture * **Horizontal Scaling**: Add more components as needed * **Component Isolation**: Each service can be scaled independently * **Storage Flexibility**: Support for local, cloud, and distributed storage ### Real-Time Processing * **Live Streaming**: Sub-second latency for new blocks * **Historical Access**: Efficient querying of past data * **Fault Tolerance**: Automatic recovery from failures ## System Components The Firehose system consists of several key components that work together to provide a complete blockchain data pipeline: ### [Components Overview](/firehose/architecture/components) Detailed information about each component in the Firehose stack: * [**Reader Node**](/firehose/architecture/components/reader) - Wraps blockchain nodes and extracts block data * [**Merger**](/firehose/architecture/components/merger) - Combines individual blocks into larger files * [**Relayer**](/firehose/architecture/components/relayer) - Provides real-time streaming and high availability * [**Firehose**](/firehose/architecture/components/firehose) - Serves the Firehose gRPC API to clients * [**Substreams**](/firehose/architecture/components/substreams) - High-performance parallel data transformation engine * [**High Availability**](/firehose/architecture/components/high-availability) - Redundancy and failover strategies ### [Data Flow](/firehose/architecture/data-flow) Understanding how data moves through the Firehose system from blockchain nodes to client applications. ### [Data Storage](/firehose/architecture/data-storage) Storage patterns, formats, and strategies used by Firehose for different types of blockchain data. ## Deployment Patterns ### Single-Machine Deployment All components running on one machine for development or small-scale use: ``` ┌─────────────────────────────────────────┐ │ Single Machine │ ├─────────────────────────────────────────┤ │ Reader Process │ Firehose Stack │ │ ┌─────────────┐ │ ┌──────────────┐ │ │ │ Node │ │ │ Reader │ │ │ │ (subprocess)│───┼──│ Merger │ │ │ │ │ │ │ Relayer │ │ │ └─────────────┘ │ │ Firehose & │ │ │ │ │ Substreams │ │ │ │ └──────────────┘ │ └─────────────────────────────────────────┘ ``` {% hint style="info" %} The blockchain node runs as a subprocess of the Reader component, which manages the node's lifecycle and extracts block data. {% endhint %} ### Distributed Deployment Components spread across multiple machines for production scale: ``` ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ │ Reader │ │ Firehose │ │ Storage & │ │ Processes │ │ Processing │ │ Serving │ ├─────────────────┤ ├─────────────────┤ ├─────────────────┤ │ ┌─────────────┐ │ │ ┌─────────────┐ │ │ ┌─────────────┐ │ │ │Node1(subproc│ │ │ │ Reader 1 │ │ │ │ Storage │ │ │ │Node2(subproc│─┼────┼─│ Reader 2 │─┼────┼─│ (Cloud) │ │ │ │Node3(subproc│ │ │ │ Merger │ │ │ │ │ │ │ └─────────────┘ │ │ │ Relayer │ │ │ └─────────────┘ │ │ │ │ └─────────────┘ │ │ ┌─────────────┐ │ │ │ │ │ │ │ Firehose & │ │ │ │ │ │ │ │ Substreams │ │ │ │ │ │ │ │ (via gRPC) │ │ │ │ │ │ │ └─────────────┘ │ └─────────────────┘ └─────────────────┘ └─────────────────┘ ``` ## Key Features ### Universal Block Format * Consistent protobuf schemas across chains * Rich metadata and transaction details * Efficient binary serialization ### Streaming API * gRPC-based streaming interface * Real-time and historical data access * Fork-aware streaming with automatic reorg handling * Cursor-based resumption for reliable data delivery ### Storage Efficiency * Compressed block files * Incremental merging strategy * Cloud storage integration ### Operational Excellence * Comprehensive metrics and monitoring * Automated recovery mechanisms * Horizontal scaling capabilities ## Next Steps * [**Components**](/firehose/architecture/components): Learn about individual system components * [**Data Flow**](/firehose/architecture/data-flow): Understand how data moves through the system * [**Data Storage**](/firehose/architecture/data-storage): Explore storage patterns and formats * [**CLI Reference**](https://github.com/streamingfast/firehose-docs/blob/master/core/cli-reference.md): Learn how to operate Firehose * [**Deployment Guide**](https://github.com/streamingfast/firehose-docs/blob/master/core/deployment-guide.md): Deploy Firehose in production # Components StreamingFast Firehose components ## Firehose Component Family The Firehose system is comprised of several key components that work together to provide your desired functionality. * [Reader Node](/firehose/architecture/components/reader) * [Merger](/firehose/architecture/components/merger) * [Relayer](/firehose/architecture/components/relayer) * [Firehose](/firehose/architecture/components/firehose) * [Substreams](/firehose/architecture/components/substreams) ## Component Relationships The Firehose components work together in symphony to provide blockchain data from configured and instrumented nodes to consumers through the [Firehose](/firehose/architecture/components/firehose) component. {% hint style="warning" %} **Important***: Understanding the Firehose components individually is helpful for fully comprehending the overall system and will aid with setup and operation.* {% endhint %} # Reader Node StreamingFast Firehose Reader Node component The Reader Node is the foundational data extraction component of the Firehose stack. It wraps and manages a blockchain data source binary, reading block data from its standard output and producing one-block files for the rest of the Firehose pipeline. ## How Reader Node Works The Reader Node operates by spawning a **subprocess** and reading **Firehose Protocol** logs from the subprocess's `stdout`. This design keeps Firehose completely decoupled from the underlying blockchain node - Firehose doesn't modify, access, or interact with any other aspect of the node's operation (database, network, RPC, etc.). ``` ┌─────────────────────────────────────────────────────────────┐ │ Reader Node │ │ ┌───────────────────────────────────────────────────────┐ │ │ │ Subprocess Manager │ │ │ │ (spawns, monitors, restarts underlying process) │ │ │ └───────────────────────────────────────────────────────┘ │ │ │ │ │ │ spawns │ │ ▼ │ │ ┌───────────────────────────────────────────────────────┐ │ │ │ Underlying Binary (subprocess) │ │ │ │ │ │ │ │ • Firehose-enabled node (geth-firehose, etc.) │ │ │ │ • RPC poller binary │ │ │ │ • Any binary emitting Firehose Protocol logs │ │ │ └───────────────────────────────────────────────────────┘ │ │ │ │ │ │ stdout (Firehose Protocol logs) │ │ ▼ │ │ ┌───────────────────────────────────────────────────────┐ │ │ │ Block Parser & Writer │ │ │ │ • Parses Firehose Protocol messages │ │ │ │ • Produces one-block files │ │ │ │ • Streams to Relayer via gRPC │ │ │ └───────────────────────────────────────────────────────┘ │ │ │ │ │ ┌───────────┴───────────┐ │ │ ▼ ▼ │ │ One-Block Files gRPC Stream │ │ (Object Storage) (to Relayer) │ └─────────────────────────────────────────────────────────────┘ ``` ### Data Sources The Reader Node can work with different types of underlying binaries: | Source Type | Description | Example | | ------------------------- | ---------------------------------------------------------------------------------------- | ------------------------------------------------------ | | **Firehose-enabled Node** | A blockchain node instrumented to emit Firehose Protocol logs while syncing | `geth` with Firehose patches, `nearcore` with Firehose | | **RPC Poller** | A binary that polls an existing RPC endpoint and converts responses to Firehose Protocol | `fireeth tools poller` | | **Custom Binary** | Any binary that outputs valid Firehose Protocol logs | Custom chain implementations | ### Firehose Protocol The underlying binary must emit specially formatted log lines to `stdout`. The protocol consists of two message types: **FIRE INIT** - Sent once at startup to declare protocol version and block type: ``` FIRE INIT ``` Example: `FIRE INIT 3.0 sf.ethereum.type.v2.Block` **FIRE BLOCK** - Sent for each block with metadata and base64-encoded payload: ``` FIRE BLOCK ``` {% hint style="warning" %} **Protocol Version**: Only versions `3.0` and `3.1` are supported. Version `1.0` is a legacy protocol that is no longer supported by current Firehose releases. {% endhint %} {% hint style="info" %} The Reader Node only reads from `stdout`. It does not interact with the node's database, network layer, RPC interface, or any other component. This isolation ensures Firehose has zero impact on node operation. {% endhint %} ## Reader Node Variants Firehose provides three reader modes for different use cases: ### reader-node (Standard) The most common mode. The Reader Node spawns and manages the underlying binary as a subprocess. ```bash firecore start reader-node \ --reader-node-path="geth" \ --reader-node-arguments="--vmtrace=firehose --datadir={node-data-dir}" ``` ### reader-node-stdin Reads from an already-running process via stdin pipe. Useful when you need to manage the node process separately. ```bash geth --vmtrace=firehose | firecore start reader-node-stdin ``` ### reader-node-firehose Connects to an existing Firehose endpoint and re-emits blocks as one-block files. Useful for creating local copies of remote Firehose data. ```bash firecore start reader-node-firehose \ --reader-node-firehose-endpoint="mainnet.eth.streamingfast.io:443" ``` ## Output: One-Block Files The Reader Node produces **one-block files** - individual files containing a single block's data in Protocol Buffer format. These files: * Are named with the block number and hash for uniqueness * Enable parallel processing by the Merger * Capture all forks seen by this Reader * Are pruned after being merged (by the Merger) When running multiple Reader Nodes writing to the same storage, use `--reader-node-one-block-suffix` to give each instance a unique identifier and prevent write conflicts. ## High Availability Multiple Reader Nodes can run simultaneously for high availability: * Each Reader connects to different network peers, potentially seeing different forks * All Readers write to the same one-block storage (with unique suffixes) * The Merger consolidates all blocks, including forks from any Reader * Readers race to push data to the Relayer, minimizing latency See [High Availability](/firehose/architecture/components/high-availability) for detailed deployment patterns. ## Underlying Node Requirements When using a Firehose-enabled blockchain node, the node only needs to: * Execute transactions in consensus order * Emit Firehose Protocol logs to stdout The node does **not** need: * Archive mode * JSON-RPC service * Indexed data or query capabilities * Any special storage configuration This minimal configuration reduces resource requirements and operational complexity. ## Configuration Reference For complete configuration options, flags, and the Management API reference, see [Reader Node CLI Reference](/references/cli-reference/reader-node). # Merger StreamingFast Firehose Merger component The Merger is responsible for consolidating individual block files into efficient batched bundles. It reads one-block files produced by Reader Nodes and creates merged 100-block files that serve as the primary historical data source for the Firehose pipeline. ## How Merger Works The Merger continuously polls the one-block storage for new files, accumulates them into bundles, and writes merged files to persistent storage. It also handles fork preservation and cleanup of old data. ``` ┌─────────────────────────────────────────────────────────────────────────┐ │ Merger Component │ │ │ │ ┌───────────────────────────────────────────────────────────────────┐ │ │ │ One-Block Storage Poller │ │ │ │ (continuously polls for new one-block files) │ │ │ └───────────────────────────────────────────────────────────────────┘ │ │ │ │ │ ▼ │ │ ┌───────────────────────────────────────────────────────────────────┐ │ │ │ Bundler │ │ │ │ • Accumulates blocks until bundle boundary (100 blocks) │ │ │ │ • Validates sequential block ordering │ │ │ │ • Tracks forked blocks within current range │ │ │ └───────────────────────────────────────────────────────────────────┘ │ │ │ │ │ ┌───────────────┼───────────────┐ │ │ ▼ ▼ ▼ │ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │ │ Merged │ │ Forked │ │ One-Block │ │ │ │ Blocks │ │ Blocks │ │ Pruner │ │ │ │ Storage │ │ Storage │ │ (cleanup) │ │ │ └─────────────┘ └─────────────┘ └─────────────┘ │ └─────────────────────────────────────────────────────────────────────────┘ ``` ### Bundling Process The Merger accumulates blocks in memory until it has enough to create a complete bundle: 1. **Polling**: Continuously checks one-block storage for new files 2. **Validation**: Ensures blocks arrive in sequential order, starting at bundle boundaries 3. **Accumulation**: Collects blocks until reaching the bundle size (100 blocks) 4. **Merge & Store**: Writes the accumulated blocks as a single merged file 5. **Advance**: Moves to the next bundle boundary and repeats {% hint style="info" %} The Merger retains the last block from each bundle as a "bootstrap block" - this facilitates connection to downstream systems and cursor resolution. {% endhint %} ### Fork Handling The Merger preserves all forks encountered by Reader Nodes: * **Fork Detection**: Identifies blocks on non-canonical branches within the current bundle range * **Fork Storage**: Moves forked blocks to dedicated fork storage for cursor resolution * **Fork Pruning**: Cleans up old forked blocks beyond the configured pruning distance This fork preservation enables the Firehose component to resume streaming from any cursor, even if that cursor points to a block on a fork that was later abandoned. ### Retry Logic When encountering failures (storage errors, network issues), the Merger implements automatic retry: * **12 retry attempts** with **5-second intervals** between attempts * Handles temporary storage unavailability gracefully * Logs warnings for holes in merged files that can be recovered ## Bundle Size The Merger creates bundles of exactly **100 blocks** each. This fixed size provides: * Predictable file sizes for storage planning * Efficient batch access for historical queries * Consistent compression ratios ## Pruning Operations Two concurrent pruning operations run in the background: 1. **One-Block Pruner**: Removes source files after they've been merged and are beyond the pruning distance from the Last Irreversible Block (LIB) 2. **Forked Blocks Pruner**: Removes forked block files that are older than the configured pruning distance (default 50,000 blocks) {% hint style="warning" %} The pruning distance should be set large enough to handle chain reorganizations. The default of 50,000 blocks is conservative and suitable for most chains. {% endhint %} ## Storage Access ### One-Block Storage (Input) * **Access**: Read (to fetch blocks) and Delete (to prune) * **Volume**: Temporary - files are deleted after merging * **Naming**: Files include block number and hash for uniqueness ### Merged Blocks Storage (Output) * **Access**: Write (to store bundles) and Read (to detect gaps) * **Volume**: Permanent - the primary historical data store * **Format**: Compressed 100-block bundles ### Forked Blocks Storage * **Access**: Write (to preserve forks) and Delete (to prune) * **Volume**: Moderate - pruned regularly, retains recent forks only * **Purpose**: Enables cursor resolution on forked blocks ## Multiple Readers When running multiple Reader Nodes, the Merger consolidates blocks from all sources: * Each Reader may see different forks depending on network peers * The Merger captures all forks from all Readers * Forked blocks from any Reader are preserved for cursor resolution This architecture ensures no block data is lost, even when different Readers observe different blockchain states. ## Configuration Reference For complete configuration options and flags, see [Merger CLI Reference](/references/cli-reference/merger). # Relayer StreamingFast Firehose Relayer component The Relayer is the live block distribution hub of the Firehose stack. It connects to one or more Reader Nodes, consolidates their block streams, and provides a unified, deduplicated live block feed to downstream consumers including the Firehose and Substreams components. ## How Relayer Works The Relayer creates a multiplexed connection to multiple block sources, manages their health and latency, and exposes a single gRPC endpoint that other components use to receive live blocks. ``` ┌─────────────────────────────────────────────────────────────────────────┐ │ Relayer Component │ │ │ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │ │ Reader 1 │ │ Reader 2 │ │ Reader N │ │ │ │ (source) │ │ (source) │ │ (source) │ │ │ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ │ │ │ │ │ │ │ └────────────────┼────────────────┘ │ │ ▼ │ │ ┌───────────────────────────────────────────────────────────────────┐ │ │ │ Multiplexed Source │ │ │ │ • Connects to multiple Reader endpoints │ │ │ │ • Races sources for fastest block delivery │ │ │ │ • Filters lagging sources based on latency threshold │ │ │ └───────────────────────────────────────────────────────────────────┘ │ │ │ │ │ ▼ │ │ ┌───────────────────────────────────────────────────────────────────┐ │ │ │ ForkableHub │ │ │ │ • Maintains fork-aware block graph │ │ │ │ • Tracks head block and time drift │ │ │ │ • Signals readiness when synchronized │ │ │ └───────────────────────────────────────────────────────────────────┘ │ │ │ │ │ ▼ │ │ ┌───────────────────────────────────────────────────────────────────┐ │ │ │ gRPC BlockStream Server │ │ │ │ • Streams live blocks to Firehose, Substreams, etc. │ │ │ │ • Same interface as Reader Node (BlockStream::Blocks) │ │ │ └───────────────────────────────────────────────────────────────────┘ │ │ │ │ │ ┌───────────┴───────────┐ │ │ ▼ ▼ │ │ Firehose Substreams Tier 1 │ │ Component │ └─────────────────────────────────────────────────────────────────────────┘ ``` ### Source Multiplexing The Relayer connects to multiple Reader Nodes simultaneously for: * **Redundancy**: If one Reader fails, others continue providing blocks * **Fork Coverage**: Different Readers may observe different forks from their network peers * **Latency Optimization**: Sources race to deliver blocks; the fastest wins Each source connection includes automatic reconnection and health monitoring. ### Realtime Gating The Relayer monitors each source's latency against a configurable threshold: * Sources exceeding the latency threshold are temporarily filtered out * This prevents lagging sources from degrading overall performance * Sources automatically rejoin when they catch up ### ForkableHub The ForkableHub is the core data structure that: * Maintains a fork-aware graph of recent blocks * Tracks the current head block number and timestamp * Calculates time drift between chain time and wall clock * Provides readiness signaling to downstream components The hub reads from both the live source factory (for new blocks) and one-block storage (for recent historical blocks not yet merged). ## gRPC Interface The Relayer exposes the same `BlockStream::Blocks` interface as the Reader Node. This consistent interface allows: * Downstream components to connect to either Reader or Relayer * Simple failover configurations * Uniform client code regardless of data source Blocks are streamed with metadata including: * Block number and hash * Parent block information * Fork step (new, undo, irreversible) * Partial block indicators (when applicable) ## Operational Patterns ### Single Relayer For simple deployments, one Relayer connects to all Readers: ``` Reader 1 ──┐ Reader 2 ──┼──► Relayer ──► Firehose Reader 3 ──┘ ``` ### Regional Relayers For geographically distributed deployments, regional Relayers can aggregate local Readers: ``` Region A: Region B: Reader A1 ──┐ Reader B1 ──┐ Reader A2 ──┼──► Relayer A Reader B2 ──┼──► Relayer B │ │ └──────────┬─────────────────┘ ▼ Global Relayer ──► Firehose ``` ## Storage Access The Relayer requires read access to one-block storage: * **Purpose**: Bootstrap ForkableHub with recent blocks * **Access pattern**: Read-only * **Volume**: Only reads recent blocks, not full history This allows the Relayer to maintain fork awareness even when starting up after all Readers have moved past certain blocks. ## Configuration Reference For complete configuration options, flags, and health check endpoints, see [Relayer CLI Reference](/references/cli-reference/relayer). # Firehose StreamingFast Firehose component for serving blockchain data The Firehose component is the primary interface for clients to access blockchain data. It serves the Firehose gRPC API, providing both historical and real-time block streaming with sophisticated fork handling and cursor-based resumption. ## Overview The Firehose component reads from multiple data sources and serves unified block streams to clients: ``` ┌──────────────────────────────────────────────────────────────┐ │ Firehose Component │ ├──────────────────────────────────────────────────────────────┤ │ │ │ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ │ │ One-Blocks │ │ Merged │ │ Forked │ │ │ │ Storage │ │ Blocks │ │ Blocks │ │ │ │ (bootstrap) │ │ (historical) │ │ (fork data) │ │ │ └──────┬───────┘ └──────┬───────┘ └──────┬───────┘ │ │ │ │ │ │ │ └────────────────┬┴─────────────────┘ │ │ ▼ │ │ ┌──────────────┐ ┌──────────────┐ │ │ │ ForkDB │◄──────│ Relayer │ │ │ │ │ │ (live blocks)│ │ │ └──────┬───────┘ └──────────────┘ │ │ │ │ │ ▼ │ │ ┌──────────────┐ │ │ │ gRPC Stream │──────► Clients │ │ │ (cursored) │ │ │ └──────────────┘ │ │ │ └──────────────────────────────────────────────────────────────┘ ``` ## Data Sources ### One-Block Files (Bootstrapping) One-block files are individual block files written by Reader components. Firehose uses these for: * **Recent blocks**: Blocks not yet merged into 100-block bundles * **Gap filling**: When merged blocks have gaps * **Bootstrap scenarios**: Initial startup before merged blocks exist ### Merged Blocks (Historical Reprocessing) Merged block files contain 100 blocks each and are the primary source for historical data: * **Efficient historical access**: Large batch reads from object storage * **Compressed storage**: Reduced storage costs and faster transfers * **Parallel processing**: Multiple block ranges can be served simultaneously ### Forked Blocks (Cursor Resolution) Forked block storage contains blocks from non-canonical chain branches: * **Fork preservation**: All forks are kept until finality * **Cursor resolution**: Enables resumption from any point, including forks * **Reorg handling**: Clients can resume even after chain reorganizations ### Live Blocks (Real-Time Feed) Live blocks come from the Relayer component: * **Sub-second latency**: New blocks streamed immediately * **Multiple sources**: Relayer connects to multiple Readers for redundancy * **Seamless transition**: Automatic switch from historical to live data ## Core Features ### Fork-Aware Streaming Firehose uses a `ForkDB` data structure that mirrors the blockchain's fork behavior: ``` Block Events: ───────────── new 100 ─── new 101 ─── new 102 ─── new 103a │ └── new 103b ─── undo 103a ─── new 104b The ForkDB tracks all branches and emits appropriate events when the canonical chain changes, allowing clients to maintain accurate local state. ``` Each block event includes: * **Step type**: `new`, `undo`, or `irreversible` * **Block data**: Full block with all transactions and receipts * **Cursor**: Unique position identifier for resumption ### Cursor-Based Resumption Cursors enable reliable, exactly-once delivery: ```protobuf message Cursor { string block_id = 1; // Current block hash uint64 block_num = 2; // Current block number string lib_id = 3; // Last irreversible block hash uint64 lib_num = 4; // Last irreversible block number string fork_step = 5; // new, undo, or irreversible } ``` **Resumption guarantees:** * Resume from exact position after disconnect * Resume from forked blocks (using forked blocks storage) * Skip already-processed blocks automatically * Handle reorgs that occurred during disconnect ### Automatic Reconnection Clients can implement robust reconnection logic: 1. Store the cursor from each received block 2. On disconnect, reconnect with the stored cursor 3. Firehose reconstructs the exact stream position 4. Continue processing without duplicates or gaps ### Seamless Historical-to-Live Transition Firehose automatically transitions between data sources: ``` Request: blocks 1000 to live Timeline: ───────── [Merged Blocks: 1000-2000] → [One-Blocks: 2001-2010] → [Live: 2011+] ▲ ▲ ▲ │ │ │ Historical data Recent blocks Real-time stream from storage not yet merged from Relayer ``` The transition is invisible to clients—they receive a continuous stream regardless of the underlying data source. ## Streaming Modes 1. **Historical range**: Specify start and stop block numbers 2. **Historical to live**: Specify start, omit stop to continue streaming indefinitely 3. **Live only**: Start from a recent block or "head" 4. **Cursor resume**: Provide cursor from previous session ## Operational Considerations ### Storage Requirements * **Merged blocks**: Primary storage, highly compressed * **One-blocks**: Temporary, cleaned up by Merger * **Forked blocks**: Pruned after configurable block count (default 50,000) ### High Availability For production deployments: 1. Run multiple Firehose instances behind a load balancer 2. All instances read from the same storage backend 3. Clients can reconnect to any instance using cursors 4. Use discovery service for automatic instance registration ## Configuration Reference For complete configuration options and flags, see [Firehose CLI Reference](/references/cli-reference/firehose). # Substreams StreamingFast Firehose Substreams component ## Substreams Component in Detail The Substreams component provides high-performance, parallelized blockchain data transformation and filtering capabilities. It enables consumers to define custom data pipelines that execute directly within the Firehose infrastructure. ### Why Operators Should Run Substreams Running Substreams alongside Firehose provides significant value to operators and their users: * **Value-Added Service**: Offer more than raw block data. Substreams enables users to transform, filter, and aggregate blockchain data in real-time. * **Infrastructure Reuse**: Substreams shares the same storage and data sources as Firehose. Running it adds minimal overhead while significantly expanding your service capabilities. * **Parallel Processing**: Substreams processes historical blockchain data in a massively parallelized manner, enabling processing speeds previously thought impossible. * **Ecosystem Compatibility**: Immediate support for dozens of data sinks (PostgreSQL, MongoDB, Kafka, ClickHouse, BigQuery, etc.), and the broader StreamingFast ecosystem. * **Developer Attraction**: Activate a community of developers who already know Substreams and will be eager to use your infrastructure. ### How Substreams Works Substreams modules are written in Rust, compiled to WebAssembly (WASM), and executed within the Substreams engine. Users define their data transformation logic in `.spkg` (Substreams Package) files containing: * **Map modules**: Transform input blocks into custom output types * **Store modules**: Aggregate data across blocks (counters, accumulators, state) * **Index modules**: Create filter conditions for efficient block skipping When a user runs a Substreams request, the engine: 1. Reads block data from the same merged block storage used by Firehose 2. Executes the user's WASM modules against each block 3. Streams the transformed output back to the consumer 4. Caches intermediate results for future requests ### Substreams Tier Architecture Substreams uses a two-tier architecture to optimize for both real-time streaming and parallel historical processing: #### Substreams Tier 1 The Tier 1 component is the primary entry point for Substreams requests: * Handles incoming gRPC requests from consumers * Manages live block streaming (connects to Relayer for real-time data) * Coordinates parallel historical processing by dispatching work to Tier 2 instances * Merges results from Tier 2 workers into a cohesive output stream * Manages output caching for frequently-requested data ranges #### Substreams Tier 2 The Tier 2 component provides the parallel processing backbone: * Executes WASM modules against historical block ranges * Processes multiple block ranges simultaneously across multiple instances * Stores execution results in cache storage for reuse * Scales horizontally to handle large parallel workloads {% hint style="info" %} **Scaling Strategy**: Tier 1 handles request coordination and live blocks, while Tier 2 scales horizontally for historical processing. In high-load scenarios, add more Tier 2 instances while keeping fewer Tier 1 instances. {% endhint %} ### Component Dependencies The Substreams components depend on other Firehose components: * **Merged Blocks Storage**: Both tiers read block data from the same storage used by Firehose * **One-Block Storage**: Required for accessing recent blocks not yet merged * **Relayer**: Tier 1 connects to the Relayer for live block streaming * **Cache Storage**: Both tiers use shared cache storage for execution results ``` ┌─────────────────────────────────────────────────────────────┐ │ Consumer Request │ └─────────────────────────────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────┐ │ Substreams Tier 1 │ │ • Request handling • Live streaming (via Relayer) │ │ • Work coordination • Result merging │ └─────────────────────────────────────────────────────────────┘ │ │ │ (historical ranges) │ (live blocks) ▼ ▼ ┌─────────────────────────┐ ┌─────────────────────────────┐ │ Substreams Tier 2 │ │ Relayer │ │ (parallel workers) │ │ (live block source) │ │ ┌─────┐ ┌─────┐ ┌─────┐│ └─────────────────────────────┘ │ │ T2 │ │ T2 │ │ T2 ││ │ └─────┘ └─────┘ └─────┘│ └─────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────┐ │ Shared Storage │ │ • Merged Blocks • One-Blocks • Substreams Cache │ └─────────────────────────────────────────────────────────────┘ ``` ### Production Considerations #### Caching Substreams execution results are cached, meaning subsequent requests for the same block ranges with the same modules will be served from cache. This is particularly valuable for: * Popular Substreams packages used by multiple consumers * Development workflows where users iterate on downstream processing * High-availability setups where results survive instance restarts #### Resource Requirements * **Tier 1**: Moderate CPU/memory for coordination; network-intensive for streaming * **Tier 2**: CPU-intensive for WASM execution; scales with parallelism needs * **Storage**: Cache storage grows with unique Substreams package usage #### High Availability For production deployments: * Run multiple Tier 1 instances behind a load balancer * Scale Tier 2 instances based on historical processing demand * Use shared cache storage accessible by all instances ### Default Ports | Component | Default Port | Purpose | | ----------------- | ------------ | ------------------------ | | Substreams Tier 1 | `:10016` | Consumer-facing gRPC API | | Substreams Tier 2 | `:10017` | Internal processing API | ### gRPC Services Substreams exposes the following gRPC services: * `sf.substreams.rpc.v2.Stream/Blocks` - Main streaming endpoint for Substreams execution ### What Consumers Can Build With Substreams, your users can build: * **Real-time Analytics**: Live dashboards, trading signals, protocol metrics * **Custom Indexers**: Purpose-built indexes for specific protocols or use cases * **Data Pipelines**: ETL workflows feeding databases, data warehouses, or message queues * **Cross-Chain Applications**: Consistent data processing across multiple blockchains {% hint style="success" %} **Operator Benefit**: Running Substreams differentiates your infrastructure from basic RPC providers. Users get powerful data transformation capabilities without managing their own indexing infrastructure. {% endhint %} # High Availability High availability for StreamingFast Firehose components ## Reader Node Placing multiple [Reader Node](/firehose/architecture/components/reader) components side by side, and fronted by one or more Relayers, allows for highly available setups; a core attribute of the Firehose design. A Relayer connected to multiple Readers will deduplicate the incoming stream and push the first block downstream. {% hint style="success" %} ***Tip**: Two Reader Node components will even race to push the data first. The system is designed to leverage this racing* [*Reader Node*](/firehose/architecture/components/reader) *feature to the benefit of the end-user by producing the lowest latency possible.* {% endhint %} ### Data Aggregation Firehose also aggregates any forked blocks that would be seen by a single Reader Node component, and not seen by any other [Reader Node](/firehose/architecture/components/reader) components. ### Component Cooperation Adding Reader Node components and dispersing each one geographically will result in the components actually racing to transfer blocks to the Relayer component. This cooperation between the [Reader Node](/firehose/architecture/components/reader) and [Relayer](/firehose/architecture/components/relayer) components *significantly* increases the performance of Firehose. ## Merger A single [Merger](/firehose/architecture/components/merger) component is required for Reader Node components in a highly available Firehose. Highly available systems usually connect to the [Relayer](/firehose/architecture/components/relayer) component to receive real-time blocks. Merged blocked files are used when Relayer components can't provide the requested data or satisfy a range. Restarts from other components can be sustained and time provided for [Merger](/firehose/architecture/components/merger) components to be down when Relayer components provide 200 to 300 blocks in RAM. {% hint style="info" %} **Note***: Merged blocks generally aren't read by other Firehose components in a running, live highly available system.* {% endhint %} ## Relayer A [Relayer](/firehose/architecture/components/relayer) component in a highly available Firehose will feed from all of the Reader Node components to gain a complete view of all possible forks. {% hint style="success" %} **Tip***: Multiple* [*Reader Node*](/firehose/architecture/components/reader) *components will ensure blocks are flowing efficiently to the* [*Relayer*](/firehose/architecture/components/relayer) *component and throughout Firehose.* {% endhint %} ## Firehose gRPC Server Firehose can be scaled horizontally to provide a highly available system. The network speed and data throughput between consumers and Firehose deployments will dictate the speed of data availability. {% hint style="info" %} **Note***: The network speed and data throughput between* [*Relayer*](/firehose/architecture/components/relayer) *components and* [*Firehose gRPC Server* ](https://github.com/streamingfast/firehose-docs/blob/master/architecture/components/grpc-server.md)*components will impact the speed of data availability.* {% endhint %} Firehose [gRPC Server](https://github.com/streamingfast/firehose-docs/blob/master/architecture/components/grpc-server.md) components have the ability to connect to a subset of Relayer components or all Relayers available. When the Firehose [gRPC Server](https://github.com/streamingfast/firehose-docs/blob/master/architecture/components/grpc-server.md) component is connected to all available Relayer components the probability that all forks will be viewed increases. Inbound requests made by consumers will be fulfilled with in-memory fork data. Block navigation can be delayed when forked data isn't completely communicated to the Firehose [gRPC Server](https://github.com/streamingfast/firehose-docs/blob/master/architecture/components/grpc-server.md) component. Understanding how data flows through Firehose is beneficial for harnessing its full power. # Data Flow StreamingFast Firehose data flow ## Data Flow in Firehose ### Data Flow in Detail The path and process of how data flows through the Firehose component family are important facets to understand when using the application. Blockchain data flows from instrumented nodes to the gRPC server through the Firehose [component family](/firehose/architecture/components). ### Data Flows Through Components Each Firehose component plays an important role as the blockchain data flows through it. ![StreamingFast Firehose architecture diagram](/files/ZYbofR5HkYkV8DwQg9jA) ### Data Flow Component Relationship The StreamingFast Instrumentation feeds to Reader components. The Reader components feed the Relayer component. Finally, the Firehose component hands data back to any consumers through its gRPC API. ### Key Points * An instrumented version of the native blockchain node process streams pieces of block data in a custom StreamingFast text-based protocol. * Firehose Reader components read data streams from instrumented blockchain nodes. * Reader components will then write the data to persistent storage. The data is then broadcast to the rest of the components in Firehose. * The Relayer component reads block data provided by one or more Reader components and provides a live data source for the other Firehose components. * The Merger component combines blocks created by Reader components into batches of one hundred individually merged blocks. The merged blocks are stored in an object store or written to disk. * The Firehose component receives blocks from: * **Merged blocks storage** for historical data requests * **One-block storage** for recent blocks not yet merged * **Forked blocks storage** for cursor resolution on forks * **Relayer** for live block data * The Firehose component then joins and returns the block data to its consumers through a cursor-based gRPC stream. * *Tradeoffs and benefits are presented for how data is stored and how it flows from the instrumented blockchain nodes through Firehose.* ## Reader Data Flow ### Firehose Instrumentation Firehose begins at the instrumentation conducted on nodes for targeted blockchains. The instrumentation itself is called Firehose Instrumentation and generate Firehose Logs. Firehose instrumentation is an augmentation to the target blockchain node's source code. The instrumentation is placed within the node where blockchain state synchronization happen, when the chain receives block from the P2P network and execute the transactions it contains locally to update its internal global state. ### Firehose Logs Firehose logs use a simple, unified text-based protocol over the operating system's standard output pipe. This protocol is **chain-agnostic** — any blockchain that implements the Firehose Logs specification automatically benefits from the entire Firehose ecosystem. The protocol consists of just two message types that the instrumented node outputs to stdout. {% hint style="info" %} For a reference implementation of Firehose instrumentation, see the [dummy-blockchain](https://github.com/streamingfast/dummy-blockchain) project. It demonstrates how to implement the protocol in a simple, educational blockchain node. {% endhint %} ### Firehose Logs Protocol The Firehose Logs protocol is intentionally simple, consisting of only two messages. {% hint style="info" %} For the complete protocol specification, including partial block support and block hash format requirements, see the [Firehose Protocol Reference](/references/firehose-protocol). {% endhint %} #### FIRE INIT The initialization message is sent once at startup to declare the protocol version and the protobuf type used for blocks: ``` FIRE INIT ``` | Field | Description | | --------------------- | --------------------------------------------------------------------- | | `version` | Protocol version. Supported: `1.0`, `3.0`, `3.1` | | `protobuf_block_type` | Fully qualified protobuf message name (e.g., `sf.acme.type.v1.Block`) | #### FIRE BLOCK Block messages are sent for each block, containing metadata and the full block payload: **Protocol version 1.0/3.0:** ``` FIRE BLOCK ``` **Protocol version 3.1 (with partial block support):** ``` FIRE BLOCK ``` | Field | Description | | ----------------- | -------------------------------------------------------------------------------------------------- | | `block_num` | Block height/number (uint64) | | `partial_idx` | (v3.1 only) Partial block index. Values ≥1000 indicate final partial (actual index = value - 1000) | | `block_hash` | Block identifier/hash | | `parent_num` | Parent block number (uint64) | | `parent_hash` | Parent block hash | | `lib_num` | Last irreversible block number (uint64) | | `timestamp_nanos` | Block timestamp in Unix nanoseconds | | `base64_block` | Base64-encoded protobuf block payload | {% hint style="warning" %} **Block Hash Consistency**: The `block_hash` and `parent_hash` fields must use a consistent string format throughout the chain's lifetime. The `parent_hash` of block N must exactly match the `block_hash` of block N-1. For new chains, use lowercase hexadecimal without the `0x` prefix. See [Block Hash Representation](/references/firehose-protocol#block-hash-representation) for details. {% endhint %} ### Example Firehose Logs Example output from an instrumented blockchain node: ```shell FIRE INIT 3.0 sf.acme.type.v1.Block FIRE BLOCK 1 4e07408562bedb8b60ce05c1decfe3ad16b72230967de01f640b7e4729b49fce 0 0000000000000000000000000000000000000000000000000000000000000000 0 1609459200000000000 CgRhY21lEg... FIRE BLOCK 2 4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a 1 4e07408562bedb8b60ce05c1decfe3ad16b72230967de01f640b7e4729b49fce 0 1609459201000000000 CgRhY21lEg... FIRE BLOCK 3 ef2d127de37b942baad06145e54b0c619a1f22327b2ebbcfbec78f5564afe39d 2 4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a 1 1609459202000000000 CgRhY21lEg... ``` The `base64_block` field contains the chain-specific block data serialized as protobuf and encoded in base64. The protobuf schema can model any blockchain's data structures while maintaining compatibility with the Firehose ecosystem. ### Firehose Logs & Reader Coordination The Firehose logs messages are read and processed by the Reader component from `firehose-core`. The `reader` component: * Launches the instrumented native node process and manages its lifecycle (start/stop/monitor) * Connects to the native node process' standard output pipe * Parses `FIRE INIT` to learn the protocol version and block type * Parses each `FIRE BLOCK` message, decodes the base64 payload, and wraps it in a Firehose block envelope After a block has been received, it is stored in persistent storage and simultaneously broadcast to all gRPC streaming subscribers. The persistent blocks enable historical access to all data in the blockchain without reliance on native node processes. The easily accessible block data enables StreamingFast's highly parallelized reprocessing tools to read and manipulate different sections of the chain at the developer's convenience. ## Relayer Data Flow ### Relayer Data Flow in Detail The Relayer component is responsible for connecting to one or more Reader components and receiving live block data from them. ### Multiple Relayer Connections The Relayer component uses multiple connections to provide data redundancy for scenarios where Reader components have crashed or require maintenance. The Relayer also deduplicates incoming blocks resulting in speeds that match the fastest Reader available to read data from. ### Racing Relayer Data The design of the Relayer component enables them to race to push data to consumers. ### Live Data Through Relayer The Relayer component can function as a live data source for blocks in Firehose. ### Relayer & Reader Overlap Relayer components serve the same interface as Reader components in simple setups without the need for high availability. ## Merger Data Flow ### Merger Data Flow in Detail Merger components create bundles containing one hundred blocks per bundle. The Merger component utilizes persisted one-block files to create the one hundred blocks bundle. ### Merger Data Flow Responsibilities The Merger component assists with the reduction of storage costs, improved data compression, and more efficient metered network access to single 100 blocks bundles. ### Historical Data Access The blocks bundled by the Merger component become the file-based historical data source of blocks for all Firehose components. ## Firehose Data Flow ### Firehose Data Flow in Detail The Firehose component is responsible for supplying the stream of block data to requesting consumers. The Firehose component can be thought of as the top most component in the Firehose architectural stack. ### Data Sources Firehose components connect to multiple data sources to serve consumer requests: * **Merged blocks storage**: Primary source for historical data (100-block bundles) * **One-block storage**: Recent blocks not yet merged * **Forked blocks storage**: Blocks from non-canonical branches for cursor resolution * **Relayer**: Live blocks for real-time streaming {% hint style="info" %} *Firehose was designed to seamlessly switch between data sources as it fulfills inbound requests from consumers.* {% endhint %} ### Historical Data Requests Consumer requests for historical blocks are fetched from merged blocks storage. The historical blocks are passed through a `ForkDB` and sent with a cursor uniquely identifying the block and its position in the blockchain. ### Fork Preservation Firehose has the ability to resume from forked blocks because all forks are preserved in the forked blocks storage during node data processing. ### Cursor-Based Streaming Each block sent to clients includes a cursor containing: * Block number and hash * Last irreversible block information * Fork step (new, undo, or irreversible) Clients can resume streaming from any cursor, even if the block was on a fork that has since been abandoned. ## `bstream` ### bstream in Detail The StreamingFast bstream package manages flows of blocks and forks in a blockchain through a handler-based interface, similar to Go's net/http package. ### bstream Orchestration The bstream package is responsible for collaboration between all other Firehose components. The bstream package abstracts details surrounding files and block streaming from instrumented blockchain nodes. {% hint style="success" %} **Tip***: The bstream package presents an extremely powerful and simplified interface for dealing will all blockchain reorganizations.* {% endhint %} ### bstream Design & Motivation StreamingFast built, refined, and enhanced the bstream package over the period of several years. Key design considerations for bstream included high speed for data transfers and fast data throughput. Capabilities include downloading multiple files in parallel, decoding multiple blocks in parallel, and inline filtering. ### bstream & ForkDB An extremely important element of proper blockchain linearity is the StreamingFast `ForkDB.` The `bstream` package utilizes the `ForkDB` data structure for data storage. ### ForkDB in Detail The `ForkDB` is a graph-based data structure that mimics the forking logic used by the native blockchain node. `The ForkDB` receives all blocks and orders them based on the parent-child relationship defined by the chain. The ForkDB will keep around active forked branches and reorganizations that are occurring on-chain. ### ForkDB Events When a block branch becomes the longest block chain, the `ForkDB` will switch to it. The `ForkDB` will emit a series of events for proper handling of forks for example `new 1b`, `new 2b`, `undo 2b`, `new 2a`, `new 3a`, etc. ### Active Forks Active forks are kept until a certain level of confirmation is achieved or when blocks become final or irreversible. The exact rules for the confirmation can be configured for specific blockchains. ### ForkDB Irreversibility Events Specific irreversibility events are emitted by the `ForkDB` instance. Each event emitted by the ForkDB instance contains: * the step’s type of `new`, `undo`, or `irreversible,` * the block the step relates to, * and a cursor. ### ForkDB Cursor The ForkDB cursor points to a specific position in the stream of events emitted by `ForkDB` and the blockchain itself. The ForkDB cursor contains information that is required to reconstruct an equivalent forked or canonical instance of the `ForkDB`. ### Start & Stop Event Streaming The ForkDB is created in the correct branch, enabling the ability to perfectly resume the event streaming where the consumer last stopped. ### Chain-agnostic ForkDB The `bstream` library is chain-agnostic, and is only concerned about the concept of a `Block`. ### bstream Metadata The `bstream` library contains the minimally required metadata to maintain the consistency of the chain. ### Block & Protocol Buffers `Block` carries a payload of Protocol Buffer bytes. The payload can be decoded by the consumer in accordance with one of the supported chain-specific `Block` definitions, for example, `sf.ethereum.type.v1.Block`. ## Substreams Data Flow ### Substreams Data Flow in Detail The Substreams component provides parallel data transformation capabilities on top of the Firehose data pipeline. Substreams reads from the same data sources as other Firehose components and adds a powerful processing layer. ### Data Sources Substreams consumes data from: * **Merged Blocks Storage**: Historical block data processed by the Merger component * **One-Block Storage**: Recent blocks not yet merged * **Relayer**: Live blocks for real-time streaming ### Two-Tier Processing Substreams uses a two-tier architecture to optimize processing: * **Tier 1**: Handles request coordination, live block streaming (from Relayer), and merges results from Tier 2 workers * **Tier 2**: Executes WASM modules against historical block ranges in parallel ### Execution Flow 1. Consumer sends a Substreams request with a `.spkg` package to Tier 1 2. For live blocks: Tier 1 executes modules directly using blocks from the Relayer 3. For historical blocks: Tier 1 dispatches block ranges to multiple Tier 2 instances 4. Tier 2 instances read blocks from merged block storage and execute WASM modules 5. Results are cached for future requests with the same modules and block ranges 6. Tier 1 merges all results and streams them back to the consumer ### Caching Strategy Substreams caches execution results at module output boundaries. This enables: * Fast subsequent requests for the same data * Efficient composability where downstream modules reuse cached upstream outputs * Reduced computational overhead for popular Substreams packages ### Data Storage in Detail Understanding the storage mechanisms and methodologies used for data in Firehose is another important topic. Additional details on Firehose [data storage](/firehose/architecture/data-storage) are provided in the documentation. # Data Storage StreamingFast Firehose data storage ## Data Storage in Firehose ### Data Storage in Detail Data and the locations where it is stored are important facets of Firehose deployment and operation. Key Firehose data storage topics include [Data Stores](#data-stores), [Merged blocks files](#merged-blocks-files), [serialization](#serialization), [one block files](#one-block-files), and [100-blocks files](#one-hundred-blocks-files). ## Data Stores Firehose Stores are abstractions sitting on top of Object Storage. {% hint style="info" %} **Note***:* *Object Storage is a data storage technique that manages data as objects in opposition to other data storage architectures like hierarchical file systems.* {% endhint %} ### Abstraction Library Stores utilize the Firehose [dstore abstraction library](https://github.com/streamingfast/dstore) to provide support for local file systems, [Azure](https://www.google.com/aclk?sa=l\&ai=DChcSEwjr3Yqr9r75AhVuH60GHaPqCPAYABAAGgJwdg\&sig=AOD64_1oS9RVQu923fWqHBIH9TUq9RxM_w\&q\&adurl\&ved=2ahUKEwjZ_4Or9r75AhXjKX0KHR_eBJYQ0Qx6BAgDEAE), [Google Cloud](https://cloud.google.com/), [Amazon S3](https://www.google.com/aclk?sa=l\&ai=DChcSEwiitIe_9r75AhXMwsIEHaRvBvsYABAAGgJwdg\&sig=AOD64_0zvgrb2ySU8puRmtykCtCNbLSHQw\&q\&adurl\&ved=2ahUKEwiqpoC_9r75AhWjKn0KHbOGDaYQ0Qx6BAgDEAE), and other Amazon S3 API compatible object storage solutions such as [MinIO](https://min.io/) or [Ceph](https://ceph.com/en/). ### Production Environments For production deployments outside of cloud providers, StreamingFast recommends [Ceph](https://ceph.com/en/) as the distributed storage instead of its compatible Amazon S3 API system. ## Serialization Firehose primarily utilizes [Protocol Buffers version 3](https://developers.google.com/protocol-buffers) for serialization. ## Merged Blocks Files ### Merged Blocks in Detail Merged blocks files are also referred to as `100-blocks files`, and merged bundles. These terms are all used interchangeably within Firehose. Merged blocks are binary files that use the [dbin](https://github.com/streamingfast/dbin) packing format to store a series of [bstream block objects](https://github.com/streamingfast/proto/blob/develop/sf/bstream/v1/bstream.proto), serialized as [protocol buffers](https://developers.google.com/protocol-buffers). ### Merged Block Creation Firehose uses [Reader Node](/firehose/architecture/components/reader) components that have been set with a special flag to work in *catch-up* mode to create merged blocks. ### Highly-available Merged Blocks In [high-availability](/firehose/architecture/components/high-availability) Firehose configurations, merged blocks will be created by the [Merger](/firehose/architecture/components/merger) component. The [Reader Node](/firehose/architecture/components/reader) component will provide the Merger component with one-block files. ### Block Bundles The [Merger](/firehose/architecture/components/merger) component will also collate all of the one-block files into a single bundle of blocks. ### One Hundred Blocks Files Up to one hundred blocks can be contained within a single 100-blocks file. The 100-blocks files can include multiple versions such as a fork block or a given block number, ensuring continuity through the previous block link. ### Blocks Files Consumption & Use Nearly all components in Firehose rely on or utilize 100-blocks files. The bstream library consumes 100-blocks files for example. Protocol-specific decoded block objects, like Ethereum, are what circulate amongst all processes that work with executed block data in Firehose. ## One Block Files ### One Block Files in Detail In [high availability](/firehose/architecture/components/high-availability) configurations, one-block files are transient and ensure the [Merger](/firehose/architecture/components/merger) component gathers all visible forks from any [Reader Node](/firehose/architecture/components/reader) components. {% hint style="warning" %} **Important***: One-block files contain only one `bstream.Block` as a serialized protocol buffer.* {% endhint %} ### One-block File Consumption & Use One-block files are consumed by the `Merger` component, bundled in executed \_\_ 100-blocks files. The one-block files are then stored to `dstore` storage and consumed by most of the other Firehose processes. # Deployment Guide This deployment guide provides **chain-agnostic** instructions for deploying Firehose. The concepts, commands, and deployment patterns shown here can be applied to any blockchain that has Firehose support. {% hint style="info" %} We use [dummy-blockchain](https://github.com/streamingfast/dummy-blockchain) as our example throughout this guide. This is a simple test blockchain that demonstrates all Firehose concepts without the complexity of a real blockchain. {% endhint %} ## Prerequisites Before starting, you'll need: 1. **Firecore binary**: Download from [firehose-core releases](https://github.com/streamingfast/firehose-core/releases) 2. **Dummy blockchain binary**: Install the example blockchain client ### Installing Dummy Blockchain ```bash # Install the dummy blockchain binary go install github.com/streamingfast/dummy-blockchain@latest # Verify installation dummy-blockchain --help ``` The dummy blockchain can be run standalone with: ```bash dummy-blockchain start --store-dir= --block-rate=180 ``` ## Deployment Options Choose your deployment approach based on your needs: ### 🏠 [Single Machine Deployment](/firehose/overview/single-machine-deployment) **Recommended for**: Development, testing, small-scale production * All components run on one machine * Shared local storage between components * Simpler setup and management * Lower resource requirements ### 🏢 [Distributed Deployment](/firehose/overview/distributed-deployment) **Recommended for**: Production, high-availability, scalability * Components run as separate processes/services * Shared object storage (cloud storage) * Horizontal scalability * Production-ready architecture ## Architecture Overview Both deployment patterns use the same core [Firehose architecture](/firehose/architecture): 1. **Reader**: Manages the blockchain node and extracts block data 2. **Merger**: Combines one-block files into merged block files 3. **Relayer**: Streams live blocks to consumers 4. **Firehose**: Serves historical and live block data via gRPC 5. **Substreams Tier 1**: Handles consumer requests and coordinates parallel processing 6. **Substreams Tier 2**: Executes WASM modules for parallel historical data transformation {% hint style="info" %} **About Substreams**: Substreams is a high-performance parallel data transformation engine that runs alongside Firehose. It enables users to define custom data pipelines in Rust/WASM that execute directly within your infrastructure. Running Substreams adds significant value for users while reusing the same block storage as Firehose. See [Substreams Component](/firehose/architecture/components/substreams) for details. {% endhint %} {% hint style="info" %} For detailed information about each component, see the [Architecture Components](/firehose/architecture/components) documentation. {% endhint %} # Single Machine Deployment This guide shows how to deploy all Firehose components on a single machine using shared local storage. This approach is ideal for development, testing, and small-scale production deployments. ## Overview In this deployment, all components (`reader-node`, `merger`, `relayer`, `firehose`, `substreams-tier1`, `substreams-tier2`) run as a single process with shared local storage. ``` ┌─────────────────────────────────────────┐ │ Single Machine │ ├─────────────────────────────────────────┤ │ Reader Process │ Firehose Stack │ │ ┌─────────────┐ │ ┌──────────────┐ │ │ │dummy-blockchain │ │ Reader │ │ │ │ (subprocess)│───┼──│ Merger │ │ │ │ │ │ │ Relayer │ │ │ └─────────────┘ │ │ Firehose & │ │ │ │ │ Substreams │ │ │ │ └──────────────┘ │ │ │ │ │ Shared Local Storage: ./firehose-data │ └─────────────────────────────────────────┘ ``` ## Prerequisites Before starting, ensure you have: 1. **Firecore binary**: Download from [firehose-core releases](https://github.com/streamingfast/firehose-core/releases) 2. **Dummy blockchain binary**: Install with `go install github.com/streamingfast/dummy-blockchain@latest` 3. **Both binaries available in PATH**: Verify with `firecore --help` and `dummy-blockchain --help` 4. **Available ports**: Ensure ports 10010, 10012, 10014, 10015, 10016, 10017 are not in use {% hint style="warning" %} **Port Conflicts**: If you encounter "address already in use" errors, check which ports are occupied: ```bash # Check if ports are in use netstat -tulpn | grep -E ':(10010|10012|10014|10015|10016|10017)' ``` {% endhint %} ## Step 1: Basic Configuration Create a working directory: ```bash # Create working directory mkdir firehose-workspace cd firehose-workspace ``` ## Step 2: Start the Firehose Stack Launch all components using the `firecore` binary: ```bash firecore start \ reader-node merger relayer firehose substreams-tier1 substreams-tier2 \ --config-file="" \ --data-dir="./firehose-data" \ --advertise-chain-name="acme-dummy-blockchain" \ --reader-node-path="dummy-blockchain" \ --reader-node-data-dir="./firehose-data/reader-node" \ --reader-node-arguments="start --tracer=firehose --store-dir={data-dir}/reader --block-rate=120" ``` {% hint style="info" %} **Default Ports Used:** * **Firehose**: `:10015` (gRPC - [sf.firehose.v2.Stream](https://buf.build/streamingfast/firehose/docs/main:sf.firehose.v2)) * **Reader**: `:10010` (gRPC - internal reader protocol) * **Relayer**: `:10014` (gRPC - live block streaming) * **Merger**: `:10012` (gRPC - internal merger protocol) * **Substreams Tier1**: `:10016` (gRPC - [sf.substreams.rpc.v2.Stream](https://buf.build/streamingfast/substreams/docs/main:sf.substreams.rpc.v2)) * **Substreams Tier2**: `:10017` (gRPC - internal tier1 <=> tier2 protocol) The `--config-file=""` flag disables automatic config file loading switching into a flags only mode. {% endhint %} {% hint style="info" %} The `dummy-blockchain` runs as a subprocess of the Reader component. The Reader manages its lifecycle and extracts block data from it. Extracted data is exchanged through stdout pipe to the Reader component and contains chain's specific Protobuf block and metadata. See [Reader Component](/firehose/architecture/components/reader) for more details. {% endhint %} ## Step 3: Verify the Deployment Once the stack is running, you should see logs indicating that components are starting up. Let's verify each component is working correctly. ### Check One-Block Files The Reader component extracts individual blocks and stores them as one-block files: ```bash # List one-block files (should appear after a few seconds) ls ./firehose-data/storage/one-blocks/ # Inspect a specific one-block file firecore tools print one-block ./firehose-data/storage/one-blocks 1 --output=text ``` {% hint style="info" %} The output will be in protobuf text format, which is expected. This shows the raw block data as extracted by the Reader component. {% endhint %} {% hint style="info" %} One-block files contain individual block data as extracted by the Reader. Learn more about [Data Storage](/firehose/architecture/data-storage) patterns. {% endhint %} ### Check Merged Blocks The Merger component combines one-block files into larger merged block files: ```bash # List merged block files (should appear after merger processes one-blocks) ls ./firehose-data/storage/merged-blocks/ # Inspect a merged block file (use an actual filename from the directory) firecore tools print merged-blocks ./firehose-data/storage/merged-blocks 100 --output=text ``` {% hint style="info" %} Merged blocks are optimized for efficient storage and streaming. See [Merger Component](/firehose/architecture/components/merger) for details. {% endhint %} ### Check Relayer Stream The Relayer provides live block streaming: ```bash # Stream live blocks from the relayer (in a separate terminal) firecore tools relayer stream localhost:10010 -o text +3 ``` This command will show the last 3 blocks and then stop the stream. {% hint style="info" %} The Relayer enables real-time block streaming for live applications. Learn more about [Relayer Component](/firehose/architecture/components/relayer). {% endhint %} ## Step 4: Test the Firehose API Test the Firehose API using the built-in client tools: ```bash # Get blocks 1-5 from the Firehose API firecore tools firehose-client -p localhost:10015 -o text -- 1:5 # Get a single block (block 5) firecore tools firehose-single-block-client -p localhost:10015 -o text -- 5 # View full block data in JSON format firecore tools firehose-single-block-client -p localhost:10015 -o protojson -- 5 # Alternative JSON output firecore tools firehose-single-block-client -p localhost:10015 -o json -- 5 ``` ## Step 5: Test Substreams Verify that Substreams tiers are working: ```bash # Test a simple Substreams request substreams run -e localhost:10016 -p common@v0.1.0 -s 1 -t +5 ``` ## Configuration Options ### Storage Locations By default, all data is stored under `./firehose-data/storage`: * **One-blocks**: `./firehose-data/storage/one-blocks` (controlled by `--common-one-block-store-url`) * **Merged blocks**: `./firehose-data/storage/merged-blocks` (controlled by `--common-merged-blocks-store-url`) These paths are shared among all components and can be customized using the respective flags. The `--data-dir` flag sets the base directory for all storage locations. ## Next Steps * **Production deployment**: Consider the [Distributed Deployment](/firehose/overview/distributed-deployment) for production use * **Chain-specific setup**: Adapt these concepts for your target blockchain using [Chain-Specific Implementations](https://github.com/streamingfast/firehose-docs/blob/master/ethereum/README.md) * **Advanced configuration**: Explore the [CLI Reference](/references/cli-reference) for more options {% hint style="success" %} You now have a fully functional Firehose deployment! The same patterns shown here with `dummy-blockchain` can be applied to any Firehose-enabled blockchain. {% endhint %} # Distributed Deployment This guide shows how to deploy Firehose components as separate processes using shared object storage. This approach is recommended for production environments where you need scalability, high availability, and proper service isolation. ## Overview In this deployment, each component (`reader-node`, `merger`, `relayer`, `firehose`, `substreams-tier1`, `substreams-tier2`) runs as a separate process. Components communicate through shared object storage and gRPC endpoints. ``` ┌─────────────────-┐ ┌─────────────────┐ ┌─────────────────┐ │ Reader │ │ Processing │ │ Serving │ │ Process │ │ Components │ │ Components │ ├─────────────────-┤ ├─────────────────┤ ├─────────────────┤ │ ┌─────────────┐ │ │ ┌─────────────┐ │ │ ┌─────────────┐ │ │ │dummy-blockchain│ │ │ Merger │ │ │ │ Firehose │ │ │ │ (subprocess)│ │ │ │ Relayer │ │ │ │ Substreams │ │ │ │ Reader │ │ │ │ │ │ │ │ │ │ │ └─────────────┘ │ │ └─────────────┘ │ │ └─────────────┘ │ └─────────────────-┘ └─────────────────┘ └─────────────────┘ │ │ │ └───────────────────────┼───────────────────────┘ │ ┌─────────────────┐ │ Shared Object │ │ Storage │ │ (Cloud Storage) │ └─────────────────┘ ``` {% hint style="info" %} While this guide shows all components running on a single machine for simplicity, in production you would typically deploy these across multiple machines with proper ingress, DNS, and service discovery. {% endhint %} ## Prerequisites 1. **Shared Object Storage**: Set up cloud storage (AWS S3, Google Cloud Storage, etc.) or On-Premise Storage Solution compatible with S3 like Ceph (*recommended*). 2. **Binaries**: Install `firecore` and `dummy-blockchain` as described in [Prerequisites](/firehose/overview#prerequisites) ## Storage Configuration First, configure your shared object storage. For this example, we'll use a local filesystem path that simulates cloud storage: ```bash # Create shared storage directory (in production, this would be cloud storage) mkdir -p ./shared-storage/firehose-data export SHARED_STORAGE_URL="file://$(pwd)/shared-storage/firehose-data" # For cloud storage, you would use URLs like: # export SHARED_STORAGE_URL="gs://your-bucket/firehose-data" # export SHARED_STORAGE_URL="s3://your-bucket/firehose-data" ``` ## Component 1: Reader Node The Reader manages the blockchain node and extracts block data. ```bash # Terminal 1: Start the Reader firecore start reader-node \ --config-file="" \ --data-dir="./reader-data" \ --advertise-block-id-encoding="hex" \ --advertise-chain-name="acme-dummy-blockchain" \ --reader-node-path="dummy-blockchain" \ --reader-node-data-dir="./reader-data/node" \ --reader-node-arguments="start --tracer=firehose --store-dir=./reader-data/node --block-rate=120 --genesis-height=0 --genesis-block-burst=100" \ --common-one-block-store-url="${SHARED_STORAGE_URL}/one-blocks" \ --reader-node-grpc-listen-addr=":10010" \ --reader-node-manager-api-addr=":10013" ``` {% hint style="info" %} The Reader runs the `dummy-blockchain` as a subprocess and extracts block data to shared storage. See [Reader Component](/firehose/architecture/components/reader) for details. {% endhint %} ### Verify Reader Operation ```bash # Check that one-block files are being created ls ./shared-storage/firehose-data/one-blocks/ # Inspect a one-block file firecore tools print one-block ./shared-storage/firehose-data/one-blocks 1 --output=text ``` ## Component 2: Merger The Merger combines one-block files into merged block files for efficient storage. ```bash # Terminal 2: Start the Merger firecore start merger \ --config-file="" \ --data-dir="./merger-data" \ --advertise-block-id-encoding="hex" \ --advertise-chain-name="acme-dummy-blockchain" \ --common-one-block-store-url="${SHARED_STORAGE_URL}/one-blocks" \ --common-merged-blocks-store-url="${SHARED_STORAGE_URL}/merged-blocks" \ --merger-grpc-listen-addr=":10011" ``` {% hint style="info" %} The Merger processes one-block files from shared storage and creates optimized merged block files. Learn more about [Merger Component](/firehose/architecture/components/merger). {% endhint %} ### Verify Merger Operation ```bash # Check that merged block files are being created ls ./shared-storage/firehose-data/merged-blocks/ # Inspect a merged block file firecore tools print merged-blocks ./shared-storage/firehose-data/merged-blocks 100 --output=text ``` ## Component 3: Relayer The Relayer provides live block streaming capabilities. ```bash # Terminal 3: Start the Relayer firecore start relayer \ --config-file="" \ --data-dir="./relayer-data" \ --advertise-block-id-encoding="hex" \ --advertise-chain-name="acme-dummy-blockchain" \ --common-one-block-store-url="${SHARED_STORAGE_URL}/one-blocks" \ --relayer-grpc-listen-addr=":10012" \ --relayer-source="reader-node:localhost:10010" ``` {% hint style="info" %} The Relayer connects to the Reader to stream live blocks and provides real-time data access. See [Relayer Component](/firehose/architecture/components/relayer) for more details. {% endhint %} ### Verify Relayer Operation ```bash # Stream live blocks from the relayer firecore tools relayer stream localhost:10012 -o text +3 ``` ## Component 4: Firehose The Firehose component serves historical and live block data via gRPC. ```bash # Terminal 4: Start Firehose firecore start firehose \ --config-file="" \ --data-dir="./firehose-data" \ --advertise-block-id-encoding="hex" \ --advertise-chain-name="acme-dummy-blockchain" \ --common-one-block-store-url="${SHARED_STORAGE_URL}/one-blocks" \ --common-merged-blocks-store-url="${SHARED_STORAGE_URL}/merged-blocks" \ --firehose-grpc-listen-addr=":10015" \ --common-live-blocks-addr="localhost:10012" ``` ### Verify Firehose Operation ```bash # Test the Firehose gRPC API grpcurl -plaintext -d '{"start_block_num": 1, "stop_block_num": 5}' \ localhost:10015 sf.firehose.v2.Stream/Blocks ``` ## Component 5: Substreams Tier 1 Substreams Tier 1 serves as the entry point for Substreams requests, handling live blocks directly and delegating historical block processing to Tier 2 workers. ```bash # Terminal 5: Start Substreams Tier 1 firecore start substreams-tier1 \ --config-file="" \ --data-dir="./substreams-tier1-data" \ --advertise-block-id-encoding="hex" \ --advertise-chain-name="acme-dummy-blockchain" \ --common-one-block-store-url="${SHARED_STORAGE_URL}/one-blocks" \ --common-merged-blocks-store-url="${SHARED_STORAGE_URL}/merged-blocks" \ --substreams-tier1-grpc-listen-addr=":10016" \ --common-live-blocks-addr="localhost:10012" \ --substreams-tier1-subrequests-endpoint="localhost:10017" ``` ## Component 6: Substreams Tier 2 Substreams Tier 2 workers handle the actual block processing for historical data. Tier 1 delegates work to Tier 2 workers. {% hint style="info" %} Start Tier 2 before Tier 1, as Tier 1 connects to Tier 2 workers via `--substreams-tier1-subrequests-endpoint`. {% endhint %} ```bash # Terminal 6: Start Substreams Tier 2 firecore start substreams-tier2 \ --config-file="" \ --data-dir="./substreams-tier2-data" \ --advertise-block-id-encoding="hex" \ --advertise-chain-name="acme-dummy-blockchain" \ --common-merged-blocks-store-url="${SHARED_STORAGE_URL}/merged-blocks" \ --substreams-tier2-grpc-listen-addr=":10017" ``` ### Verify Substreams Operation ```bash # List available Substreams services grpcurl -plaintext localhost:10016 list | grep substreams grpcurl -plaintext localhost:10017 list | grep substreams ``` ## Load Balancer / API Gateway In production, you would typically put a load balancer or API gateway in front of your services: ```bash # Example nginx configuration for load balancing # upstream firehose_backend { # server localhost:10015; # } # # upstream substreams_backend { # server localhost:10016; # server localhost:10017; # } # # server { # listen 80; # location /firehose/ { # grpc_pass grpc://firehose_backend; # } # location /substreams/ { # grpc_pass grpc://substreams_backend; # } # } ``` ## Monitoring and Health Checks Monitor each component's health: ```bash # Check component health via gRPC health checks grpcurl -plaintext localhost:10011 grpc.health.v1.Health/Check # Merger grpcurl -plaintext localhost:10012 grpc.health.v1.Health/Check # Relayer grpcurl -plaintext localhost:10015 grpc.health.v1.Health/Check # Firehose grpcurl -plaintext localhost:10016 grpc.health.v1.Health/Check # Substreams Tier1 grpcurl -plaintext localhost:10017 grpc.health.v1.Health/Check # Substreams Tier2 ``` ## Production Considerations ### Service Discovery In production, components need to discover each other. Consider using: * **Kubernetes**: Service discovery via DNS * **Consul**: Service mesh with health checking * **AWS ELB/ALB**: Load balancing with health checks ### Storage Replace the local filesystem with proper cloud storage: ```bash # AWS S3 --common-one-block-store-url="s3://your-bucket/one-blocks" --common-merged-blocks-store-url="s3://your-bucket/merged-blocks" # Google Cloud Storage --common-one-block-store-url="gs://your-bucket/one-blocks" --common-merged-blocks-store-url="gs://your-bucket/merged-blocks" ``` ### High Availability For high availability: 1. **Run multiple instances** of each component 2. **Use health checks** and automatic restarts 3. **Implement proper monitoring** and alerting 4. **Use redundant storage** with replication ### Security Secure your deployment: 1. **TLS encryption** for gRPC communications 2. **Authentication** and authorization 3. **Network segmentation** and firewalls 4. **Secrets management** for storage credentials ## Scaling Scale components based on load: * **Reader**: Usually one per blockchain network * **Merger**: Can run multiple instances for different block ranges * **Relayer**: Multiple instances for high availability * **Firehose**: Scale horizontally based on query load * **Substreams**: Scale both tiers based on processing needs ## Next Steps * **Adapt for your blockchain**: Use these patterns with your target blockchain * **Production deployment**: Implement proper orchestration (Kubernetes, Docker Swarm) * **Monitoring**: Set up comprehensive monitoring and alerting * **Performance tuning**: Optimize based on your specific requirements {% hint style="success" %} You now have a distributed Firehose deployment! This architecture can be adapted to any cloud provider or orchestration platform for production use. {% endhint %} # Chain Specific Chain-specific Firehose configuration guides This section contains chain-specific configuration guides for Firehose. Each guide covers the Reader Node configuration required to extract blockchain data for that particular chain. Before reading these guides, ensure you understand the general deployment concepts from: * [Single Machine Deployment](/firehose/overview/single-machine-deployment) - For development or small-scale deployments * [Distributed Deployment](/firehose/overview/distributed-deployment) - For production deployments ## Supported Chains ### EVM Chains Ethereum and EVM-compatible chains use `fireeth` and share similar configuration patterns: | Chain | Type | Notes | | ---------------------------------------------------------------------------------------------------- | --------------- | -------------------------- | | [Ethereum](/firehose/overview/chains/ethereum) | Native Firehose | Mainnet, Sepolia, Hoodi | | [Arbitrum](/firehose/overview/chains/ethereum/arbitrum) | Native Firehose | Requires L1 connection | | [Base](/firehose/overview/chains/ethereum/base) | Native Firehose | OP Stack, requires OP Node | | [BNB Smart Chain](/firehose/overview/chains/ethereum/bsc) | Native Firehose | BSC Geth fork | | [Katana](/firehose/overview/chains/ethereum/katana) | Native Firehose | OP Stack | | [Optimism](/firehose/overview/chains/ethereum/optimism) | Native Firehose | OP Stack, requires OP Node | | [Polygon](/firehose/overview/chains/ethereum/polygon) | Native Firehose | Bor client | | [Unichain](/firehose/overview/chains/ethereum/unichain) | Native Firehose | OP Stack | | [Monad](https://github.com/streamingfast/firehose-docs/blob/master/firehose-setup/ethereum/monad.md) | Native Firehose | Monad BFT | | [Worldchain](/firehose/overview/chains/ethereum/worldchain) | Native Firehose | OP Stack | | [Avalanche](/firehose/overview/chains/avalanche) | RPC Poller | C-Chain via RPC | | [Sei](/firehose/overview/chains/sei) | Native Firehose | Cosmos SDK + EVM | ### Non-EVM Chains These chains use `firecore` with chain-specific binaries: | Chain | Type | Notes | | ------------------------------------------------ | --------------- | ------------------------ | | [Injective](/firehose/overview/chains/injective) | RPC Poller | Cosmos SDK chain | | [NEAR](/firehose/overview/chains/near) | Native Firehose | NEAR Protocol | | [Solana](/firehose/overview/chains/solana) | Geyser Plugin | Solana validator plugin | | [Starknet](/firehose/overview/chains/starknet) | RPC Poller | Requires Ethereum L1 RPC | | [Stellar](/firehose/overview/chains/stellar) | RPC Poller | Soroban RPC | | [Tron](/firehose/overview/chains/tron) | RPC Poller | Native and EVM formats | ## Integration Types ### Native Firehose Chains with native Firehose instrumentation have a patched blockchain client that outputs block data in Firehose format. This provides the most complete data extraction. ### RPC Poller For chains without native instrumentation, an RPC poller fetches blocks from standard RPC endpoints. You can use your own node or an RPC provider. ### Geyser Plugin Solana uses a Geyser plugin that hooks into the validator to extract block data. # Ethereum Firehose chain-specific configuration for Ethereum and EVM chains This page covers Reader Node configuration for Ethereum and EVM-compatible chains. For general Firehose architecture and deployment, see the [Single Machine Deployment](/firehose/overview/single-machine-deployment) or [Distributed Deployment](/firehose/overview/distributed-deployment) guides. {% hint style="warning" %} This guide does not cover how to sync an Ethereum node. For node synchronization, storage requirements, and network configuration, refer to the official documentation of your chosen client. {% endhint %} ## Binary & Releases | Component | Repository | Binary | | --------- | ----------------------------------------------------------------------- | --------- | | Firehose | [firehose-ethereum](https://github.com/streamingfast/firehose-ethereum) | `fireeth` | The `fireeth` binary includes all `firecore` functionality plus Ethereum-specific features and commands. Download releases from the [GitHub releases page](https://github.com/streamingfast/firehose-ethereum/releases) or use published Docker images from [GitHub Container Registry](https://github.com/streamingfast/firehose-ethereum/pkgs/container/firehose-ethereum). {% hint style="info" %} For Reader Nodes, each network provides bundle Docker images containing both `fireeth` and the network's Firehose-patched node binary. See the [network-specific guides](#supported-networks) below for Docker image references. {% endhint %} ## Supported Networks Firehose for Ethereum supports **Geth and Geth forks**. Each network requires a Firehose-patched version of its client: | Network | Guide | | -------------------------------- | ---------------------------------------------------------------------------------------------------- | | Ethereum Mainnet, Sepolia, Hoodi | [Ethereum Mainnet](/firehose/overview/chains/ethereum/mainnet) | | Arbitrum One, Nova | [Arbitrum](/firehose/overview/chains/ethereum/arbitrum) | | Base Mainnet, Sepolia | [Base](/firehose/overview/chains/ethereum/base) | | BNB Smart Chain | [BNB Smart Chain](/firehose/overview/chains/ethereum/bsc) | | Katana | [Katana](/firehose/overview/chains/ethereum/katana) | | Optimism Mainnet, Sepolia | [Optimism](/firehose/overview/chains/ethereum/optimism) | | Polygon PoS | [Polygon](/firehose/overview/chains/ethereum/polygon) | | Unichain | [Unichain](/firehose/overview/chains/ethereum/unichain) | | Monad Mainnet | [Monad](https://github.com/streamingfast/firehose-docs/blob/master/firehose-setup/ethereum/monad.md) | | Worldchain | [Worldchain](/firehose/overview/chains/ethereum/worldchain) | ## Common Configuration All Ethereum-based networks share these characteristics: ### Chain Name (The Graph Network Registry) Only `--advertise-chain-name` needs to be specified. All other advertise fields (block ID encoding, chain aliases, etc.) are automatically derived from [The Graph Network Registry](https://thegraph.com/networks/). ```bash --advertise-chain-name="" ``` ### VM Trace Flag The Firehose-patched Geth (and forks) require the `--vmtrace=firehose` flag to emit Firehose Protocol logs to stdout. ### Basic Reader Node Pattern ```bash fireeth start reader-node \ --reader-node-path="" \ --reader-node-arguments="--vmtrace=firehose --datadir={node-data-dir} " \ --advertise-chain-name="" \ ``` ## Resources * [firehose-ethereum GitHub](https://github.com/streamingfast/firehose-ethereum) * [The Graph Network Registry](https://thegraph.com/networks/) * [Geth Documentation](https://geth.ethereum.org/docs) # Ethereum Mainnet Firehose configuration for Ethereum Mainnet and testnets This page provides Ethereum Mainnet specific configuration. First read the [Ethereum general notes](/firehose/overview/chains/ethereum) for common information. ## Docker Image ``` ghcr.io/streamingfast/go-ethereum:geth--fh3.0 ``` [View available versions on GitHub Packages](https://github.com/streamingfast/go-ethereum/pkgs/container/go-ethereum) The image contains both the Firehose-patched Geth binary and `fireeth`. ## Client Binary | Client | Repository | Source Branch | Releases | | ----------------------- | ------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------- | | Geth (Firehose-patched) | [streamingfast/go-ethereum](https://github.com/streamingfast/go-ethereum) | [release/geth-1.x-fh3.0](https://github.com/streamingfast/go-ethereum/tree/release/geth-1.x-fh3.0) | [geth-\*](https://github.com/streamingfast/go-ethereum/releases?q=geth\&expanded=true) | ## Networks | Network | Chain Name | | ------- | ------------- | | Mainnet | `eth-mainnet` | | Sepolia | `eth-sepolia` | | Hoodi | `eth-hoodi` | ## Consensus Client Requirement {% hint style="warning" %} Ethereum requires a consensus (beacon) client to sync. You must run a consensus client (Lighthouse, Prysm, Teku, etc.) alongside Geth. Refer to the [Geth documentation](https://geth.ethereum.org/docs/getting-started/consensus-clients) for setup instructions. {% endhint %} ## Reader Node Configuration ### Ethereum Mainnet ```bash fireeth start reader-node \ --advertise-chain-name="eth-mainnet" \ --reader-node-path="geth" \ --reader-node-arguments="--datadir={node-data-dir} --networkid=1 --authrpc.addr=0.0.0.0 --authrpc.port=8551 --authrpc.vhosts=* --authrpc.jwtsecret=/path/to/jwt.hex --vmtrace=firehose" \ ``` ### Sepolia Testnet ```bash fireeth start reader-node \ --advertise-chain-name="eth-sepolia" \ --reader-node-path="geth" \ --reader-node-arguments="--datadir={node-data-dir} --sepolia --syncmode=full --authrpc.addr=0.0.0.0 --authrpc.port=8551 --authrpc.vhosts=* --authrpc.jwtsecret=/path/to/jwt.hex --vmtrace=firehose" \ ``` ### Hoodi Testnet ```bash fireeth start reader-node \ --advertise-chain-name="eth-hoodi" \ --reader-node-path="geth" \ --reader-node-arguments="--datadir={node-data-dir} --hoodi --syncmode=full --authrpc.addr=0.0.0.0 --authrpc.port=8551 --authrpc.vhosts=* --authrpc.jwtsecret=/path/to/jwt.hex --vmtrace=firehose" \ ``` ## Key Geth Flags | Flag | Description | | ------------------------------------- | -------------------------------------------------- | | `--vmtrace=firehose` | **Required.** Enables Firehose Protocol output | | `--networkid` | Network ID (1 for mainnet) | | `--mainnet` / `--sepolia` / `--hoodi` | Network selection (alternative to networkid) | | `--authrpc.jwtsecret` | Path to JWT secret for consensus client connection | | `--authrpc.addr` | Auth RPC listen address for consensus client | | `--authrpc.port` | Auth RPC port (default 8551) | | `--datadir` | Data directory (use `{node-data-dir}` template) | ## Resources * [Geth Documentation](https://geth.ethereum.org/docs) * [Consensus Client Setup](https://geth.ethereum.org/docs/getting-started/consensus-clients) * [streamingfast/go-ethereum](https://github.com/streamingfast/go-ethereum) # Arbitrum Firehose configuration for Arbitrum One This page provides Arbitrum One specific configuration. First read the [Ethereum general notes](/firehose/overview/chains/ethereum) for common information. ## Docker Image ``` ghcr.io/streamingfast/nitro:-fh3.0 ``` [View available versions on GitHub Packages](https://github.com/streamingfast/nitro/pkgs/container/nitro) The image contains both the Firehose-patched Nitro binary and `fireeth`. ## Client Binary | Client | Repository | | ------------------------ | ------------------------------------------------------------- | | Nitro (Firehose-patched) | [streamingfast/nitro](https://github.com/streamingfast/nitro) | ## Networks | Network | Chain Name | | ---------------- | ------------- | | Arbitrum One | `arb-one` | | Arbitrum Nova | `arb-nova` | | Arbitrum Sepolia | `arb-sepolia` | ## L1 Dependency {% hint style="warning" %} Arbitrum requires access to an Ethereum L1 node (both execution and beacon client) for operation. The L1 connection is used for state validation and blob data retrieval. {% endhint %} ## Reader Node Configuration ### Arbitrum One ```bash fireeth start reader-node \ --advertise-chain-name="arb-one" \ --reader-node-path="nitro" \ --reader-node-arguments="--parent-chain.connection.url= --parent-chain.blob-client.beacon-url= --chain.id=42161 --execution.vmtrace.tracer-name=firehose" \ ``` ### Arbitrum Nova ```bash fireeth start reader-node \ --advertise-chain-name="arb-nova" \ --reader-node-path="nitro" \ --reader-node-arguments="--parent-chain.connection.url= --parent-chain.blob-client.beacon-url= --chain.id=42170 --execution.vmtrace.tracer-name=firehose" \ ``` ## Key Nitro Flags | Flag | Description | | ------------------------------------------ | ------------------------------------------------- | | `--execution.vmtrace.tracer-name=firehose` | **Required.** Enables Firehose Protocol output | | `--chain.id` | Chain ID (42161 for Arbitrum One, 42170 for Nova) | | `--parent-chain.connection.url` | Ethereum L1 RPC endpoint | | `--parent-chain.blob-client.beacon-url` | Ethereum L1 beacon client endpoint | ## Resources * [Arbitrum Documentation](https://docs.arbitrum.io/) * [streamingfast/nitro](https://github.com/streamingfast/nitro) * [Arbitrum Node Running](https://docs.arbitrum.io/run-arbitrum-node/run-full-node) # Base Firehose configuration for Base This page provides Base specific configuration. First read the [Ethereum general notes](/firehose/overview/chains/ethereum) for common information. ## Docker Image ``` ghcr.io/streamingfast/go-ethereum:optimism--fh3.0 ``` [View available versions on GitHub Packages](https://github.com/streamingfast/go-ethereum/pkgs/container/go-ethereum) Base is built on the OP Stack, so it uses the same Firehose-patched OP Geth. The image contains both the binary and `fireeth`. ## Client Binary | Client | Repository | | -------------------------- | ----------------------------------------------------------------- | | OP Geth (Firehose-patched) | [streamingfast/op-geth](https://github.com/streamingfast/op-geth) | ## Networks | Network | Chain Name | | ------------ | -------------- | | Base Mainnet | `base-mainnet` | | Base Sepolia | `base-sepolia` | ## OP Node Dependency {% hint style="warning" %} Base requires an OP Node (consensus client) running alongside OP Geth. The OP Node connects to Ethereum L1 for state derivation. Refer to the [Base documentation](https://docs.base.org/guides/run-a-base-node/) for OP Node setup. {% endhint %} ## Reader Node Configuration ### Base Mainnet ```bash fireeth start reader-node \ --advertise-chain-name="base-mainnet" \ --reader-node-path="geth" \ --reader-node-arguments="--datadir={node-data-dir} --networkid=8453 --op-network=base-mainnet --rollup.sequencerhttp=https://mainnet-sequencer.base.org/ --authrpc.addr=0.0.0.0 --authrpc.port=8551 --authrpc.vhosts=* --authrpc.jwtsecret=/path/to/jwt.hex --vmtrace=firehose --state.scheme=path" \ ``` ### Base Sepolia ```bash fireeth start reader-node \ --advertise-chain-name="base-sepolia" \ --reader-node-path="geth" \ --reader-node-arguments="--datadir={node-data-dir} --networkid=84532 --op-network=base-sepolia --rollup.sequencerhttp=https://sepolia-sequencer.base.org/ --authrpc.addr=0.0.0.0 --authrpc.port=8551 --authrpc.vhosts=* --authrpc.jwtsecret=/path/to/jwt.hex --vmtrace=firehose --state.scheme=path" \ ``` ## Key OP Geth Flags | Flag | Description | | ------------------------ | -------------------------------------------------------- | | `--vmtrace=firehose` | **Required.** Enables Firehose Protocol output | | `--op-network` | OP Stack network name (`base-mainnet` or `base-sepolia`) | | `--networkid` | Network ID (8453 for Base Mainnet) | | `--rollup.sequencerhttp` | Sequencer endpoint for transaction submission | | `--authrpc.jwtsecret` | Path to JWT secret for OP Node connection | | `--state.scheme=path` | Recommended state storage scheme | | `--datadir` | Data directory (use `{node-data-dir}` template) | ## Resources * [Base Documentation](https://docs.base.org/) * [Run a Base Node](https://docs.base.org/guides/run-a-base-node/) * [streamingfast/op-geth](https://github.com/streamingfast/op-geth) # BNB Smart Chain Firehose configuration for BNB Smart Chain This page provides BNB Smart Chain (BSC) specific configuration. First read the [Ethereum general notes](/firehose/overview/chains/ethereum) for common information. ## Docker Image ``` ghcr.io/streamingfast/go-ethereum:bnb--fh3.0 ``` [View available versions on GitHub Packages](https://github.com/streamingfast/go-ethereum/pkgs/container/go-ethereum) The image contains both the Firehose-patched BSC Geth binary and `fireeth`. ## Client Binary | Client | Repository | Source Branch | Releases | | --------------------------- | ------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------ | | BSC Geth (Firehose-patched) | [streamingfast/go-ethereum](https://github.com/streamingfast/go-ethereum) | [release/bnb-1.x-fh3.0](https://github.com/streamingfast/go-ethereum/tree/release/bnb-1.x-fh3.0) | [bnb-\*](https://github.com/streamingfast/go-ethereum/releases?q=bnb\&expanded=true) | ## Networks | Network | Chain Name | | ----------- | ------------- | | BSC Mainnet | `bsc-mainnet` | | BSC Testnet | `bsc-testnet` | ## Reader Node Configuration ### BSC Mainnet ```bash fireeth start reader-node \ --advertise-chain-name="bsc-mainnet" \ --reader-node-path="geth" \ --reader-node-arguments="--datadir={node-data-dir} --networkid=56 --syncmode=full --vmtrace=firehose" \ ``` ### BSC Testnet ```bash fireeth start reader-node \ --advertise-chain-name="bsc-testnet" \ --reader-node-path="geth" \ --reader-node-arguments="--datadir={node-data-dir} --networkid=97 --syncmode=full --vmtrace=firehose" \ ``` ## Key BSC Geth Flags | Flag | Description | | -------------------- | ----------------------------------------------- | | `--vmtrace=firehose` | **Required.** Enables Firehose Protocol output | | `--networkid` | Network ID (56 for mainnet, 97 for testnet) | | `--syncmode` | `full` recommended for BSC | | `--datadir` | Data directory (use `{node-data-dir}` template) | ## Resources * [BSC Documentation](https://docs.bnbchain.org/) * [BSC Node Best Practices](https://docs.bnbchain.org/bnb-smart-chain/developers/node_operators/node_best_practices/) * [streamingfast/go-ethereum](https://github.com/streamingfast/go-ethereum) # Katana Firehose configuration for Katana This page provides Katana specific configuration. First read the [Ethereum general notes](/firehose/overview/chains/ethereum) for common information. ## Docker Image ``` ghcr.io/streamingfast/go-ethereum:optimism--fh3.0 ``` [View available versions on GitHub Packages](https://github.com/streamingfast/go-ethereum/pkgs/container/go-ethereum) Katana is built on the OP Stack, so it uses the same Firehose-patched OP Geth. The image contains both the binary and `fireeth`. ## Client Binary | Client | Repository | | -------------------------- | ----------------------------------------------------------------- | | OP Geth (Firehose-patched) | [streamingfast/op-geth](https://github.com/streamingfast/op-geth) | ## Networks | Network | Chain Name | | -------------- | ---------------- | | Katana Mainnet | `katana-mainnet` | ## OP Node Dependency {% hint style="warning" %} Katana requires an OP Node (consensus client) running alongside OP Geth with a custom rollup configuration. The OP Node connects to Ethereum L1 for state derivation. {% endhint %} ## Reader Node Configuration ### Katana Mainnet ```bash fireeth start reader-node \ --advertise-chain-name="katana-mainnet" \ --reader-node-path="geth" \ --reader-node-arguments="--datadir={node-data-dir} --networkid=747474 --rollup.sequencerhttp=https://rpc.katana.network --authrpc.addr=0.0.0.0 --authrpc.port=8551 --authrpc.vhosts=* --authrpc.jwtsecret=/path/to/jwt.hex --vmtrace=firehose --state.scheme=hash" \ ``` ## Key OP Geth Flags | Flag | Description | | ------------------------ | ----------------------------------------------- | | `--vmtrace=firehose` | **Required.** Enables Firehose Protocol output | | `--networkid` | Network ID (747474 for Katana Mainnet) | | `--rollup.sequencerhttp` | Sequencer endpoint for transaction submission | | `--authrpc.jwtsecret` | Path to JWT secret for OP Node connection | | `--state.scheme=hash` | State storage scheme (Katana uses `hash`) | | `--datadir` | Data directory (use `{node-data-dir}` template) | ## Resources * [Katana Network](https://katana.network/) * [streamingfast/op-geth](https://github.com/streamingfast/op-geth) # Optimism Firehose configuration for Optimism (OP Mainnet) This page provides Optimism (OP Mainnet) specific configuration. First read the [Ethereum general notes](/firehose/overview/chains/ethereum) for common information. ## Docker Image ``` ghcr.io/streamingfast/go-ethereum:optimism--fh3.0 ``` [View available versions on GitHub Packages](https://github.com/streamingfast/go-ethereum/pkgs/container/go-ethereum) The image contains both the Firehose-patched OP Geth binary and `fireeth`. ## Client Binary | Client | Repository | Source Branch | Releases | | -------------------------- | ------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------- | | OP Geth (Firehose-patched) | [streamingfast/go-ethereum](https://github.com/streamingfast/go-ethereum) | [release/optimism-1.x-fh3.0](https://github.com/streamingfast/go-ethereum/tree/release/optimism-1.x-fh3.0) | [optimism-\*](https://github.com/streamingfast/go-ethereum/releases?q=optimism\&expanded=true) | ## Networks | Network | Chain Name | | ---------- | ------------------ | | OP Mainnet | `optimism-mainnet` | | OP Sepolia | `optimism-sepolia` | ## OP Node Dependency {% hint style="warning" %} Optimism requires an OP Node (consensus client) running alongside OP Geth. The OP Node connects to Ethereum L1 for state derivation. Refer to the [Optimism documentation](https://docs.optimism.io/builders/node-operators/rollup-node) for OP Node setup. {% endhint %} ## Reader Node Configuration ### OP Mainnet ```bash fireeth start reader-node \ --advertise-chain-name="optimism-mainnet" \ --reader-node-path="geth" \ --reader-node-arguments="--datadir={node-data-dir} --networkid=10 --op-network=op-mainnet --rollup.sequencerhttp=https://mainnet-sequencer.optimism.io/ --authrpc.addr=0.0.0.0 --authrpc.port=8551 --authrpc.vhosts=* --authrpc.jwtsecret=/path/to/jwt.hex --vmtrace=firehose --state.scheme=path" \ ``` ### OP Sepolia ```bash fireeth start reader-node \ --advertise-chain-name="optimism-sepolia" \ --reader-node-path="geth" \ --reader-node-arguments="--datadir={node-data-dir} --networkid=11155420 --op-network=op-sepolia --rollup.sequencerhttp=https://sepolia-sequencer.optimism.io/ --authrpc.addr=0.0.0.0 --authrpc.port=8551 --authrpc.vhosts=* --authrpc.jwtsecret=/path/to/jwt.hex --vmtrace=firehose --state.scheme=path" \ ``` ## Key OP Geth Flags | Flag | Description | | ------------------------ | ----------------------------------------------- | | `--vmtrace=firehose` | **Required.** Enables Firehose Protocol output | | `--op-network` | OP Stack network name | | `--networkid` | Network ID (10 for OP Mainnet) | | `--rollup.sequencerhttp` | Sequencer endpoint for transaction submission | | `--authrpc.jwtsecret` | Path to JWT secret for OP Node connection | | `--state.scheme=path` | Recommended state storage scheme | | `--datadir` | Data directory (use `{node-data-dir}` template) | ## Resources * [Optimism Documentation](https://docs.optimism.io/) * [OP Node Setup](https://docs.optimism.io/builders/node-operators/rollup-node) * [streamingfast/go-ethereum](https://github.com/streamingfast/go-ethereum) # Polygon Firehose configuration for Polygon PoS This page provides Polygon PoS specific configuration. First read the [Ethereum general notes](/firehose/overview/chains/ethereum) for common information. ## Docker Image ``` ghcr.io/streamingfast/go-ethereum:polygon--fh3.0 ``` [View available versions on GitHub Packages](https://github.com/streamingfast/go-ethereum/pkgs/container/go-ethereum) The image contains both the Firehose-patched Bor binary and `fireeth`. ## Client Binary | Client | Repository | Source Branch | Releases | | ---------------------- | ------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------- | | Bor (Firehose-patched) | [streamingfast/go-ethereum](https://github.com/streamingfast/go-ethereum) | [release/polygon-2.x-fh3.0](https://github.com/streamingfast/go-ethereum/tree/release/polygon-2.x-fh3.0) | [polygon-\*](https://github.com/streamingfast/go-ethereum/releases?q=polygon\&expanded=true) | ## Networks | Network | Chain Name | | --------------- | ----------------- | | Polygon Mainnet | `polygon-mainnet` | | Polygon Amoy | `polygon-amoy` | ## Heimdall Dependency {% hint style="warning" %} Polygon PoS requires a Heimdall node running alongside Bor. Heimdall provides checkpoint validation. You must run your own Heimdall node. Refer to the [Polygon documentation](https://docs.polygon.technology/pos/how-to/full-node/) for Heimdall setup instructions. {% endhint %} ## Reader Node Configuration ### Polygon Mainnet ```bash fireeth start reader-node \ --advertise-chain-name="polygon-mainnet" \ --reader-node-path="bor" \ --reader-node-arguments="server --chain=mainnet --config=/path/to/config.toml --datadir={node-data-dir} --bor.heimdall=http://heimdall:1317 --syncmode=full --vmtrace=firehose" \ ``` ### Polygon Amoy Testnet ```bash fireeth start reader-node \ --advertise-chain-name="polygon-amoy" \ --reader-node-path="bor" \ --reader-node-arguments="server --chain=amoy --config=/path/to/config.toml --datadir={node-data-dir} --bor.heimdall=http://heimdall:1317 --syncmode=full --vmtrace=firehose" \ ``` ## Key Bor Flags | Flag | Description | | -------------------- | --------------------------------------------------------- | | `server` | Bor subcommand to start the node | | `--vmtrace=firehose` | **Required.** Enables Firehose Protocol output | | `--syncmode` | `full` recommended for Polygon | | `--chain` | `mainnet` or `amoy` | | `--config` | Path to Bor configuration TOML file | | `--bor.heimdall` | Heimdall REST API endpoint (e.g., `http://heimdall:1317`) | | `--datadir` | Data directory (use `{node-data-dir}` template) | ## Resources * [Polygon Documentation](https://docs.polygon.technology/) * [Polygon Node Requirements](https://docs.polygon.technology/pos/how-to/full-node/) * [streamingfast/go-ethereum](https://github.com/streamingfast/go-ethereum) # Unichain Firehose configuration for Unichain This page provides Unichain specific configuration. First read the [Ethereum general notes](/firehose/overview/chains/ethereum) for common information. ## Docker Image ``` ghcr.io/streamingfast/go-ethereum:optimism--fh3.0 ``` [View available versions on GitHub Packages](https://github.com/streamingfast/go-ethereum/pkgs/container/go-ethereum) Unichain is built on the OP Stack, so it uses the same Firehose-patched OP Geth. The image contains both the binary and `fireeth`. ## Client Binary | Client | Repository | | -------------------------- | ----------------------------------------------------------------- | | OP Geth (Firehose-patched) | [streamingfast/op-geth](https://github.com/streamingfast/op-geth) | ## Networks | Network | Chain Name | | ---------------- | ------------------ | | Unichain Mainnet | `unichain-mainnet` | ## OP Node Dependency {% hint style="warning" %} Unichain requires an OP Node (consensus client) running alongside OP Geth. The OP Node connects to Ethereum L1 for state derivation. Refer to the [Unichain documentation](https://docs.unichain.org/) for OP Node setup. {% endhint %} ## Reader Node Configuration ### Unichain Mainnet ```bash fireeth start reader-node \ --advertise-chain-name="unichain-mainnet" \ --reader-node-path="geth" \ --reader-node-arguments="--datadir={node-data-dir} --networkid=130 --op-network=unichain-mainnet --rollup.sequencerhttp=https://mainnet-sequencer.unichain.org --authrpc.addr=0.0.0.0 --authrpc.port=8551 --authrpc.vhosts=* --authrpc.jwtsecret=/path/to/jwt.hex --vmtrace=firehose --state.scheme=path" \ ``` ## Key OP Geth Flags | Flag | Description | | ------------------------ | ----------------------------------------------- | | `--vmtrace=firehose` | **Required.** Enables Firehose Protocol output | | `--op-network` | OP Stack network name (`unichain-mainnet`) | | `--networkid` | Network ID (130 for Unichain Mainnet) | | `--rollup.sequencerhttp` | Sequencer endpoint for transaction submission | | `--authrpc.jwtsecret` | Path to JWT secret for OP Node connection | | `--state.scheme=path` | Recommended state storage scheme | | `--datadir` | Data directory (use `{node-data-dir}` template) | ## Resources * [Unichain Documentation](https://docs.unichain.org/) * [streamingfast/op-geth](https://github.com/streamingfast/op-geth) # Worldchain Firehose configuration for Worldchain This page provides Worldchain specific configuration. First read the [Ethereum general notes](/firehose/overview/chains/ethereum) for common information. ## Docker Image ``` ghcr.io/streamingfast/go-ethereum:optimism--fh3.0 ``` [View available versions on GitHub Packages](https://github.com/streamingfast/go-ethereum/pkgs/container/go-ethereum) Worldchain is built on the OP Stack, so it uses the same Firehose-patched OP Geth. The image contains both the binary and `fireeth`. ## Client Binary | Client | Repository | | -------------------------- | ----------------------------------------------------------------- | | OP Geth (Firehose-patched) | [streamingfast/op-geth](https://github.com/streamingfast/op-geth) | ## Networks | Network | Chain Name | | ------------------ | -------------------- | | Worldchain Mainnet | `worldchain-mainnet` | ## OP Node Dependency {% hint style="warning" %} Worldchain requires an OP Node (consensus client) running alongside OP Geth. The OP Node connects to Ethereum L1 for state derivation. {% endhint %} ## Reader Node Configuration ### Worldchain Mainnet ```bash fireeth start reader-node \ --advertise-chain-name="worldchain-mainnet" \ --reader-node-path="geth" \ --reader-node-arguments="--datadir={node-data-dir} --op-network=worldchain-mainnet --authrpc.addr=0.0.0.0 --authrpc.port=8551 --authrpc.vhosts=* --authrpc.jwtsecret=/path/to/jwt.hex --vmtrace=firehose --state.scheme=path" \ ``` ## Key OP Geth Flags | Flag | Description | | --------------------- | ----------------------------------------------- | | `--vmtrace=firehose` | **Required.** Enables Firehose Protocol output | | `--op-network` | OP Stack network name (`worldchain-mainnet`) | | `--authrpc.jwtsecret` | Path to JWT secret for OP Node connection | | `--state.scheme=path` | Recommended state storage scheme | | `--datadir` | Data directory (use `{node-data-dir}` template) | ## Resources * [World Documentation](https://world.org/developers) * [streamingfast/op-geth](https://github.com/streamingfast/op-geth) # Avalanche Firehose chain-specific configuration for Avalanche C-Chain This page covers Reader Node configuration specific to Avalanche C-Chain. For general Firehose architecture and deployment, see the [Single Machine Deployment](/firehose/overview/single-machine-deployment) or [Distributed Deployment](/firehose/overview/distributed-deployment) guides. {% hint style="warning" %} Firehose for Avalanche uses an RPC poller approach. You can either run your own Avalanche node or use an RPC provider. For node setup, refer to the [official Avalanche documentation](https://docs.avax.network/). {% endhint %} ## Docker Image ``` ghcr.io/streamingfast/go-ethereum:geth--fh3.0 ``` [View available versions on GitHub Packages](https://github.com/streamingfast/go-ethereum/pkgs/container/go-ethereum) The image contains the `fireeth` binary with the RPC poller. ## Binary & Releases | Component | Repository | Binary | | --------- | ----------------------------------------------------------------------- | --------- | | Firehose | [firehose-ethereum](https://github.com/streamingfast/firehose-ethereum) | `fireeth` | Download releases from the [GitHub releases page](https://github.com/streamingfast/firehose-ethereum/releases). ## Networks | Network | Chain Name | | ----------------- | ------------------- | | Avalanche C-Chain | `avalanche-mainnet` | ## Architecture Firehose for Avalanche uses an **RPC poller** approach. The poller fetches blocks from Avalanche C-Chain RPC endpoints (which are EVM-compatible) and converts them to Firehose format. ``` ┌──────────────────┐ RPC ┌──────────────────┐ stdout ┌──────────────┐ │ Avalanche RPC │◄────────────│ fireeth │──────────────►│ Reader Node │ │ Endpoint │ │ poller │ │ (Firehose) │ └──────────────────┘ └──────────────────┘ └──────────────┘ ``` ## Reader Node Configuration ### Avalanche C-Chain Mainnet ```bash fireeth start reader-node \ --advertise-chain-name="avalanche-mainnet" \ --common-first-streamable-block= \ --reader-node-path="fireeth" \ --reader-node-arguments="tools poller optimism {first-streamable-block}" \ ``` ## Key Poller Flags | Flag | Description | | -------------------------- | ----------------------------------------------------------- | | `tools poller optimism` | Subcommand to run the EVM RPC poller | | `` | Avalanche C-Chain RPC endpoint URL | | `{first-streamable-block}` | Variable substituted from `--common-first-streamable-block` | ## Resources * [Avalanche Documentation](https://docs.avax.network/) * [Avalanche C-Chain RPC](https://docs.avax.network/apis/avalanchego/apis/c-chain) * [firehose-ethereum GitHub](https://github.com/streamingfast/firehose-ethereum) # Injective Firehose chain-specific configuration for Injective This page covers Reader Node configuration specific to Injective. For general Firehose architecture and deployment, see the [Single Machine Deployment](/firehose/overview/single-machine-deployment) or [Distributed Deployment](/firehose/overview/distributed-deployment) guides. {% hint style="warning" %} Firehose for Injective uses an RPC poller approach. You can either run your own Injective node or use an RPC provider. For node setup, refer to the [official Injective documentation](https://docs.injective.network/). {% endhint %} ## Docker Image ``` ghcr.io/streamingfast/firehose-cosmos: ``` [View available versions on GitHub Packages](https://github.com/streamingfast/firehose-cosmos/pkgs/container/firehose-cosmos) The image contains the `firecore` and `fireinjective` binaries. ## Binary & Releases | Component | Repository | Binary | | --------- | ------------------------------------------------------------------- | --------------------------- | | Firehose | [firehose-cosmos](https://github.com/streamingfast/firehose-cosmos) | `firecore`, `fireinjective` | Download releases from the [GitHub releases page](https://github.com/streamingfast/firehose-cosmos/releases). ## Networks | Network | Chain Name | | ----------------- | ------------------- | | Injective Mainnet | `injective-mainnet` | | Injective Testnet | `injective-testnet` | ## Architecture Firehose for Injective uses an **RPC poller** approach. The poller fetches blocks from Injective RPC endpoints and converts them to Firehose format. ``` ┌──────────────────┐ RPC ┌──────────────────┐ stdout ┌──────────────┐ │ Injective RPC │◄──────────── │ fireinjective │──────────────►│ Reader Node │ │ Endpoint │ │ poller │ │ (Firehose) │ └──────────────────┘ └──────────────────┘ └──────────────┘ ``` ## Reader Node Configuration ### Injective Mainnet ```bash firecore start reader-node \ --advertise-chain-name="injective-mainnet" \ --common-first-streamable-block= \ --reader-node-path="fireinjective" \ --reader-node-arguments="fetch rpc {first-streamable-block} --state-dir={node-data-dir}/poller/states --block-fetch-batch-size=4 --endpoints=" \ ``` ### Injective Testnet ```bash firecore start reader-node \ --reader-node-path="fireinjective" \ --reader-node-arguments="fetch rpc {first-streamable-block} --state-dir={node-data-dir}/poller/states --block-fetch-batch-size=1 --endpoints=" \ --common-first-streamable-block= \ --advertise-chain-name="injective-testnet" \ ``` ## Key Poller Flags | Flag | Description | | -------------------------- | -------------------------------------------------------------- | | `fetch rpc` | Subcommand to run the RPC poller | | `{first-streamable-block}` | Variable substituted from `--common-first-streamable-block` | | `--state-dir` | Directory to store poller state | | `--block-fetch-batch-size` | Number of blocks to fetch in parallel | | `--endpoints` | Injective RPC endpoint URL(s), can be specified multiple times | ## Resources * [Injective Documentation](https://docs.injective.network/) * [firehose-cosmos GitHub](https://github.com/streamingfast/firehose-cosmos) # NEAR Firehose chain-specific configuration for NEAR This page covers Reader Node configuration specific to NEAR. For general Firehose architecture and deployment, see the [Single Machine Deployment](/firehose/overview/single-machine-deployment) or [Distributed Deployment](/firehose/overview/distributed-deployment) guides. {% hint style="warning" %} This guide does not cover how to run a NEAR node. For node setup, hardware requirements, and network configuration, refer to the [official NEAR documentation](https://docs.near.org/). {% endhint %} ## Docker Image ``` ghcr.io/streamingfast/near-firehose-indexer:-fh3.0 ``` [View available versions on GitHub Packages](https://github.com/streamingfast/near-firehose-indexer/pkgs/container/near-firehose-indexer) The image contains both the Firehose-patched NEAR indexer binary and `firenear`. ## Binary & Releases | Component | Repository | Binary | | ------------ | ------------------------------------------------------------------------------- | ----------------------- | | Firehose | [firehose-near](https://github.com/streamingfast/firehose-near) | `firenear` | | NEAR Indexer | [near-firehose-indexer](https://github.com/streamingfast/near-firehose-indexer) | `near-firehose-indexer` | Download releases from the GitHub releases pages. ## Networks | Network | Chain Name | | ------------ | -------------- | | NEAR Mainnet | `near-mainnet` | | NEAR Testnet | `near-testnet` | ## Reader Node Configuration NEAR Firehose uses a custom indexer binary (`near-firehose-indexer`) that wraps the NEAR node with Firehose instrumentation. ### NEAR Mainnet ```bash firenear start reader-node \ --advertise-chain-name="near-mainnet" \ --reader-node-path="near-firehose-indexer" \ --reader-node-config-file=/path/to/config.json \ --reader-node-genesis-file=/path/to/genesis.json \ --reader-node-key-file=/path/to/node_key.json \ --reader-node-arguments="--home={node-data-dir} run" \ ``` ### NEAR Testnet ```bash firenear start reader-node \ --advertise-chain-name="near-testnet" \ --reader-node-path="near-firehose-indexer" \ --reader-node-config-file=/path/to/config.json \ --reader-node-genesis-file=/path/to/genesis.json \ --reader-node-key-file=/path/to/node_key.json \ --reader-node-arguments="--home={node-data-dir} run" \ ``` ## Key Configuration Files NEAR requires several configuration files: | File | Description | | --------------- | ----------------------- | | `config.json` | NEAR node configuration | | `genesis.json` | Network genesis file | | `node_key.json` | Node identity key | Download network-specific configuration files from the [NEAR documentation](https://docs.near.org/). ## Key Reader Node Flags | Flag | Description | | ---------------------------- | ---------------------------------------------------- | | `--reader-node-config-file` | Path to NEAR config.json | | `--reader-node-genesis-file` | Path to NEAR genesis.json | | `--reader-node-key-file` | Path to NEAR node\_key.json | | `--home` | Node data directory (use `{node-data-dir}` template) | ## Resources * [firehose-near GitHub](https://github.com/streamingfast/firehose-near) * [near-firehose-indexer GitHub](https://github.com/streamingfast/near-firehose-indexer) * [NEAR Documentation](https://docs.near.org/) # Sei Firehose chain-specific configuration for Sei This page covers Reader Node configuration specific to Sei. For general Firehose architecture and deployment, see the [Single Machine Deployment](/firehose/overview/single-machine-deployment) or [Distributed Deployment](/firehose/overview/distributed-deployment) guides. {% hint style="warning" %} This guide does not cover how to run a Sei node. For node setup, hardware requirements, and network configuration, refer to the [official Sei documentation](https://www.sei.io/developers). {% endhint %} ## Docker Image ``` ghcr.io/streamingfast/sei-chain:-fh3.0 ``` [View available versions on GitHub Packages](https://github.com/streamingfast/sei-chain/pkgs/container/sei-chain) The image contains both the Firehose-patched Sei binary and `fireeth`. ## Binary & Releases | Component | Repository | Binary | | ---------------------- | ----------------------------------------------------------------------- | --------- | | Firehose | [firehose-ethereum](https://github.com/streamingfast/firehose-ethereum) | `fireeth` | | Sei (Firehose-patched) | [streamingfast/sei-chain](https://github.com/streamingfast/sei-chain) | `seid` | Download releases from the GitHub releases pages. ## Networks | Network | Chain Name | | ------------------------ | ------------- | | Sei Mainnet (Pacific-1) | `sei-mainnet` | | Sei Testnet (Atlantic-2) | `sei-testnet` | ## Architecture Sei is a Cosmos SDK-based blockchain with an integrated EVM layer. Firehose extracts EVM execution traces from the Sei node. ## Reader Node Configuration ### Sei Mainnet ```bash fireeth start reader-node \ --advertise-chain-name="sei-mainnet" \ --reader-node-path="seid" \ --reader-node-arguments="start --home={node-data-dir} --trace --chain-id=pacific-1" \ ``` ### Sei Testnet ```bash fireeth start reader-node \ --advertise-chain-name="sei-testnet" \ --reader-node-path="seid" \ --reader-node-arguments="start --home={node-data-dir} --trace --chain-id=atlantic-2" \ ``` ## Key Sei Flags | Flag | Description | | ------------ | ------------------------------------------------------------ | | `start` | Sei subcommand to start the node | | `--trace` | **Required.** Enables Firehose Protocol output | | `--chain-id` | Chain ID (`pacific-1` for mainnet, `atlantic-2` for testnet) | | `--home` | Node data directory (use `{node-data-dir}` template) | ## Resources * [Sei Documentation](https://www.sei.io/developers) * [streamingfast/sei-chain](https://github.com/streamingfast/sei-chain) # Solana Firehose chain-specific configuration for Solana This page covers Reader Node configuration specific to Solana. For general Firehose architecture and deployment, see the [Single Machine Deployment](/firehose/overview/single-machine-deployment) or [Distributed Deployment](/firehose/overview/distributed-deployment) guides. {% hint style="warning" %} This guide does not cover how to run a Solana validator. For validator setup, hardware requirements, and network configuration, refer to the [official Solana documentation](https://docs.solana.com/). {% endhint %} ## Architecture Firehose for Solana uses a **Geyser plugin** approach. The plugin hooks into the Solana validator and emits Firehose Protocol data through a gRPC interface. ``` ┌──────────────────────────────────────────────────────┐ │ Solana Validator │ │ ┌────────────────────────────────────────────────┐ │ │ │ Firehose Geyser Plugin │ │ │ │ (emits blocks via gRPC on port 10015) │ │ │ └────────────────────────────────────────────────┘ │ └──────────────────────────────────────────────────────┘ │ ▼ gRPC ┌──────────────────────────────────────────────────────┐ │ firesolana (reader-node-firehose mode) │ │ Consumes blocks and produces one-block files │ └──────────────────────────────────────────────────────┘ ``` ## Docker Image ``` ghcr.io/streamingfast/firehose-geyser-plugin: ``` [View available versions on GitHub Packages](https://github.com/streamingfast/firehose-geyser-plugin/pkgs/container/firehose-geyser-plugin) The image contains the Solana validator with the Firehose Geyser plugin pre-installed. ## Binary & Releases | Component | Repository | Binary | | ------------- | --------------------------------------------------------------------------------- | -------------- | | Firehose | [firehose-solana](https://github.com/streamingfast/firehose-solana) | `firesolana` | | Geyser Plugin | [firehose-geyser-plugin](https://github.com/streamingfast/firehose-geyser-plugin) | Shared library | Download releases from the GitHub releases pages. ## Networks | Network | Chain Name | | -------------- | --------------------- | | Solana Mainnet | `solana-mainnet-beta` | | Solana Devnet | `solana-devnet` | | Solana Testnet | `solana-testnet` | ## Reader Node Configuration Solana Firehose uses `reader-node-firehose` mode, which connects to the Geyser plugin's gRPC endpoint instead of spawning a subprocess. ### Reader Node (Firehose Mode) ```bash firesolana start reader-node-firehose \ --reader-node-firehose-grpc-listen-addr= \ --advertise-chain-name="solana-mainnet-beta" \ ``` ## Geyser Plugin Configuration The Geyser plugin is configured via a JSON file passed to the Solana validator. The plugin emits Firehose data on a gRPC endpoint. ### Example Plugin Configuration ```json { "libpath": "/path/to/libfirehose_geyser_plugin.so", "blocks_grpc_listen_addr": "0.0.0.0:10015", "accounts_grpc_listen_addr": "0.0.0.0:10016" } ``` ### Starting the Validator ```bash solana-validator \ --geyser-plugin-config /path/to/firehose-geyser-config.json \ ``` ## Resources * [firehose-solana GitHub](https://github.com/streamingfast/firehose-solana) * [firehose-geyser-plugin GitHub](https://github.com/streamingfast/firehose-geyser-plugin) * [Solana Documentation](https://docs.solana.com/) * [Solana Geyser Plugin Interface](https://docs.solana.com/developing/plugins/geyser-plugins) # Starknet Firehose chain-specific configuration for Starknet This page covers Reader Node configuration specific to Starknet. For general Firehose architecture and deployment, see the [Single Machine Deployment](/firehose/overview/single-machine-deployment) or [Distributed Deployment](/firehose/overview/distributed-deployment) guides. {% hint style="warning" %} Firehose for Starknet uses an RPC poller approach. You can either run your own Starknet node or use an RPC provider. For node setup, refer to the [official Starknet documentation](https://docs.starknet.io/). {% endhint %} ## Docker Image ``` ghcr.io/streamingfast/firehose-starknet: ``` [View available versions on GitHub Packages](https://github.com/streamingfast/firehose-starknet/pkgs/container/firehose-starknet) The image contains the `firecore` and `firestarknet` binaries. ## Binary & Releases | Component | Repository | Binary | | --------- | ----------------------------------------------------------------------- | -------------------------- | | Firehose | [firehose-starknet](https://github.com/streamingfast/firehose-starknet) | `firecore`, `firestarknet` | Download releases from the [GitHub releases page](https://github.com/streamingfast/firehose-starknet/releases). ## Networks | Network | Chain Name | | ---------------- | ------------------ | | Starknet Mainnet | `starknet-mainnet` | | Starknet Sepolia | `starknet-sepolia` | ## Architecture Firehose for Starknet uses an **RPC poller** approach. The poller fetches blocks from Starknet RPC endpoints and converts them to Firehose format. It also requires an Ethereum L1 endpoint for finality information. ``` ┌──────────────────┐ RPC ┌──────────────────┐ stdout ┌──────────────┐ │ Starknet RPC │◄────────────│ firestarknet │──────────────►│ Reader Node │ │ Endpoint │ │ poller │ │ (Firehose) │ └──────────────────┘ └──────────────────┘ └──────────────┘ ▲ │ │ │ ┌──────────────────┐ │ │ Ethereum L1 │◄────────────────────┘ │ Endpoint │ (for finality) └──────────────────┘ ``` ## Reader Node Configuration ### Starknet Mainnet ```bash firecore start reader-node \ --advertise-chain-name="starknet-mainnet" \ --common-first-streamable-block= \ --reader-node-path="firestarknet" \ --reader-node-arguments="fetch {first-streamable-block} --state-dir={node-data-dir}/poller/states --starknet-endpoints= --eth-endpoints=" \ ``` ### Starknet Sepolia ```bash firecore start reader-node \ --advertise-chain-name="starknet-sepolia" \ --common-first-streamable-block= \ --reader-node-path="firestarknet" \ --reader-node-arguments="fetch {first-streamable-block} --state-dir={node-data-dir}/poller/states --starknet-endpoints= --eth-endpoints=" \ ``` ## Key Poller Flags | Flag | Description | | -------------------------- | ----------------------------------------------------------- | | `fetch` | Subcommand to run the RPC poller | | `{first-streamable-block}` | Variable substituted from `--common-first-streamable-block` | | `--state-dir` | Directory to store poller state | | `--starknet-endpoints` | Starknet RPC endpoint URL | | `--eth-endpoints` | Ethereum L1 RPC endpoint for finality | | `--block-fetch-batch-size` | Number of blocks to fetch in parallel | | `--interval-between-fetch` | Delay between fetch cycles | ## Resources * [Starknet Documentation](https://docs.starknet.io/) * [firehose-starknet GitHub](https://github.com/streamingfast/firehose-starknet) # Stellar Firehose chain-specific configuration for Stellar This page covers Reader Node configuration specific to Stellar. For general Firehose architecture and deployment, see the [Single Machine Deployment](/firehose/overview/single-machine-deployment) or [Distributed Deployment](/firehose/overview/distributed-deployment) guides. {% hint style="warning" %} Firehose for Stellar uses an RPC poller approach. You can either run your own Stellar/Soroban RPC node or use an RPC provider. For node setup, refer to the [official Stellar documentation](https://developers.stellar.org/). {% endhint %} ## Docker Image ``` ghcr.io/streamingfast/firehose-stellar: ``` [View available versions on GitHub Packages](https://github.com/streamingfast/firehose-stellar/pkgs/container/firehose-stellar) The image contains the `firecore` and `firestellar` binaries. ## Binary & Releases | Component | Repository | Binary | | --------- | --------------------------------------------------------------------- | ------------------------- | | Firehose | [firehose-stellar](https://github.com/streamingfast/firehose-stellar) | `firecore`, `firestellar` | Download releases from the [GitHub releases page](https://github.com/streamingfast/firehose-stellar/releases). ## Networks | Network | Chain Name | | --------------- | ----------------- | | Stellar Mainnet | `stellar-mainnet` | | Stellar Testnet | `stellar-testnet` | ## Architecture Firehose for Stellar uses an **RPC poller** approach. The poller fetches ledgers from Stellar Soroban RPC endpoints and converts them to Firehose format. ``` ┌──────────────────┐ RPC ┌──────────────────┐ stdout ┌──────────────┐ │ Soroban RPC │◄────────────│ firestellar │──────────────►│ Reader Node │ │ Endpoint │ │ poller │ │ (Firehose) │ └──────────────────┘ └──────────────────┘ └──────────────┘ ``` ## Reader Node Configuration ### Stellar Mainnet ```bash firecore start reader-node \ --advertise-chain-name="stellar-mainnet" \ --common-first-streamable-block= \ --reader-node-path="firestellar" \ --reader-node-arguments="fetch rpc {first-streamable-block} --state-dir={node-data-dir}/poller/states --endpoints=" \ ``` ### Stellar Testnet ```bash firecore start reader-node \ --reader-node-path="firestellar" \ --reader-node-arguments="fetch rpc {first-streamable-block} --state-dir={node-data-dir}/poller/states --endpoints= --is-mainnet=false" \ --common-first-streamable-block= \ --advertise-chain-name="stellar-testnet" \ ``` ## Key Poller Flags | Flag | Description | | -------------------------- | ------------------------------------------------------------ | | `fetch rpc` | Subcommand to run the RPC poller | | `{first-streamable-block}` | Variable substituted from `--common-first-streamable-block` | | `--state-dir` | Directory to store poller state | | `--endpoints` | Soroban RPC endpoint URL(s), can be specified multiple times | | `--block-fetch-batch-size` | Number of ledgers to fetch in parallel | | `--is-mainnet` | Set to `false` for testnet (default `true`) | ## Resources * [Stellar Documentation](https://developers.stellar.org/) * [Soroban RPC](https://developers.stellar.org/docs/data/rpc) * [firehose-stellar GitHub](https://github.com/streamingfast/firehose-stellar) # Tron Firehose chain-specific configuration for Tron This page covers Reader Node configuration specific to Tron. For general Firehose architecture and deployment, see the [Single Machine Deployment](/firehose/overview/single-machine-deployment) or [Distributed Deployment](/firehose/overview/distributed-deployment) guides. {% hint style="warning" %} Firehose for Tron uses an RPC poller approach. You can either run your own Tron node or use an RPC provider like TronGrid. For node setup, refer to the [official Tron documentation](https://developers.tron.network/). {% endhint %} ## Docker Image ``` ghcr.io/streamingfast/firehose-tron: ``` [View available versions on GitHub Packages](https://github.com/streamingfast/firehose-tron/pkgs/container/firehose-tron) The image contains the `firecore` and `firetron` binaries. ## Binary & Releases | Component | Repository | Binary | | --------- | --------------------------------------------------------------- | ---------------------- | | Firehose | [firehose-tron](https://github.com/streamingfast/firehose-tron) | `firecore`, `firetron` | Download releases from the [GitHub releases page](https://github.com/streamingfast/firehose-tron/releases). ## Networks | Network | Chain Name | | ------------------ | ------------------ | | Tron Mainnet | `tron-mainnet` | | Tron Mainnet (EVM) | `tron-evm-mainnet` | ## Architecture Firehose for Tron uses an **RPC poller** approach. The poller fetches blocks from Tron gRPC and/or JSON-RPC endpoints and converts them to Firehose format. Tron supports two block formats: * **Native Tron blocks**: Full Tron protocol data via gRPC * **EVM-compatible blocks**: Ethereum-style blocks via JSON-RPC for EVM compatibility ``` ┌──────────────────┐ gRPC ┌──────────────────┐ stdout ┌──────────────┐ │ Tron Node │◄──────────── │ firetron │──────────────►│ Reader Node │ │ (gRPC/JSON-RPC) │ │ poller │ │ (Firehose) │ └──────────────────┘ └──────────────────┘ └──────────────┘ ``` ## Reader Node Configuration ### Tron Mainnet (Native) ```bash firecore start reader-node \ --advertise-chain-name="tron-mainnet" \ --common-first-streamable-block= \ --reader-node-path="firetron" \ --reader-node-arguments="fetch {first-streamable-block} --state-dir={node-data-dir}/poller/states --tron-endpoints=" \ ``` ### Tron Mainnet (EVM) For EVM-compatible block format: ```bash firecore start reader-node \ --advertise-chain-name="tron-evm-mainnet" \ --common-first-streamable-block= \ --reader-node-path="firetron" \ --reader-node-arguments="fetch-evm {first-streamable-block} --state-dir={node-data-dir}/poller/states --tron-evm-endpoints= --tron-endpoints=" \ ``` ## Key Poller Flags | Flag | Description | | -------------------------- | ----------------------------------------------------------- | | `fetch` | Subcommand to fetch native Tron blocks | | `fetch-evm` | Subcommand to fetch EVM-compatible blocks | | `{first-streamable-block}` | Variable substituted from `--common-first-streamable-block` | | `--state-dir` | Directory to store poller state | | `--tron-endpoints` | Tron gRPC endpoint (e.g., `grpc.trongrid.io:50051`) | | `--tron-evm-endpoints` | Tron JSON-RPC endpoint for EVM blocks | | `--tron-api-key` | API key for TronGrid access | | `--block-fetch-batch-size` | Number of blocks to fetch in parallel | | `--interval-between-fetch` | Delay between fetch cycles | ## Resources * [Tron Documentation](https://developers.tron.network/) * [TronGrid API](https://www.trongrid.io/) * [firehose-tron GitHub](https://github.com/streamingfast/firehose-tron) # Benefits Why integrate StreamingFast Firehose? From a Layer 1 blockchain core development team's perspective, integrating Firehose into your chain brings you and your community the following benefits. * Immediate support for [Substreams](https://substreams.streamingfast.io/), and its high-speed parallel processing engine. * Immediate integration into the latest `graph-node` which can feed directly from Firehose. * Immediate support for dozens of sinks (think loading to PostgreSQL, MongoDB, CDC replication, Kafka, Flink, writing to buckets for ingestion by BigQuery, AWS Redshift, Clickhouse, etc.). * Activates a community of developers that already know how to index blockchain data, and are going to be happy to discover your blockchain’s data model and activity. From a Layer 1’s users’ perspective, you’ll get the following benefits. * The capability to quickly index and sift through your favorite blockchain’s history. * Stream its output with low latency. * Hook it to any data systems you have, trading bots, cross-chain bridges, databases. * All of it in an extremely reliable way. * Benefit from the anti-fragility brought by The Graph network. * Offer an army of indexers to service your network. # Integration Overview StreamingFast Firehose new blockchains ## Introduction Firehose is blockchain agnostic. It requires two main elements: * **Node instrumentation**: The blockchain node must output [Firehose Protocol](/references/firehose-protocol) messages — a unified, chain-agnostic protocol consisting of `FIRE INIT` and `FIRE BLOCK` messages * **Protocol Buffers data model**: A protobuf schema defining your chain's block structure The [Firehose Protocol](/references/firehose-protocol) is intentionally simple: the node outputs block data as base64-encoded protobuf, and `firehose-core` handles everything else (parsing, storage, streaming, flat files management). No chain-specific Go code is required for the core functionality. ## Methods of extraction ### Node instrumentation The recommended method of extraction is direct node instrumentation, where the blockchain node outputs the [Firehose Protocol](/references/firehose-protocol). This provides the richest data extraction, including state changes and total ordering of events. The protocol is simple: 1. Output `FIRE INIT ` once at startup 2. Output `FIRE BLOCK ` for each block See the [Firehose Protocol Reference](/references/firehose-protocol) for the complete specification, including block hash format requirements and partial block support. See the [dummy-blockchain tracer](https://github.com/streamingfast/dummy-blockchain/tree/main/tracer) for a reference implementation. **Examples of node instrumentation:** * Ethereum `go-ethereum`: Uses Geth's [Live Tracing](https://geth.ethereum.org/docs/developers/evm-tracing/live-tracing) feature to extract block data. See the [go-ethereum Firehose integration](https://github.com/streamingfast/go-ethereum/blob/firehose-fh3.0/eth/tracers/firehose.go#L75) for the implementation. For EVM chains, this is the recommended method, as this provides drop-in support for all the Substreams already built for Ethereum. Alternative extraction methods (below) result in "light" blocks with missing data, reducing compatibility with existing Substreams modules. ### Indexing frameworks Some blockchains, like NEAR, offer a native indexing framework: node extensions or libraries that allow you to tap into the native chain's codebase. The integration then transforms those indexing frameworks into the streaming + flat files structure of Firehose, and can then feed engines like [Substreams](https://substreams.streamingfast.io/). The downside of this method is that often state changes, full ordering and deeper relational data is lost in transit (or never captured by these frameworks). ### RPC Poller For chains where direct node instrumentation is not feasible, an RPC poller can fetch block data from the node's RPC interface. The poller runs alongside the node, fetching each new block via RPC, assembling it into a Firehose block, and outputting it using the [Firehose Protocol](/references/firehose-protocol). `firehose-core` provides a [blockpoller](https://github.com/streamingfast/firehose-core/tree/develop/blockpoller) library that handles the common polling logic. You implement the chain-specific RPC fetching, and the library manages cursor tracking, retries, and Firehose Logs output. **Examples of RPC poller implementations:** * [firehose-solana block fetcher](https://github.com/streamingfast/firehose-solana/tree/develop/block/fetcher) * [firehose-ethereum block fetcher](https://github.com/streamingfast/firehose-ethereum/tree/develop/blockfetcher) {% hint style="warning" %} This method limits the data available to what the node's RPC exposes. State changes and deep relational data are typically unavailable, resulting in "lighter" blocks compared to direct instrumentation. {% endhint %} ### Validator Plugins (Solana) Some blockchains offer plugin architectures that allow tapping into the validator's internal data flow. For Solana, the [Firehose Geyser Plugin](https://github.com/streamingfast/firehose-geyser-plugin) implements Solana's Geyser plugin interface to extract both block data and account updates directly from the validator. This approach provides richer data than RPC polling while avoiding the need to patch the node's source code. ## Protobuf Data Modeling Designing the protobuf structures for your given blockchain is one of the most important steps in an integrator's journey. Please review the [Naming Conventions](/references/naming-conventions) and the [different blockchain's data models](/references/protobuf-schemas), especially the Ethereum one which is thorough and well documented. {% hint style="warning" %} **Important***: It's imperative that the data structures in the protobuf's of the custom integration are represented as precisely as possible.* {% endhint %} ## Firehose Implementation Since the [Firehose Protocol](/references/firehose-protocol) is unified and chain-agnostic, `firehose-core` handles parsing and processing out of the box. For most integrations, you only need: 1. Instrument your node to output [Firehose Protocol](/references/firehose-protocol) messages 2. Define your protobuf block schema {% hint style="info" %} **Block Hash Format**: When implementing the protocol, pay careful attention to the [block hash representation requirements](/references/firehose-protocol#block-hash-representation). The format must remain consistent throughout the chain's lifetime, and `parent_hash` must exactly match the previous block's `block_hash`. For new chains, we recommend lowercase hexadecimal without the `0x` prefix. {% endhint %} Optionally, you can create a `firehose-` wrapper around `firehose-core` for chain-specific CLI commands or tooling. StreamingFast provides a starter template for this — see the [Firehose Acme](/integrate-new-chains/firehose-starter) page for details. # Design Principles StreamingFast Firehose design principles Firehose was heavily inspired by large-scale data science machinery and other processes previously developed by the StreamingFast team. ## The Firehose "North Star" Principles and assumptions: * Flat files provide more efficiency than live running CPU and RAM-consuming and intensive processes. * Fast iteration is preferred for data processes because data is inherently messy. * Data agility is only achievable when data processes can be parallelized. * Clear data contracts between tasks and processes including APIs, RPC query formats, and data model definitions, are critical * Maximum precision is required for defining, referencing, and identifying concepts or data models. *Leave no stone unturned.* * The only guarantee in data science is that data processes change and evolve indefinitely. * Migrating data is annoying, careful consideration must be taken for: * file formats, * forward and backward compatibility, * versioning, * and performance. ## Extraction During the extraction phase, our goals are: * All data captured should be deterministic, with the single exception of the block number at which the chain reaches finality (this number can vary depending on a node's relative topology to the network, and it could, for certain chains, not arrive at the same moment for all). * Performance-wise, we want the impact to be minimal on the node that is usually doing write operations. Deep data extraction is also one of the goals of our design, for the purposes of rich indexing downstream. For example: * Extracting both the previous value, and the new value on balance changes, and state changes to accounts, storage locations, key/value stores, etc. This also helps with integrity checking (to know if any changes were missing, all `prev -> new` pairs can be checked to match up, for a given storage key). * Extracting all the relationships, between blocks and transactions, between transactions and single function/calls executions within a transaction, call trees, and a thing we call **total ordering**, meaning having an **ordinal** that can help order all things tracked (beginning/end of transactions, function calls, state changes, events emitted during execution, etc..) all relative to one another. For example, Ethereum has log indexes, allowing ordering of a log vs another log. But it doesn't allow for ordering a log versus a state change, or a log within a tree of calls (where perhaps the input of the call is what you're watching). * Some blockchains allow you to query state, and query events separately. Oftentime, it's not possible to link those things. We like to instrumented to be link changes to the source of events, to be able to build better downstream systems, and not lose relations between state and events. * Most indexing strategies hinge on events, but having state changes allows for new opportunities of indexing, triggering on the actual values being mutated. On certain chains, this allows you to avoid some gas costs by limiting the events, as you're able to trigger "virtual" events based on state changes. * Picking up on those changes can also avoid needing to (re-)design contracts when new data is needed, and wasn't thought of at first. Also, when building an extractor is to **extract all the data necessary to recreate an archive node**. If nothing is missing, then someone indexing downstream should be satisfied. Another principle is: like in any database, **transactions/calls are the true boundaries of state changes**, blocks exist only to facilitate and accelerate consensus (there would be great overhead if networks needed to agree on each individual transaction) but are as such an artificial boundary. RPC nodes usually round up things to the block level, but with Firehose, data should be extracted in a way that makes the transaction, or even the individual smart contract calls, the unit of change. Concretely, this means state changes should be modeled at the lowest level. ## Pure Data, Files & Streams ### Flat Files StreamingFast chose to utilize flat files instead of the traditional request and response model of data acquisition. Using flat files alleviates the challenges presented by querying pools of, generally load-balanced, nodes in some type of replication configuration. ### Simplification with Flat Files The decision to rely on flat files assists with the reduction of massive consistency issues, retries, and incurred latency. In addition, using flat files greatly simplifies the task developers face writing code to interface with blockchain node data. ## Adhering to the Unix Philosophy Flat-file and data stream abstractions adhere to the Unix philosophy of *writing programs that do one thing, do it well, and work together with other programs by handling streams of data as input and output.* ## **State Transition Hierarchy** StreamingFast uses state transitions scoped to calls, indexed within transactions, indexed within a block. {% hint style="success" %} **Tip***: Blockchains typically “round up” state changes for all transactions into a block to facilitate consensus.* {% endhint %} ### Smart Contract Execution The basic unit of execution always remains a single smart contract execution resulting in a single EVM call. However, calling another contract from within the first contract means a second execution will occur. ### Keeping Track of State Contracts lose state precision when the state is changed in the middle of a transaction or block. Attempting to locate the balance for calculations at the exact point needed, during the processing of a log event, for example, will result in receiving the balance value at the end of the block. The balance value may have changed state in a subsequent transaction after the transaction currently being indexed. {% hint style="warning" %} **Important***: The process of querying nodes can cause substantial issues for developers wanting finite node data access.* {% endhint %} ### Blockchain Data Consumption Consuming blockchain state is difficult and each blockchain presents its own issues. [Solidity](https://docs.soliditylang.org/en/v0.8.16/), for example, uses `bytes32` => `bytes32` mapping, making such a data retrieval endeavour rather opaque and difficult to reason about. This data is available with additional effort, but not easily. {% hint style="success" %} **Tip***: Developers having access to state data presents tremendous opportunities for indexing and application development.* {% endhint %} ### Protocol Buffers Google's [Protocol Buffers version 3](https://developers.google.com/protocol-buffers/docs/proto3) met the requirements identified by StreamingFast for versioning, compatibility, and speed of file content. Optional and required fields were removed in Google's Protocol Buffers version 3 simplifying the data extraction process. # Firehose Acme StreamingFast Firehose template for new chain integrations Instrumenting a new chain requires the blockchain node to output [Firehose Protocol](/references/firehose-protocol) messages — a simple, unified protocol consisting of `FIRE INIT` and `FIRE BLOCK` messages. Any chain that implements this protocol automatically benefits from the entire Firehose ecosystem. The [Firehose Protocol](/references/firehose-protocol) is chain-agnostic: the node outputs block data as base64-encoded protobuf, and `firehose-core` handles the rest. The chain-specific parts are: 1. **Node instrumentation**: Modify your blockchain node to output `FIRE INIT` and `FIRE BLOCK` messages 2. **Protobuf schema**: Define your chain's block data model That's it — most chains only need these two pieces. No custom binary required. ## Implement Node Instrumentation The key integration work is implementing the [Firehose Protocol](/references/firehose-protocol) in your blockchain node. The node must output: 1. `FIRE INIT` once at startup with the protocol version and protobuf block type 2. `FIRE BLOCK` for each block with metadata and base64-encoded protobuf payload {% hint style="warning" %} **Block Hash Format**: Pay careful attention to the [block hash representation requirements](/references/firehose-protocol#block-hash-representation). The `parent_hash` of each block must exactly match the `block_hash` of the previous block. For new chains, use lowercase hexadecimal without the `0x` prefix. {% endhint %} See the [dummy-blockchain tracer](https://github.com/streamingfast/dummy-blockchain/tree/main/tracer) for a reference implementation. ## Define Protobuf Types Create protobuf definitions that model your chain's block data structure. The block type name must match what your instrumented node outputs in the `FIRE INIT` message. Publish your protobuf definitions to the [Buf registry](https://buf.build) to use `firecore` directly, or keep them local if building a custom binary. ## Using `firecore` Directly If your protobuf block definitions are published to the [Buf registry](https://buf.build), you can use `firecore` directly without building any chain-specific binary: ```bash firecore start reader-node \ --reader-node-path=/path/to/your/instrumented-node \ --reader-node-proto-type=buf.build/myorg/mychain:sf.mychain.type.v1.Block ``` This is the simplest integration path and works for most chains. ## Optional: Custom `firehose-` Binary For chains that need custom CLI commands, specialized tooling, or prefer not to use the Buf registry, you can create a `firehose-` wrapper around `firehose-core`. We provide a template project [firehose-acme](https://github.com/streamingfast/firehose-acme) as a starting point. It includes scaffolding code and a [dummy-blockchain](https://github.com/streamingfast/dummy-blockchain) example. Copy the template: ```bash git clone git@github.com:streamingfast/firehose-acme firehose-mychain cd firehose-mychain ``` ### Rename to Your Chain Rename all references from "acme" to your chain's name. Choose two names: * **Long form**: Full chain name (e.g., `arweave`, `solana`, `ethereum`) * **Short form**: 3-4 letter abbreviation (e.g., `arw`, `sol`, `eth`) Perform the following replacements: * Rename `cmd/fireacme` → `cmd/fire` (e.g., `cmd/firearw`) * Search and replace `fireacme` → `fire` * Search and replace `acme` → `` (case-sensitive for `acme`, `ACME`, `Acme`) Update the proto file path `sf/acme/type/v1/type.proto` and regenerate Go structs: ```bash ./types/pb/generate.sh ``` ### Development & Testing The template includes a development script that starts Firehose with the dummy-blockchain: ```bash ./devel/standard/start.sh ``` This uses the configuration in [devel/standard/standard.yaml](https://github.com/streamingfast/firehose-acme/blob/master/devel/standard/standard.yaml). Modify `start.flags.reader-node-path` to point to your instrumented node binary. Run the test suite: ```bash go test ./... ``` {% hint style="info" %} You can reach out to the StreamingFast team on Discord. We usually maintain these Go-side integrations and can help with the initial setup. {% endhint %} ## Register Your Chain Once your integration is working, register your chain in The Graph's networks registry to make it discoverable by the ecosystem. ### The Graph Networks Registry Add your chain to [The Graph Networks Registry](https://github.com/graphprotocol/networks-registry) by following their [adding/updating a chain guide](https://github.com/graphprotocol/networks-registry?tab=readme-ov-file#addingupdating-a-chain). The registry entry includes metadata about your chain such as: * Chain ID and name * Firehose and Substreams endpoints * Block type information ### Update Well-Known Protobuf Descriptors After your chain is registered, update `firehose-core` to include your chain's protobuf descriptors in the well-known types. This enables better tooling support across the ecosystem. 1. Run the [registry generator](https://github.com/streamingfast/firehose-core/blob/develop/proto/registry.go#L21) to regenerate the well-known descriptors 2. Open a PR to `firehose-core` with the updated descriptors {% hint style="info" %} The StreamingFast team can help with this step — reach out on Discord after your chain is registered. {% endhint %} # Firehose Protocol Firehose Protocol 3.x Specification Reference The Firehose Protocol defines a line-based text protocol for streaming blockchain block data from an instrumented node to a Firehose reader process. This document specifies versions 3.0 and 3.1 of the protocol. ## Overview The protocol operates over stdout/stderr, where the instrumented blockchain node emits specially formatted lines prefixed with `FIRE` . The Firehose reader process parses these lines to construct block objects that are then stored and served. All protocol messages follow this general format: ``` FIRE ``` ## Protocol Versions | Version | Description | | ------- | ------------------------------------------------------- | | 3.0 | Standard protocol for tracer-based block extraction | | 3.1 | Extends 3.0 with partial block support for large blocks | ## Message Types The Firehose Protocol 3.x defines two message types: | Message | Purpose | | ------- | --------------------------------------------------------------------- | | `INIT` | Initialization handshake establishing protocol version and block type | | `BLOCK` | Complete or partial block data transmission | *** ## FIRE INIT The `INIT` message must be the first Firehose message emitted by the node. It establishes the protocol version and declares the Protobuf message type used for block payloads. ### Format ``` FIRE INIT ``` ### Fields | Field | Type | Description | | --------------- | ------ | ----------------------------------------------------------- | | `version` | string | Protocol version: `3.0` or `3.1` | | `protobuf_type` | string | Fully qualified Protobuf message type for the block payload | ### Examples ``` FIRE INIT 3.0 sf.ethereum.type.v2.Block FIRE INIT 3.1 sf.ethereum.type.v2.Block FIRE INIT 3.0 sf.solana.type.v1.Block ``` ### Behavior * The reader validates the protocol version is supported * The `protobuf_type` is used to construct the `type.googleapis.com/` URL in the Any wrapper * For version 3.1, partial block parsing is enabled *** ## FIRE BLOCK The `BLOCK` message transmits block data. The format differs slightly between versions 3.0 and 3.1. ### Version 3.0 Format ``` FIRE BLOCK ``` **Field Count:** 7 fields after `FIRE BLOCK` ### Version 3.1 Format ``` FIRE BLOCK ``` **Field Count:** 8 fields after `FIRE BLOCK` ### Fields | Field | Type | Version | Description | | --------------------- | ------ | -------- | ----------------------------------------------------------- | | `block_num` | uint64 | 3.0, 3.1 | Block number/height | | `partial_idx` | int64 | 3.1 only | Partial block index (see [Partial Blocks](#partial-blocks)) | | `block_hash` | string | 3.0, 3.1 | Block identifier/hash | | `parent_num` | uint64 | 3.0, 3.1 | Parent block number | | `parent_hash` | string | 3.0, 3.1 | Parent block identifier/hash | | `lib_num` | uint64 | 3.0, 3.1 | Last Irreversible Block number | | `timestamp_unix_nano` | uint64 | 3.0, 3.1 | Block timestamp in Unix nanoseconds | | `payload_base64` | string | 3.0, 3.1 | Base64-encoded Protobuf block payload | ### Block Hash Representation The protocol does not mandate a specific string format for block hashes. However, **consistency is critical**: * The chosen format (e.g., hexadecimal, base58, base64) must remain constant throughout the chain's lifetime * Both `block_hash` and `parent_hash` must use the same encoding format * The `parent_hash` of block N must exactly match the `block_hash` of block N-1 {% hint style="warning" %} **Chain Continuity Requirement** The Firehose system relies on parent hash linking to establish block continuity and detect forks. If `parent_hash` does not exactly match the previous block's `block_hash` (as a string), the reader will fail to establish the chain relationship. This means: * If using hex encoding, use it consistently (always lowercase or always uppercase) * If using a prefix like `0x`, always include it * Never change the encoding format after genesis {% endhint %} {% hint style="info" %} **Recommendation for New Chains** For new chain integrations, we strongly recommend using **lowercase hexadecimal without the `0x` prefix**: ``` abc123def456... (recommended) 0xabc123def456... (not recommended) ABC123DEF456... (not recommended) ``` This format is being considered as the canonical standard for future protocol versions. Adopting it now ensures forward compatibility and consistency across the Firehose ecosystem. {% endhint %} ### Examples **Version 3.0:** ``` FIRE BLOCK 12345678 abc123def456789012345678901234567890123456789012345678901234 12345677 789abcdef012345678901234567890123456789012345678901234567890 12345600 1699900000000000000 CgR0ZXN0... ``` **Version 3.1 (complete block):** ``` FIRE BLOCK 12345678 1000 abc123def456789012345678901234567890123456789012345678901234 12345677 789abcdef012345678901234567890123456789012345678901234567890 12345600 1699900000000000000 CgR0ZXN0... ``` **Version 3.1 (partial block, not final):** ``` FIRE BLOCK 12345678 0 abc123def456789012345678901234567890123456789012345678901234 12345677 789abcdef012345678901234567890123456789012345678901234567890 12345600 1699900000000000000 CgR0ZXN0... FIRE BLOCK 12345678 1 abc123def456789012345678901234567890123456789012345678901234 12345677 789abcdef012345678901234567890123456789012345678901234567890 12345600 1699900000000000000 CgR0ZXN0... ``` *** ## Partial Blocks Protocol version 3.1 introduces support for partial blocks, enabling transmission of large blocks in multiple chunks. ### Partial Index Encoding The `partial_idx` field uses a special encoding: | Value | Meaning | | ---------------- | ----------------------------------------------------- | | `0` to `999` | Partial block chunk at index N, more chunks to follow | | `1000` to `1999` | Final partial block chunk at index (N - 1000) | ### Examples | `partial_idx` | Actual Index | Is Final | | ------------- | ------------ | ------------------------------ | | `0` | 0 | No | | `1` | 1 | No | | `5` | 5 | No | | `1000` | 0 | Yes (single complete block) | | `1001` | 1 | Yes (2 partials, this is last) | | `1005` | 5 | Yes (6 partials, this is last) | ### Behavior 1. When `partial_idx < 1000`: Reader accumulates the partial payload, expecting more 2. When `partial_idx >= 1000`: Reader treats this as the final chunk and assembles the complete block 3. A value of `1000` indicates a single complete block (equivalent to version 3.0 behavior) *** ## Payload Format The `payload_base64` field contains a Base64-encoded Protobuf message. The message type is declared in the `FIRE INIT` message. ### Wrapping The decoded payload is wrapped in a `google.protobuf.Any` message: ```protobuf message Any { string type_url = 1; // "type.googleapis.com/" bytes value = 2; // The decoded payload bytes } ``` ### Block Container The Any-wrapped payload is then placed in a `sf.bstream.v1.Block` container: ```protobuf message Block { string id = 1; // block_hash uint64 number = 2; // block_num string parent_id = 3; // parent_hash uint64 parent_num = 4; // parent_num uint64 lib_num = 5; // lib_num Timestamp timestamp = 6; // Converted from timestamp_unix_nano Any payload = 7; // The wrapped block payload } ``` *** ## Error Handling ### Invalid Protocol Version If the `FIRE INIT` message specifies an unsupported version, the reader should terminate with an error indicating the supported versions. ### Malformed Messages Lines that: * Don't start with `FIRE` * Have incorrect field counts * Contain unparseable numeric values * Have invalid Base64 encoding Should be logged and typically cause the reader to terminate, as they indicate a protocol mismatch or corrupted output. ### Missing INIT If a `FIRE BLOCK` message is received before `FIRE INIT`, the reader should terminate with an error. *** ## Implementation Notes ### Line Parsing 1. Check line starts with `FIRE` prefix 2. Extract message type (`INIT` or `BLOCK`) 3. Split remaining content by spaces with bounded chunks (last field accumulates all remaining content) 4. Parse fields according to their expected types ### Numeric Parsing * All numeric fields use base-10 representation * `block_num`, `parent_num`, `lib_num`, `timestamp_unix_nano`: Parse as unsigned 64-bit integers * `partial_idx`: Parse as signed 64-bit integer ### Base64 Decoding * Use standard Base64 encoding (RFC 4648) * The decoded bytes are the raw Protobuf message (not Any-wrapped at this stage) # CLI Reference This document provides a comprehensive reference for the Firehose CLI. ## Binaries | Binary | Description | Use Case | | ---------- | ------------------------ | ------------------------------------- | | `firecore` | Core Firehose binary | All chains except Ethereum-compatible | | `fireeth` | Ethereum Firehose binary | Ethereum and EVM-compatible chains | {% hint style="info" %} `fireeth` includes all `firecore` functionality plus Ethereum-specific commands. All examples below use `firecore` but work identically with `fireeth`. {% endhint %} ## Start Command The `firecore start` command launches Firehose components. ```bash firecore start [flags] [all|component1 [component2...]] ``` ### Available Components | Component | Description | Default Port | | ---------------------- | --------------------------------------------------- | --------------------------------------- | | `reader-node` | Spawns and manages blockchain node, extracts blocks | `:10010` (gRPC), `:10011` (manager API) | | `reader-node-stdin` | Reads blocks from stdin pipe | `:10010` | | `reader-node-firehose` | Reads blocks from remote Firehose endpoint | `:10010` | | `merger` | Combines one-block files into merged bundles | `:10012` | | `relayer` | Streams live blocks, provides high availability | `:10014` | | `firehose` | Serves Firehose gRPC API | `:10015` | | `substreams-tier1` | Serves Substreams API, handles live blocks | `:10016` | | `substreams-tier2` | Substreams worker for historical processing | `:10017` | For component-specific configuration, see: * [Reader Node CLI Reference](/references/cli-reference/reader-node) * [Merger CLI Reference](/references/cli-reference/merger) * [Relayer CLI Reference](/references/cli-reference/relayer) * [Firehose CLI Reference](/references/cli-reference/firehose) * [Substreams CLI Reference](/references/cli-reference/substreams) ## Global Flags These flags are available for all `firecore` commands. | Flag | Description | Default | | ---------------------------------- | -------------------------------------------------------- | ----------------- | | `-c, --config-file` | Configuration file path. Set to empty string to disable. | `./firehose.yaml` | | `-d, --data-dir` | Data storage directory for all components | `./firehose-data` | | `--log-format` | Log output format: `text` or `stackdriver` | `text` | | `--log-to-file` | Also write logs to `{data-dir}/app.log.json` | `true` | | `-v, --log-verbosity` | Verbose output level (use `-vvvv` for max) | `0` | | `--metrics-listen-addr` | Prometheus metrics endpoint | `:9102` | | `--pprof-listen-addr` | pprof profiling endpoint | `localhost:6060` | | `--startup-delay` | Delay before launching components | `0` | | `--log-level-switcher-listen-addr` | HTTP endpoint to dynamically change log levels | `localhost:1065` | ### Dynamic Log Level Switching Change log levels at runtime: ```bash curl -XPUT -d '{"level":"debug","inputs":"*"}' http://localhost:1065 ``` Valid levels: `trace`, `debug`, `info`, `warn`, `error`, `panic` ## Common Flags These flags are shared across multiple components and prefixed with `--common-`. | Flag | Description | Default | | --------------------------------------- | -------------------------------------------- | ----------------------------------------- | | `--common-one-block-store-url` | Store URL for one-block files | `file://{data-dir}/storage/one-blocks` | | `--common-merged-blocks-store-url` | Store URL for merged block files | `file://{data-dir}/storage/merged-blocks` | | `--common-forked-blocks-store-url` | Store URL for forked block files | `file://{data-dir}/storage/forked-blocks` | | `--common-index-store-url` | Store URL for index files | `file://{data-dir}/storage/index` | | `--common-live-blocks-addr` | gRPC endpoint for real-time blocks (Relayer) | `:10014` | | `--common-first-streamable-block` | First streamable block number | `0` | | `--common-auth-plugin` | Authentication plugin URI | `null://` | | `--common-metering-plugin` | Metering plugin URI | `null://` | | `--common-session-plugin` | Session plugin URI | `local://...` | | `--common-tmp-dir` | Temporary files directory | `{data-dir}/tmp` | | `--common-auto-max-procs` | Auto-set GOMAXPROCS from cgroup | `false` | | `--common-auto-mem-limit-percent` | Auto-set GOMEMLIMIT percentage from cgroup | `0` | | `--common-system-shutdown-signal-delay` | Delay between SIGTERM and shutdown | `0` | ### Index Flags (Legacy) {% hint style="warning" %} Index flags are for legacy graph-node integration. For new projects, use Substreams instead. {% endhint %} | Flag | Description | Default | | ---------------------------- | ------------------------ | ------------------------- | | `--common-index-block-sizes` | Valid index bundle sizes | `[100000,10000,1000,100]` | ## Advertise Flags These flags configure the Info endpoint for Firehose and Substreams Tier 1. | Flag | Description | Default | | ------------------------------- | ------------------------------------------------------------------- | ----------------------- | | `--advertise-chain-name` | Chain name to advertise | (inferred from genesis) | | `--advertise-chain-aliases` | Chain name aliases | (inferred from genesis) | | `--advertise-block-id-encoding` | Block ID encoding: `hex`, `0x_hex`, `base58`, `base64`, `base64url` | (inferred from genesis) | | `--advertise-block-features` | Block features to advertise | (inferred from genesis) | | `--ignore-advertise-validation` | Skip genesis block validation | `false` | ## Configuration File As an alternative to command-line flags, use a YAML configuration file: ```yaml start: args: - reader-node - merger - relayer - firehose flags: data-dir: /data/firehose reader-node-path: /usr/local/bin/geth reader-node-arguments: "--datadir {node-data-dir} --vmtrace=firehose" common-merged-blocks-store-url: "s3://my-bucket/merged-blocks" ``` ## Environment Variables Flags can be set via environment variables. Convert flag names by replacing `-` with `_` and uppercasing. ### Global Flags Global flags (those available on all commands like `--data-dir`, `--log-format`) use the `FIRECORE_GLOBAL_` prefix: | Flag | Environment Variable | | ----------------------- | ------------------------------------- | | `--data-dir` | `FIRECORE_GLOBAL_DATA_DIR` | | `--log-format` | `FIRECORE_GLOBAL_LOG_FORMAT` | | `--metrics-listen-addr` | `FIRECORE_GLOBAL_METRICS_LISTEN_ADDR` | ### Start Command Flags Flags specific to the `start` command use the `FIRECORE_` prefix (without `GLOBAL_`): | Flag | Environment Variable | | ---------------------------------- | ----------------------------------------- | | `--common-first-streamable-block` | `FIRECORE_COMMON_FIRST_STREAMABLE_BLOCK` | | `--common-merged-blocks-store-url` | `FIRECORE_COMMON_MERGED_BLOCKS_STORE_URL` | | `--reader-node-path` | `FIRECORE_READER_NODE_PATH` | | `--reader-node-arguments` | `FIRECORE_READER_NODE_ARGUMENTS` | | `--firehose-grpc-listen-addr` | `FIRECORE_FIREHOSE_GRPC_LISTEN_ADDR` | ## Storage URLs Firehose supports multiple storage backends via URL schemes: | Scheme | Example | Description | | --------- | --------------------- | -------------------- | | `file://` | `file:///data/blocks` | Local filesystem | | `gs://` | `gs://bucket/path` | Google Cloud Storage | | `s3://` | `s3://bucket/path` | Amazon S3 | | `az://` | `az://container/path` | Azure Blob Storage | {% hint style="info" %} For S3-compatible storage (MinIO, Ceph), use `s3://` with appropriate endpoint configuration via environment variables. {% endhint %} ## Getting Help ```bash # General help firecore --help # Start command help firecore start --help # List available tools firecore tools --help ``` # Reader Node This page documents the configuration flags for the Reader Node component. For architectural concepts and how the Reader Node works, see [Reader Node Architecture](/firehose/architecture/components/reader). ## Reader Node Variants Firehose provides three reader modes: | Component | Description | | ---------------------- | ----------------------------------------------- | | `reader-node` | Spawns and manages a blockchain node subprocess | | `reader-node-stdin` | Reads blocks from stdin pipe | | `reader-node-firehose` | Reads blocks from a remote Firehose endpoint | ## Core Configuration | Flag | Description | Default | | -------------------------------- | -------------------------------------------------- | ------------------------ | | `--reader-node-path` | Path to blockchain node binary | (required) | | `--reader-node-arguments` | Arguments passed to the node (supports templating) | | | `--reader-node-data-dir` | Data directory for the blockchain node | `{data-dir}/reader/data` | | `--reader-node-working-dir` | Working directory for reader files | `{data-dir}/reader/work` | | `--reader-node-grpc-listen-addr` | gRPC address for streaming blocks | `:10010` | | `--reader-node-manager-api-addr` | HTTP address for node manager API | `:10011` | ## Argument Templating The `--reader-node-arguments` flag is parsed using standard shell quoting rules, allowing you to: * Use double quotes for values with spaces: `--flag "value with spaces"` * Use single quotes for literal strings * Escape special characters as needed The flag also supports these template variables: | Template | Description | | -------------------------- | ------------------------------------------ | | `{data-dir}` | Value of `--data-dir` flag | | `{node-data-dir}` | Value of `--reader-node-data-dir` flag | | `{hostname}` | Machine hostname | | `{start-block-num}` | Value of `--reader-node-start-block-num` | | `{stop-block-num}` | Value of `--reader-node-stop-block-num` | | `{first-streamable-block}` | Value of `--common-first-streamable-block` | Environment variables are also expanded: `${VAR}` becomes the value of `VAR`. **Example:** ```bash --reader-node-arguments="--datadir={node-data-dir} --port=${P2P_PORT} --vmtrace=firehose" ``` ## Block Range Control | Flag | Description | Default | | -------------------------------------- | -------------------------------- | ------- | | `--reader-node-start-block-num` | Skip blocks before this number | `0` | | `--reader-node-stop-block-num` | Stop after reaching this block | (none) | | `--reader-node-discard-after-stop-num` | Discard blocks after stop number | `false` | ## Performance Tuning | Flag | Description | Default | | ------------------------------------- | -------------------------------------------------------- | ------------------- | | `--reader-node-blocks-chan-capacity` | Block channel capacity (shutdown at 90%) | `100` | | `--reader-node-line-buffer-size` | Max line buffer size in bytes | `209715200` (200MB) | | `--reader-node-readiness-max-latency` | Max head block latency for health check | `30s` | | `--reader-node-one-block-suffix` | Unique suffix for one-block files (for multiple readers) | `default` | ## Bootstrap Configuration | Flag | Description | Default | | ---------------------------------- | ------------------------------------------------- | ------- | | `--reader-node-bootstrap-data-url` | URL to bootstrap empty node from backup or script | | Bootstrap URL formats: * `gs://bucket/backup.tar.zst` - Extract archive to data dir * `s3://bucket/backup.tar.zst` - Extract archive to data dir * `bash:///path/to/script.sh?arg=value` - Execute bash script ## Reader Node Firehose Mode For `reader-node-firehose` component connecting to remote Firehose: | Flag | Description | Default | | ------------------------------------------ | ----------------------------------- | ------------------------- | | `--reader-node-firehose-endpoint` | Remote Firehose endpoint | (required) | | `--reader-node-firehose-plaintext` | Use plaintext connection | `false` | | `--reader-node-firehose-insecure` | Skip TLS validation | `false` | | `--reader-node-firehose-compression` | Compression: `gzip`, `zstd`, `none` | `zstd` | | `--reader-node-firehose-state` | State file for cursor persistence | `{data-dir}/reader/state` | | `--reader-node-firehose-api-key-env-var` | Env var containing API key | `FIREHOSE_API_KEY` | | `--reader-node-firehose-api-token-env-var` | Env var containing JWT token | `FIREHOSE_API_TOKEN` | ## Backup Configuration | Flag | Description | | ----------------------- | ---------------------------------- | | `--reader-node-backups` | Backup module configuration string | Backup parameters format: `type= prefix= tag= freq-blocks=` * `type`: Backup module type (e.g., `gke-pvc-snapshot`) * `prefix`: Backup name prefix * `tag`: Backup tag/version * `freq-blocks`: Trigger backup every N blocks * `freq-time`: Trigger backup by time interval ## Debug Flags | Flag | Description | Default | | -------------------------------------- | --------------------------------------- | ------------ | | `--reader-node-debug-firehose-logs` | Print Firehose protocol logs to stdout | `false` | | `--reader-node-test-mode` | Compare blocks against another instance | `false` | | `--reader-node-test-mode-diff-against` | Address of instance to diff against | | | `--reader-node-test-mode-diff-output` | File path for diff output | `-` (stdout) | ## Management API The Reader Node exposes an HTTP management API on `--reader-node-manager-api-addr` (default `:10011`). ### Available Endpoints | Endpoint | Method | Description | | ----------------------------- | ------ | ------------------------------------------------------------ | | `/healthz` or `/v1/healthz` | GET | Health check - returns ready/not ready status | | `/v1/ping` | GET | Simple ping, returns "pong" | | `/v1/is_running` | GET | Check if subprocess is running (JSON response) | | `/v1/start_command` | GET | Show the command being executed | | `/v1/maintenance` | POST | Stop subprocess for maintenance | | `/v1/resume` | POST | Start/resume the subprocess | | `/v1/reload` or `/v1/restart` | POST | Restart the subprocess | | `/v1/backup` | POST | Trigger a backup (if backup modules configured) | | `/v1/restore` | POST | Restore from backup | | `/v1/list_backups` | GET | List available backups | | `/v1/safely_reload` | POST | Safely reload (waits for production round on producer nodes) | ### Usage Examples ```bash # Check health status curl http://localhost:10011/v1/healthz # Put node in maintenance mode (stop subprocess) curl -XPOST http://localhost:10011/v1/maintenance # Resume node operation curl -XPOST http://localhost:10011/v1/resume # Restart the underlying node curl -XPOST http://localhost:10011/v1/restart # Check if subprocess is running curl http://localhost:10011/v1/is_running # Returns: {"is_running":true} ``` {% hint style="info" %} Add `?sync=true` to POST requests to wait for the command to complete before returning. {% endhint %} # Merger This page documents the configuration flags for the Merger component. For architectural concepts and how the Merger works, see [Merger Architecture](/firehose/architecture/components/merger). ## Starting the Merger ```bash firecore start merger [flags] ``` ## Core Configuration | Flag | Description | Default | | --------------------------- | ---------------------------------------- | -------- | | `--merger-grpc-listen-addr` | gRPC listening address for health checks | `:10012` | ## Storage Configuration The Merger uses the common storage flags: | Flag | Description | Default | | ---------------------------------- | ----------------------------------- | ----------------------------------------- | | `--common-one-block-store-url` | One-block files storage URL (input) | `file://{data-dir}/storage/one-blocks` | | `--common-merged-blocks-store-url` | Merged blocks storage URL (output) | `file://{data-dir}/storage/merged-blocks` | | `--common-forked-blocks-store-url` | Forked blocks storage URL | `file://{data-dir}/storage/forked-blocks` | ## Timing Configuration | Flag | Description | Default | | ------------------------------------- | ----------------------------------- | ------- | | `--merger-time-between-store-lookups` | Polling interval for source store | `1s` | | `--merger-time-between-store-pruning` | Interval between pruning operations | `1m0s` | ## Pruning Configuration | Flag | Description | Default | | ------------------------------------ | ------------------------------------- | ------- | | `--merger-prune-forked-blocks-after` | Blocks to retain before pruning forks | `50000` | | `--merger-delete-threads` | Parallel threads for file deletion | `8` | {% hint style="warning" %} The pruning distance should be set large enough to handle chain reorganizations. The default of 50,000 blocks is conservative and suitable for most chains. {% endhint %} ## Execution Control | Flag | Description | Default | | --------------------- | -------------------------------- | ------- | | `--merger-stop-block` | Stop after merging to this block | (none) | ## Health Check The Merger exposes a gRPC health check endpoint: ```bash grpcurl -plaintext localhost:10012 grpc.health.v1.Health/Check ``` The Merger always reports `SERVING` status when running - it doesn't have dynamic readiness conditions like other components. ## Example Usage ### Basic Local Setup ```bash firecore start merger \ --merger-grpc-listen-addr=":10012" ``` ### With Cloud Storage ```bash firecore start merger \ --merger-grpc-listen-addr=":10012" \ --common-one-block-store-url="s3://my-bucket/one-blocks" \ --common-merged-blocks-store-url="s3://my-bucket/merged-blocks" \ --common-forked-blocks-store-url="s3://my-bucket/forked-blocks" ``` ### Processing to a Specific Block ```bash firecore start merger \ --merger-grpc-listen-addr=":10012" \ --merger-stop-block=1000000 ``` # Relayer This page documents the configuration flags for the Relayer component. For architectural concepts and how the Relayer works, see [Relayer Architecture](/firehose/architecture/components/relayer). ## Starting the Relayer ```bash firecore start relayer [flags] ``` ## Core Configuration | Flag | Description | Default | | ------------------------------ | ------------------------------------ | -------------------------------------- | | `--relayer-grpc-listen-addr` | gRPC listening address | `:10014` | | `--relayer-source` | Reader source addresses (repeatable) | `[:10010]` | | `--common-one-block-store-url` | One-block files storage URL | `file://{data-dir}/storage/one-blocks` | ## Performance Tuning | Flag | Description | Default | | -------------------------------- | -------------------------------------- | --------- | | `--relayer-max-source-latency` | Max tolerated source latency | `999999h` | | `--relayer-source-request-burst` | Request burst size to upstream sources | (varies) | {% hint style="info" %} The default max source latency is very high (`999999h`) - effectively disabled. Set a reasonable value like `30s` for production deployments where you want to filter unresponsive sources. {% endhint %} ## Connecting Multiple Sources The `--relayer-source` flag can be specified multiple times: ```bash firecore start relayer \ --relayer-grpc-listen-addr=":10014" \ --relayer-source="reader-1.internal:10010" \ --relayer-source="reader-2.internal:10010" \ --relayer-source="reader-3.internal:10010" ``` ## Health Check The Relayer exposes a gRPC health check endpoint with two states: | Status | Meaning | | ------------- | -------------------------------------------------------- | | `SERVING` | Relayer is synchronized and streaming blocks | | `NOT_SERVING` | Relayer is starting up or has lost connection to sources | ### Health Check Commands ```bash # Check current health status grpcurl -plaintext localhost:10014 grpc.health.v1.Health/Check # Watch for status changes (polls every 5 seconds) grpcurl -plaintext localhost:10014 grpc.health.v1.Health/Watch ``` ### Kubernetes Integration ```yaml readinessProbe: exec: command: - grpcurl - -plaintext - localhost:10014 - grpc.health.v1.Health/Check initialDelaySeconds: 10 periodSeconds: 5 ``` ## Example Usage ### Single Reader Setup ```bash firecore start relayer \ --relayer-grpc-listen-addr=":10014" \ --relayer-source="localhost:10010" ``` ### Multiple Readers with Latency Filtering ```bash firecore start relayer \ --relayer-grpc-listen-addr=":10014" \ --relayer-source="reader-1.internal:10010" \ --relayer-source="reader-2.internal:10010" \ --relayer-max-source-latency="30s" ``` # Firehose This page documents the configuration flags for the Firehose component. For architectural concepts and how the Firehose component works, see [Firehose Architecture](/firehose/architecture/components/firehose). ## Starting the Firehose ```bash firecore start firehose [flags] ``` ## Core Configuration | Flag | Description | Default | | --------------------------------- | -------------------------------------- | -------- | | `--firehose-grpc-listen-addr` | gRPC listening address | `:10015` | | `--common-live-blocks-addr` | Relayer gRPC address for live blocks | `:10014` | | `--common-first-streamable-block` | First block number available to stream | `0` | ## Storage Configuration | Flag | Description | Default | | ---------------------------------- | --------------------------- | ----------------------------------------- | | `--common-merged-blocks-store-url` | Merged blocks storage URL | `file://{data-dir}/storage/merged-blocks` | | `--common-one-block-store-url` | One-block files storage URL | `file://{data-dir}/storage/one-blocks` | | `--common-forked-blocks-store-url` | Forked blocks storage URL | `file://{data-dir}/storage/forked-blocks` | ## Performance Tuning | Flag | Description | Default | | ---------------------------------------- | --------------------------------------- | ------- | | `--firehose-enforce-compression` | Require gzip or zstd compression | `true` | | `--firehose-rate-limit-bucket-size` | Rate limit bucket size (-1 = unlimited) | `-1` | | `--firehose-rate-limit-bucket-fill-rate` | Rate limit refill rate | `10s` | ## Discovery Service For load-balanced deployments: | Flag | Description | | ---------------------------------- | -------------------------- | | `--firehose-discovery-service-url` | gRPC discovery service URL | ## Example Usage ### Basic Local Setup ```bash firecore start firehose \ --firehose-grpc-listen-addr=":10015" \ --common-live-blocks-addr="localhost:10014" ``` ### With Cloud Storage ```bash firecore start firehose \ --firehose-grpc-listen-addr=":10015" \ --common-merged-blocks-store-url="s3://my-bucket/merged-blocks" \ --common-one-block-store-url="s3://my-bucket/one-blocks" \ --common-forked-blocks-store-url="s3://my-bucket/forked-blocks" \ --common-live-blocks-addr="relayer.internal:10014" ``` ### With Rate Limiting ```bash firecore start firehose \ --firehose-grpc-listen-addr=":10015" \ --firehose-rate-limit-bucket-size=100 \ --firehose-rate-limit-bucket-fill-rate="1s" ``` ## gRPC API Firehose exposes the `sf.firehose.v2.Stream` service: ```protobuf service Stream { rpc Blocks(Request) returns (stream Response); } message Request { int64 start_block_num = 1; string stop_block_num = 2; // Empty for live streaming string cursor = 3; // Resume from cursor repeated string final_blocks_only = 4; repeated google.protobuf.Any transforms = 5; } ``` ### Streaming Modes 1. **Historical range**: Specify start and stop block numbers 2. **Historical to live**: Specify start, omit stop to continue streaming indefinitely 3. **Live only**: Start from a recent block or "head" 4. **Cursor resume**: Provide cursor from previous session # Substreams This page documents the configuration flags for the Substreams Tier 1 and Tier 2 components. For architectural concepts and how Substreams works, see [Substreams Architecture](/firehose/architecture/components/substreams). ## Substreams Tier 1 The Tier 1 component serves the Substreams API and handles live block processing. ### Starting Tier 1 ```bash firecore start substreams-tier1 [flags] ``` ### Core Configuration | Flag | Description | Default | | ---------------------------------------- | ------------------------------------------------------- | -------- | | `--substreams-tier1-grpc-listen-addr` | gRPC listening address (append `*` for TLS) | `:10016` | | `--substreams-tier1-block-type` | Protobuf block type (e.g., `sf.ethereum.type.v2.Block`) | | | `--substreams-tier1-enforce-compression` | Require gzip or zstd encoding | `true` | ### Tier 2 Connection | Flag | Description | Default | | ------------------------------------------ | --------------------------------- | -------- | | `--substreams-tier1-subrequests-endpoint` | Address to reach tier2 workers | `:10017` | | `--substreams-tier1-subrequests-plaintext` | Use plaintext connection to tier2 | `true` | | `--substreams-tier1-subrequests-insecure` | Skip tier2 TLS validation | `false` | | `--substreams-tier1-max-subrequests` | Parallel subrequests per request | `4` | ### Request Limits | Flag | Description | Default | | ------------------------------------------------------------- | ------------------------------------------------- | ---------- | | `--substreams-tier1-active-requests-soft-limit` | Soft limit for active requests (triggers unready) | `0` (none) | | `--substreams-tier1-active-requests-hard-limit` | Hard limit (rejects requests) | `0` (none) | | `--substreams-tier1-default-max-request-per-user` | Default max requests per user | `3` | | `--substreams-tier1-default-minimal-request-life-time-second` | Minimum request lifetime | `180` | ### State Storage | Flag | Description | Default | | -------------------------------------- | ------------------------------- | ------------------------- | | `--substreams-state-store-url` | URL for Substreams state data | `{sf-data-dir}/localdata` | | `--substreams-state-store-default-tag` | Tag appended to state store URL | | | `--substreams-state-bundle-size` | Blocks between store snapshots | `1000` | | `--substreams-block-execution-timeout` | Max block execution time | `3m` | | `--substreams-tier1-quicksave-store` | Store for quicksave on shutdown | | ### Discovery & Pools | Flag | Description | Default | | --------------------------------------------------------- | -------------------------------- | ------- | | `--substreams-tier1-discovery-service-url` | Discovery service for tier2 | | | `--substreams-tier1-global-worker-pool-address` | Global worker pool address | | | `--substreams-tier1-global-worker-pool-keep-alive-delay` | Worker pool keep-alive interval | `25s` | | `--substreams-tier1-global-request-pool-address` | Global request pool address | | | `--substreams-tier1-global-request-pool-keep-alive-delay` | Request pool keep-alive interval | `25s` | | `--substreams-tier1-foundational-stores-config-path` | Foundational stores config file | | ## Substreams Tier 2 The Tier 2 component provides workers for historical block processing. ### Starting Tier 2 ```bash firecore start substreams-tier2 [flags] ``` ### Configuration | Flag | Description | Default | | ---------------------------------------------- | ------------------------------------------- | -------- | | `--substreams-tier2-grpc-listen-addr` | gRPC listening address (append `*` for TLS) | `:10017` | | `--substreams-tier2-max-concurrent-requests` | Max concurrent requests (0 = no limit) | `0` | | `--substreams-tier2-segment-execution-timeout` | Max segment execution time | `1h` | | `--substreams-tier2-discovery-service-url` | Discovery service to advertise presence | | ## Example Usage ### Basic Tier 1 Setup ```bash firecore start substreams-tier1 \ --substreams-tier1-grpc-listen-addr=":10016" \ --substreams-tier1-block-type="sf.ethereum.type.v2.Block" \ --substreams-tier1-subrequests-endpoint="localhost:10017" ``` ### Basic Tier 2 Setup ```bash firecore start substreams-tier2 \ --substreams-tier2-grpc-listen-addr=":10017" ``` ### With Cloud Storage ```bash firecore start substreams-tier1 \ --substreams-tier1-grpc-listen-addr=":10016" \ --substreams-state-store-url="s3://my-bucket/substreams-state" \ --common-merged-blocks-store-url="s3://my-bucket/merged-blocks" ``` ### With Request Limits ```bash firecore start substreams-tier1 \ --substreams-tier1-grpc-listen-addr=":10016" \ --substreams-tier1-active-requests-soft-limit=50 \ --substreams-tier1-active-requests-hard-limit=100 \ --substreams-tier1-default-max-request-per-user=5 ``` ### Multiple Tier 2 Workers Run multiple Tier 2 instances for parallel historical processing: ```bash # Worker 1 firecore start substreams-tier2 --substreams-tier2-grpc-listen-addr=":10017" # Worker 2 firecore start substreams-tier2 --substreams-tier2-grpc-listen-addr=":10018" # Worker 3 firecore start substreams-tier2 --substreams-tier2-grpc-listen-addr=":10019" ``` Configure Tier 1 to use multiple workers via discovery service or manual endpoint configuration. # Supported Protocols StreamingFast Firehose repositories ## Firehose implementation repositories * [Firehose for Ethereum](https://github.com/streamingfast/sf-ethereum), Görli, Ropsten, Polygon, BNB (by StreamingFast) * [Firehose for NEAR](https://github.com/streamingfast/sf-near) (by StreamingFast) * [Firehose for Solana](https://github.com/streamingfast/sf-solana) (by StreamingFast) * [Firehose for Arweave](https://github.com/streamingfast/firehose-arweave) (by ChainSafe) * [Firehose for Aptos](https://github.com/streamingfast/firehose-aptos) (by Aptos) * [Firehose for Antelope](https://github.com/pinax-network/firehose-antelope) (by Pinax Network) * [Firehose for StarkNet](https://github.com/starknet-graph/firehose-starknet) (by zkLend) # Naming Conventions StreamingFast Firehose naming conventions ## Firehose Naming Conventions ### Naming Conventions Intro Each Firehose setup has unique naming conventions depending on versioning and the blockchain being targeted. {% hint style="info" %} *Note: This page serves as a resource to provide a unified experience for developers working between the Firehose versions.* {% endhint %} ### Short & Long Form Naming Each Firehose setup uses two forms of naming. The naming is taken from the target blockchain's protocol name. The two forms establish a *long* and a *short* form of the protocol name. The short form will be the shortest abbreviation of the chain name possible. For Ethereum, the long form would be `ethereum` and the short form would be `eth`. {% hint style="warning" %} *Important: These naming forms will be referenced throughout the Firehose naming conventions documentation.* {% endhint %} ## Firehose Instrumentation Naming ### Instrumentation Naming in Detail For line-based Firehose instrumentations, each line of output should start with the word `FIRE` followed by a simple word defining what data to expect on each line. ### Libraries Code used for instrumentation within the native node should be bundled together using `firehose` as the name of the module/crate/package is preferable. {% hint style="warning" %} **Important**: *The node flag `--vmtrace=firehose` enables Firehose Protocol output to stdout, which the Reader Node captures and processes.* {% endhint %} ## Chain-specific Binary Changes ### Chain-specific Binary Changes Intro Replace `acme` with the chain you are instrumenting for the following items. In a sample scenario instrumenting the Tezos blockchain acme would be replaced by the two forms of Tezos; something similar to `tezos` and `tez`. ### Repository Name - `firehose-acme` If the target blockchain were Tezos the name would be `firehose-tezos`. ### Binary Name -`fireeth` The binary name will be the first half of the Firehose product name "fire" combined with the short form of the target blockchain. If the target blockchain were Tezos, the name of the binary would be something similar to `firetez`. ### Protocol Buffers Schema The top-level Block protobuf definition convention for a new chain is `sf.[chain].type.v1.Block.`\ `\` For Tezos it would be `sf.tezos.type.v1.Block` ### Directory Layouts ***Proto Directory -*** `/proto` The proto directory contains properly namespaced protobuf definitions for the target chain.\ \ For example `proto/sf/acme/type/v1/type.proto` as the first, and often only, file.\ \ For Tezos it would be `proto/sf/tezos/type/v1/type.proto`\\ ***Types Directory -*** `/types` The types directory contains rendered protobuf types and some helpers.\ \ For example: `/types/pb/sf/acme/type/v1;pbacme`, using the *short form* package name prefixed with `pb`.\ \ For Tezos it would be something similar to:\ `/types/pb/sf/acme/type/v1;pbtezos`\\ `/types/go.mod`: to be able to import `github.com/streamingfast/firehose-acme/types` and pull only a limited number of dependencies. ***Codec Directory -*** `/codec` The `codec` directory contains all of the coding and decoding methods used to manipulate the stream of data coming from the Firehose-enabled Blockchain Node. *Note,* t\_his library is concerned only with the data wrangling, and not the management of nodes.\_ # Schemas StreamingFast Firehose schemas ## Chain-specific schemas Firehose currently has Layer 1 Protocol Buffer schemas for several different blockchains. Follow the links below to find the schemas relevant to the blockchain being targeted. * [Ethereum Protobuf Definitions](https://github.com/streamingfast/firehose-ethereum/blob/develop/proto/sf/ethereum/type/v2/type.proto) * [NEAR Protobuf Definitions](https://github.com/streamingfast/firehose-near/blob/develop/proto/sf/near/type/v1/type.proto) * Solana Protobuf Definitions: * [Solana Block data model](https://github.com/streamingfast/sf-solana/blob/develop/proto/sf/solana/type/v1/type.proto) (original) * [Solana Account Changes data model](https://github.com/streamingfast/firehose-solana/blob/develop/proto/sf/solana/type/v1/account.proto) (account changes only) * [Arweave Protobuf Definitions](https://github.com/streamingfast/firehose-arweave/blob/develop/proto/sf/arweave/type/v1/type.proto) * [Aptos Protobuf Definitions](https://github.com/aptos-labs/aptos-core/blob/main/crates/aptos-protos/proto/aptos/extractor/v1/extractor.proto) ## The `bstream` Block The `bstream` Block is the main blockchain-agnostic object flowing throughout Firehose. {% hint style="info" %} **Note**: *The* `stream` *Block is the envelope used to pass blockchain-specific Block objects, for Ethereum, as an example.* {% endhint %} ## Versioning ### ***Coming Soon*** Discussion on general multi-chain considerations of versioning in the Firehose suite to answer questions such as: * when do we change the namespace, * when do we bump the `version` field in the `block`. # Indexing Firehose indexing (redirects to legacy documentation) {% hint style="warning" %} **Deprecation Notice**: Firehose indexing and transforms are legacy features primarily used for graph-node integration. For new projects, use [Substreams](https://substreams.streamingfast.io/) for data filtering and transformation. {% endhint %} For documentation on legacy Firehose indexing, see [Legacy Firehose Indexing](/references/legacy-indexing). # Legacy Firehose Indexing Legacy Firehose indexing and transforms (deprecated) {% hint style="warning" %} **Deprecation Notice**: Firehose indexing and transforms are legacy features primarily used for graph-node integration. For new projects, use [Substreams](https://substreams.streamingfast.io/) for data filtering and transformation instead. {% endhint %} ## Overview Firehose can generate block indexes that optimally serve requests containing filtering parameters. This feature was designed primarily for graph-node integration and is not recommended for general use. ## How Indexes Work Firehose indexes function in two primary ways: 1. **Block skipping**: Block files aren't read for ranges within block indexes that do not match the filter provided. 2. **Payload reduction**: Block files containing both matching and non-matching transactions return a reduced payload of only matched transactions. ## Enabling Indexes {% hint style="info" %} Blockchains have varying levels of support for indexes. For example, `firehose-ethereum` has support while many other chains do not. {% endhint %} ### Index Configuration Flags Use the following flag for valid index bundle sizes when looking for block indexes: ``` --common-index-block-sizes [ints] ``` Default values: `100000,10000,1000,100` Common store URL to read and write index files: ``` --common-index-store-url [string] ``` Default: `file://{sf-data-dir}/storage/index` ## Index Builder The Firehose Index Builder generates index files from merged blocks. It can be run: * **Sequentially**: To produce indexes as merged-blocks are produced * **In parallel**: Multiple instances over different block ranges to quickly process millions of blocks ### Running the Index Builder ```bash fireeth start combined-index-builder \ --combined-index-builder-index-size=10000 \ --combined-index-builder-start-block=1000000 \ --combined-index-builder-stop-block=2000000 \ --common-index-store-url=s3://mybucket/mainnet-indexes \ --common-blocks-store-url=s3://mybucket/mainnet-blocks ``` ## Transforms Transforms are Protocol Buffer definitions used to locate specific blocks according to search criteria. They are used by graph-node to filter blockchain data before indexing. ### Ethereum Transforms For Ethereum, transforms support filtering by: * **Log filters**: Match transactions containing logs with specific addresses and topics * **Call filters**: Match transactions containing calls to specific addresses ### Example: Log Filter Match transactions with logs from a specific address and topic: ```bash fireeth tools firehose-client api.streamingfast.io:443 15289746 15307883 \ --log-filters=0xa5b7f12346048e8a3e780dbeb4c2f469be8ffcef:0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef ``` ### Example: Call Filter Match transactions containing calls to specific addresses: ```bash fireeth tools firehose-client api.streamingfast.io:443 15290180 15290300 \ --call-filters=0xa5b7f12346048e8a3e780dbeb4c2f469be8ffcef+0xfeaf24248e04ac7ad0ea6e7e617182cff429d4e5: ``` ### Transform Protocol Buffers Transforms are defined in the chain-specific protobuf definitions. For Ethereum, see the `CombinedFilter` in: [sf/ethereum/transform/v1/transforms.proto](https://github.com/streamingfast/firehose-ethereum/blob/develop/proto/sf/ethereum/transform/v1/transforms.proto) ## graph-node Integration The primary use case for indexing and transforms is graph-node integration. When running graph-node with Firehose: 1. graph-node sends transform filters based on subgraph manifest requirements 2. Firehose uses indexes to skip irrelevant block ranges 3. Only matching data is returned to graph-node for indexing For graph-node setup with Firehose, refer to [The Graph documentation](https://thegraph.com/docs/). ## Migration to Substreams For new data filtering and transformation needs, use Substreams instead: * **Better performance**: Parallel processing with caching * **More flexibility**: Custom Rust/WASM logic * **Ecosystem support**: Multiple output sinks available See the [Substreams documentation](https://docs.substreams.dev) for details. # FAQ StreamingFast Firehose frequently asked questions ## **What is Firehose?** Firehose is an extremely efficient blockchain data indexing solution. Rich data is extracted from blockchain nodes and saved to simple flat files providing capture and processing speeds previously thought to be impossible. Firehose is primarily written in the Go programming language and takes full advantage of the parallel computing aspects available. Firehose makes paramount improvements in the speed and performance of data availability for any blockchain. ## **What is Firehose for?** Firehose is available for developers creating decentralized applications, creating blockchain data-related solutions, or wanting to capture data from non-Firehose instrumented blockchains. ## **Do I need Substreams to use Firehose?** Substreams and Firehose can be used together and typically are. The two complement each other in terms of the functionality they provide. Substreams uses blockchain data extracted and provided by Firehose. Firehose is responsible for extraction and provision and Substreams handles transforming and manipulating the data. ## **Does Firehose work with GraphQL?** Firehose does not work directly with GraphQL. Firehose extracts data from blockchain nodes and provides it to Substreams. Substreams sinks are responsible for bringing data to storage engines that can then be queried by different means. ## **Do subgraphs work with Firehose?** Yes, the `graph-node` software, which powers Subgraphs, can connect directly to Firehose for Ethereum and power the traditional Subgraphs there. ## **Is Firehose made by The Graph?** Firehose was created by StreamingFast working as a core developer with The Graph Foundation. ## **How do I get a Firehose authentication token?** Authentication tokens are required to connect to the public Firehose endpoint provided by StreamingFast made available to developers for testing. Full instructions for obtaining a StreamingFast authentication token are available in the Substreams documentation. ## **My Firehose authentication token isn’t working, what do I do?** The StreamingFast team is available in Discord to assist with problems related to obtaining or using authentication tokens. [https://discord.gg/Ugc7KtkA](https://discord.gg/jZwqxJAvRs) The authentication documentation also provides general instructions surrounding authentication tokens. ## **How much does Firehose cost?** Firehose is made available as an open-source project, published under the Apache 2.0 license. As such, there are no direct fees involved. Running the Firehose and serving live blockchain data is a service that is offered by StreamingFast and other providers, with the goal of being served directly on The Graph network. ## **Does StreamingFast have a Discord?** Yes! [Join the StreamingFast Discord today](https://discord.gg/jZwqxJAvRs)! ## **Is StreamingFast on Twitter?** Yes! Find StreamingFast on their official Twitter account: ## **Is StreamingFast on YouTube?** Yes! Find StreamingFast on their official YouTube account: ## **Who is dfuse?** StreamingFast was originally called dfuse. The company changed the name and is in the process of phasing the old brand out. ## **Who is StreamingFast?** [StreamingFast](https://streamingfast.io) is a protocol infrastructure company that provides a massively scalable architecture for streaming blockchain data. StreamingFast is one of the core developers working within [The Graph ](https://thegraph.com)ecosystem. ## What limits and SLAs are there for Firehose API keys? The public StreamingFast endpoints are rate-limited and offer no guaranteed support or SLAs. Reach out to us for production usage keys. ## Is Firehose a production-ready service? Yes, the Firehose has been battle tested for many years and is ready for prime time. Its sibling technology, [Substreams](https://docs.substreams.dev/), is also ready for prime time. Reach out to us for production usages. # Change logs ### Software releases * [Firehose Core Changelog](https://github.com/streamingfast/firehose-core/blob/develop/CHANGELOG.md) * [Firehose Ethereum Changelog](https://github.com/streamingfast/firehose-ethereum/blob/develop/CHANGELOG.md) * [Firehose Solana Changelog](https://github.com/streamingfast/firehose-solana/blob/develop/CHANGELOG.md) * [Firehose NEAR Changelog](https://github.com/streamingfast/firehose-near/blob/develop/CHANGELOG.md) * [Firehose Arweave Changelog](https://github.com/streamingfast/firehose-arweave/blob/develop/CHANGELOG.md) ### Data releases This section documents the updates in data models an endpoints data revisions. * [Nov 8th 2023 - Endpoint update: fixed Polygon data model rolled out on StreamingFast endpoints.](/release-notes/change-logs/nov-8th-2023-polygon-update) # Nov 8th 2023 Polygon Update ### Firehose Polygon and Mumbai blocks were missing some StateSync transactions. #### Overview * A bug in the firehose-ethereum implementation caused some "StateSync" transactions (polygon bridge deposits) to be missing. Their logs were, instead, shown under different generated transaction hashes, with the wrong log number. * The bug was fixed in [firehose-ethereum release v1.4.16](https://github.com/streamingfast/firehose-ethereum/releases/tag/v1.4.16) (released on September 29th, 2023) * The Polygon blocks were fully reprocessed on our StreamingFast's `polygon.streamingfast.io:443` endpoint on Nov. 8th, 2023 #### How does it impact you ? * If you are a firehose operator, you will need to reprocess the whole chain to replace your merged-blocks-files. * If you are consuming a Substreams: 1. **determine if that Substreams is affected by the existence or the order of these StateSync transactions** (ex: `0x8f3e05c5af7d601a6015c4fdbb68a04cbf0305fe2740920ce70f81ed1943a194`) 2. **evaluate if the discrepancies in past blocks are impactful to your business need** * If both previous statements apply to you: 1. Make sure that the Substreams provider that you query has already upgraded his Polygon blocks 2. Make a small change to your Substreams code (anything will do), recompile and repackage your .spkg: this will cause the module hashes to change and effectively invalidate the cache. * We will not delete previous Substreams "caches" that were generated with the faulty blocks. If you believe some widely-used Substreams is affected and should have its cache pruned, you can [contact us on Discord](https://discord.gg/jZwqxJAvRs). #### More context... * Explanation of Polygon state-sync events: * How Firehose handle these: * a) Polygon transactions are done in parallel in the `bor` client, so they must be reordered after the fact (using special code in the `firehose reader` that mimics the `bor` logic) * b) The polygon StateSync events are not handled like normal transactions: A "virtual" transaction is created at the end of the block, with its hash being computed as `Keccak256("matic-bor-receipt-" + BlockNumber + BlockHash)`. * c) There are other "system events" that are emitted from the firehose-instrumented `bor` client, which are NOT bundled as part of the state-sync transaction. These are the ones that happen every 6400 block sand involve the polygon Validator Contract `0x0...1000` They are not shown in rpc get\_block or get\_transaction but they do affect the chain state, so they are in the firehose blocks. * There were three issues in our polygon-specific implementation: 1. The reordering of transactions (a) was done BEFORE the polygon system transactions were bundled together, but we only checked the very last transaction in the block to see if it was a StateSync event. When there was an event to the Polygon Validator Contract `0x0...1000`, we the transaction "bundling" was not triggered. 2. When generating the receipt logs for the "virtual system transaction", we were using the Ordinals as a reference, like we do for every other transaction. However, the way that polygon reconstructs this header in the getLogs() rpc call is different, so we must fullow the original "BlockIndex" from the reader instead of using the Ordinal. 3. The "other system events" (c) would skew the other transactions index number, since the RPC endpoint does not show them. Moving that special system transaction to the end solved this. * See the following issue for more details: #### Example impacted transactions: * * *