LogoLogo
  • Firehose
  • Introduction
    • Firehose Overview
    • Prerequisites
  • Architecture
    • Components
      • Firehose-enabled Node
      • Reader
      • Merger
      • Relayer
      • gRPC Server
      • High Availability
    • Data Flow
    • Data Storage
  • Firehose Setup
    • Overview
    • Ethereum
      • Installation
      • Single-Machine Deployment
      • Reprocessing history
      • Synchronization
    • Injective
      • Single-Machine Deployment
    • NEAR
      • Single-Machine Deployment
    • Solana
      • Single-machine Deployment
    • System Requirements
  • Community Integrations
    • Starknet
      • Networks and nodes
      • Local deployment with Docker
      • Local deployment without Docker
  • Integrate New Chains
    • Benefits
    • Integration overview
    • Design Principles
    • Firehose Acme
  • References
    • Supported Protocols
    • Naming Conventions
    • Schemas
    • Indexing
    • FAQ
  • Release Notes
    • Change logs
      • Nov 8th 2023 Polygon Update
Powered by GitBook
On this page
  • Firehose-enabled Blockchain Node
  • Reader
  • Data Aggregation
  • Component Cooperation
  • Merger
  • Relayer
  • Firehose gRPC Server

Was this helpful?

Edit on GitHub
  1. Architecture
  2. Components

High Availability

High availability for StreamingFast Firehose components

PreviousgRPC ServerNextData Flow

Last updated 2 years ago

Was this helpful?

Firehose-enabled Blockchain Node

Coming soon.

Reader

Placing multiple components side by side, and fronted by one or more Relayers, allows for highly available setups; a core attribute of the Firehose design.

A Relayer connected to multiple Readers will deduplicate the incoming stream and push the first block downstream.

Tip: Two Reader components will even race to push the data first. The system is designed to leverage this racing feature to the benefit of the end-user by producing the lowest latency possible.

Data Aggregation

Firehose also aggregates any forked blocks that would be seen by a single Reader component, and not seen by any other components.

Component Cooperation

Adding Reader components and dispersing each one geographically will result in the components actually racing to transfer blocks to the Relayer component. This cooperation between the and components significantly increases the performance of Firehose.

Merger

A single component is required for Reader nodes in a highly available Firehose.

Highly available systems usually connect to the component to receive real-time blocks. Merged blocked files are used when Relayer components can't provide the requested data or satisfy a range.

Note: Merged blocks generally aren't read by other Firehose components in a running, live highly available system.

Relayer

Firehose gRPC Server

Firehose can be scaled horizontally to provide a highly available system.

The network speed and data throughput between consumers and Firehose deployments will dictate the speed of data availability.

Understanding how data flows through Firehose is beneficial for harnessing its full power.

Restarts from other components can be sustained and time provided for components to be down when Relayer components provide 200 to 300 blocks in RAM.

A component in a highly available Firehose will feed from all of the Reader nodes to gain a complete view of all possible forks.

Tip: Multiple components will ensure blocks are flowing efficiently to the component and throughout Firehose.

Note: The network speed and data throughput between components and components will impact the speed of data availability.

Firehose components have the ability to connect to a subset of Relayer components or all Relayers available.

When the Firehose component is connected to all available Relayer components the probability that all forks will be viewed increases. Inbound requests made by consumers will be fulfilled with in-memory fork data.

Block navigation can be delayed when forked data isn't completely communicated to the Firehose component.

Reader
Reader
Reader
Reader
Relayer
Merger
Relayer
Merger
Relayer
Reader
Relayer
Relayer
Firehose gRPC Server
gRPC Server
gRPC Server
gRPC Server