Single-Machine Deployment

Firehose is capable of indexing Injective data through a valid Injective RPC endpoint.

Before You Begin

If you are not familiar with Firehose, read the Overview page, which should give you enough information about the main pieces involved in spinning up a Firehose environment.

Make sure you have Go installed, as you will need to build the Injective Firehose binary from source.

Spin Up an Injective Environment

Download and Install the Dependencies

1. Download the latest Firecore binary. Make sure you download the right binary for your computer (ARM or x86). Firecore is a Go project, so you can always build the binary from source.

2. Add the Firecore binary to the $PATH of your system. Verify that the command works:

$ firecore --version
firecore version 1.5.3 (Commit 1563afc, Built 2024-06-20T18:39:14Z)

1. Clone the Firehose Cosmos GitHub repository, which you will use to generate the Injective-specific Firehose binary.

2. Build the fireinjective binary:

go install injective/cmd/fireinjective

This will generate a new binary in the Go directory of your system. In Mac, it is located at ~/go/bin by default.

1. Move the generated binary to the $PATH of your system. Verify that the command works:

$ fireinjective --help
firecosmos fetching and tooling

Usage:
  firecosmos [command]

Available Commands:
  completion  Generate the autocompletion script for the specified shell
  help        Help about any command
  injective   firecosmos for injective chain

Flags:
  -h, --help   help for firecosmos

Additional help topics:
  firecosmos tools      Developer tools for operators and developers

Now, you have all the necessary tools installed.

Start the Firehose Process

In this tutorial, you will deploy all the Firehose components in the same machine (i.e. this is a single-machine deployment). This is NOT the recommended set-up, as you may run into memory issues.

To avoid data corruption issues, you will deploy two Firecore processes:

• Process 1 will run the following components: reader-node, merger and relayer.

• Process 2 will run the following components: firehose, substreams-tier1 and substreams-tier2.

These two processes will share the same folder to persist the data, which is called firehose-data by default. It is important that both processes have concurrent access to the data folder.

1. Create a new folder, which you will be used to host all the Firehose data generated by the extraction. Make sure the folder has enough permissions to persist data in the filesystem.

$ mkdir my-injective-firehose

1. Create the configuration file for Process 1, called reader-merger-relayer.yaml:

start:
  args: # 1.
  - reader-node
  - merger
  - relayer
  flags:
    common-first-streamable-block: 72560211 # 2.
    reader-node-path: fireinjective # 3.
    reader-node-arguments:
      injective fetch rpc 72560211 # 4.
      --state-dir "{node-data-dir}/poller/states"
      --block-fetch-batch-size=1 # 5.
      --endpoints=<RPC endpoint 1> # 6.
      --endpoints=<RPC endpoint 2>
      --endpoints=<RPC endpoint 3>

1. Specifies the components to spin up.

2. Specifies the first block to stream.

3. Specifies the Injective-specific binary location (in this case, it is globally accesible in the system).

4. Specifies the arguments of the fireinjective binary. In this case, the RPC Poller mode is used with 72560211 as starting block number.

5. Specifies how many blocks must be fetched from the RPC at a time. For example, if it is set to 20, a batch of 20 blocks will be requested to the RPC.

6. Specifies the RPC endpoint. You can specify several endpoints.

7. Create the configuration file for Process 2, called firehose-substreams.yaml:

start:
  args: # 1.
  - firehose
  - substreams-tier1
  - substreams-tier2
  flags:
    common-first-streamable-block: 72560211 # 2.
    common-live-blocks-addr: localhost:10014 # 3.
    substreams-tier1-block-type: sf.cosmos.type.v2.Block # 4.
    substreams-tier1-grpc-listen-addr: :9000 # 5.

1. Specifies the components to spin up.

2. Specifies the first block to stream.

3. Specifies the address where to reach the relayer component running in its own isolated Firehose process.

4. substreams-tier1 component flag: specifies the data model of the extracted data.

5. substreams-tier1 component flag: specifies where the Substreams server listens for connections.

6. Lastly, start two Firecore processes in two different command-line terminals.

firecore start -c reader-merger-relayer.yaml

firecore start -c firehose-substreams.yaml