Firehose Acme
StreamingFast Firehose template sample for A Company that Makes Everything
Instrumenting a new chain from scratch requires the node native code to be instrumented to output Firehose logs, but this is only one side of the coin. A Firehose instrumentation of a new chain requires also a
firehose-<chain> program that contains chain-specific code to read the data output by the instrumented node, and serves data throughout the Firehose stack.This
firehose-<chain> is a Golang project that contains the CLI, the reader code necessary to assemble Firehose node output into chain-specific Blocks, and a bunch of other small boilerplate code around the Firehose set of libraries.To ease the work of Firehose implementors, we provide a "template" project firehose-acme that is the main starting point for instrumenting new, unsupported blockchain nodes.
It consists of basic code and a Dummy Blockchain prototype. The idea is that you can play with this firehose-acme instance to see blocks produced and test some Firehose behaviors.
Clone the repository:
git clone [email protected]:streamingfast/firehose-acme
Install the
fireacme binary:cd firehose-acme
go install ./cmd/fireacme
And validate that everything is working as expected:
fireacme --version
fireacme version dev (Built 2023-02-02T13:42:20-05:00)
Obtain the Dummy Blockchain by installing from source:
go install github.com/streamingfast/dummy-blockchain@latest
And validate that it was installed correctly:
dummy-blockchain --version
dummy-blockchain version 0.0.1 (build-commit="-" build-time="-")
A simple shell script that starts
firehose-acme with sane default is located at devel/standard/start.sh. The configuration file used to launch all the applications at once is located at devel/standard/standard.yamlRun the script from your local cloned
firehose-acme version as done in firehose-acme installation section:./devel/standard/start.sh
The following messages will be printed to the terminal window if:
- All of the configuration changes were made correctly,
- All system paths have been set correctly,
- And the Dummy Blockchain was installed and set up correctly.
2023-02-02T13:54:25.882-0500 INFO (dtracing) registering development exporters from environment variables
2023-02-02T13:54:25.882-0500 INFO (fireacme) starting Firehose on Acme with config file 'standard.yaml'
2023-02-02T13:54:25.882-0500 INFO (fireacme) launching applications: firehose,merger,reader-node,relayer
start --store-dir=/Users/maoueh/work/sf/firehose-acme/devel/standard/firehose-data/reader/data --firehose-enabled --block-rate=60
...
2023-02-02T13:54:25.883-0500 INFO (reader) created acme superviser {"superviser": {"binary": "dummy-blockchain", "arguments": ["start", "--store-dir=/Users/maoueh/work/sf/firehose-acme/devel/standard/firehose-data/reader/data", "--firehose-enabled", "--block-rate=60"], "data_dir": "/Users/maoueh/work/sf/firehose-acme/devel/standard/firehose-data/reader/data", "last_block_seen": 0, "server_id": ""}}
...
2023-02-02T13:54:25.884-0500 INFO (reader) creating new command instance and launch read loop {"binary": "dummy-blockchain", "arguments": ["start", "--store-dir=/Users/maoueh/work/sf/firehose-acme/devel/standard/firehose-data/reader/data", "--firehose-enabled", "--block-rate=60"]}
...
2023-02-02T13:54:28.677-0500 INFO (bstream) hub is now Ready
2023-02-02T13:54:28.677-0500 INFO (firehose) launching gRPC firehose server {"live_support": true}
2023-02-02T13:54:28.677-0500 INFO (firehose) launching gRPC server {"listen_addr": ":18015"}
2023-02-02T13:54:28.677-0500 INFO (firehose) serving gRPC (over HTTP router) (plain-text) {"listen_addr": ":18015"}
2023-02-02T13:54:28.899-0500 INFO (reader.acme) level=info msg="processing block" hash=4e07408562bedb8b60ce05c1decfe3ad16b72230967de01f640b7e4729b49fce height=3
2023-02-02T13:54:29.897-0500 INFO (reader.acme) level=info msg="processing block" hash=4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a height=4
2023-02-02T13:54:30.897-0500 INFO (reader.acme) level=info msg="processing block" hash=ef2d127de37b942baad06145e54b0c619a1f22327b2ebbcfbec78f5564afe39d height=5
...
To integrate the target blockchain modify
devel/standard/standard.yaml and change the start.flags.mindreader-node-path flag to point to the custom integration's blockchain node binary.Update the proto file
sf/acme/type/v1/type.proto to model your chain's data model.After updating the references to "Acme" the Protocol Buffers need to be regenerated. Use the
generate shell script to make the updates../types/pb/generate.sh
The
console_reader.go file is the interface between the instrumented node's output and the Firehose ingestion process.Each blockchain has specific pieces of data, and implementation details that are paticular to that blockchain. Reach out to us if you need guidance here.
Important: Studying the StreamingFast Ethereum and other implementations and instrumentations should serve as a foundation for other custom integrations.
After completing all of the previous steps the base integration is ready for initial testing.
go test ./...
If all changes were made correctly the updated project should compile successfully.
You can reach out to the StreamingFast team on Discord. We usually maintain these Go-side integrations and keep them up-to-date. We can review, and do the renames as needed.
You can also rename the project and all files and references to
acme to your own chain's name. Choose two names, a long-form and a short form for the custom integration following the naming conventions outlined below.For example:
arweaveandarw
Then finalize the rename:
- Rename
cmd/fireacme->cmd/firearw(short form) - Search and replace
fireacme=>firearw(short form) - Conduct a global search and replace from:
acme=>arweave(long form) - Conduct a global search to replace
ACME=>ARWEAVE(long form) - Conduct a global search to replace
Acme=>Arweave(long form)
Last modified 2mo ago