Install & Run
Last updated
Last updated
Follow these steps to get started and run the Hemera Indexer:
git clone https://github.com/HemeraProtocol/hemera-indexer.git
To operate the Hemera Indexer, you essentially have two approaches to choose from, each suited to different levels of customization and control.
1. Use Docker
2.Build from Source code
If you have trouble running the following commands, consider referring to the official docker installation guide for the latest instructions.
# Add Docker's official GPG key:
sudo apt-get update
sudo apt-get install ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc
# Add the repository to Apt sources:
echo \
"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
$(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
# Install docker and docker compose
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
cd hemera_indexer
cd docker-compose
Alternatively, you might want to edit environment variables in docker-compose.yaml
. Please check out the configuration manual on how to configure the environment variables.
vim docker-compose.yaml
Now, run the following command to spin up the containers.
sudo docker compose up
You should be able to see similar logs from your console that indicate Hemera Indexer is running properly.
[+] Running 2/0
✔ Container postgresql Created 0.0s
✔ Container hemera Created 0.0s
Attaching to hemera, postgresql
postgresql |
postgresql | PostgreSQL Database directory appears to contain a database; Skipping initialization
postgresql |
postgresql | 2024-06-24 08:18:48.547 UTC [1] LOG: starting PostgreSQL 15.7 (Debian 15.7-1.pgdg120+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit
postgresql | 2024-06-24 08:18:48.548 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
postgresql | 2024-06-24 08:18:48.549 UTC [1] LOG: listening on IPv6 address "::", port 5432
postgresql | 2024-06-24 08:18:48.554 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
postgresql | 2024-06-24 08:18:48.564 UTC [27] LOG: database system was shut down at 2024-06-24 06:44:23 UTC
postgresql | 2024-06-24 08:18:48.575 UTC [1] LOG: database system is ready to accept connections
hemera | 2024-06-24 08:18:54,953 - root [INFO] - Using provider https://eth.llamarpc.com
hemera | 2024-06-24 08:18:54,953 - root [INFO] - Using debug provider https://eth.llamarpc.com
hemera | 2024-06-24 08:18:55,229 - alembic.runtime.migration [INFO] - Context impl PostgresqlImpl.
hemera | 2024-06-24 08:18:55,230 - alembic.runtime.migration [INFO] - Will assume transactional DDL.
hemera | 2024-06-24 08:18:55,278 - alembic.runtime.migration [INFO] - Context impl PostgresqlImpl.
hemera | 2024-06-24 08:18:55,278 - alembic.runtime.migration [INFO] - Will assume transactional DDL.
hemera | 2024-06-24 08:18:56,169 - root [INFO] - Current block 20160303, target block 20137200, last synced block 20137199, blocks to sync 1
hemera | 2024-06-24 08:18:56,170 - ProgressLogger [INFO] - Started work. Items to process: 1.
hemera | 2024-06-24 08:18:57,505 - ProgressLogger [INFO] - 1 items processed. Progress is 100%.
hemera | 2024-06-24 08:18:57,506 - ProgressLogger [INFO] - Finished work. Total items processed: 1. Took 0:00:01.336310.
hemera | 2024-06-24 08:18:57,529 - exporters.postgres_item_exporter [INFO] - Exporting items to table block_ts_mapper, blocks end, Item count: 2, Took 0:00:00.022562
hemera | 2024-06-24 08:18:57,530 - ProgressLogger [INFO] - Started work.
Skip this step if you already have both installed.
sudo apt update
sudo apt install python3
sudo apt install python3-pip
Skip this step if you don't want to have a dedicated python venv for Hemera Indexer.
sudo apt install python3-venv
python3 -m venv ./venv
source ./venv/bin/activate
sudo apt install libpq-dev
pip install -e .
Another option is to set up and build from source using the make script
You can use the command: make development
Hemera Indexer requires a PostgreSQL database to store all indexed data. You may skip this step if you already have a PostgreSQL set up.
Follow the instructions about how to set up a PostgreSQL database here: Setup PostgreSQL on Ubuntu.
Configure the OUTPUT
or --output
parameter according to your PostgreSQL role information. Check out Configure Hemera Indexer for details.
E.g. postgresql+psycopg2://${YOUR_USER}:${YOUR_PASSWORD}@${YOUR_HOST}:5432/${YOUR_DATABASE}
.
Please check out Configure Hemera Indexer on how to configure the indexer.
python hemera.py stream \
--provider-uri https://eth.llamarpc.com \
--debug-provider-uri https://eth.llamarpc.com \
--postgres-url postgresql+psycopg2://devuser:devpassword@localhost:5432/hemera_indexer \
--output jsonfile://output/eth_blocks_20000001_20010000/json,csvfile://output/hemera_indexer/csv,postgresql+psycopg2://devuser:devpassword@localhost:5432/eth_blocks_20000001_20010000 \
--start-block 20000001 \
--end-block 20010000 \
--output-types block,transaction,log \
--block-batch-size 200 \
--batch-size 50 \
--max-workers 8
Once you have successfully bootstrapped Hemera Indexer, you should be able to view similar logs as below.
2024-06-25 16:37:38,456 - root [INFO] - Using provider https://eth.llamarpc.com
2024-06-25 16:37:38,456 - root [INFO] - Using debug provider https://eth.llamarpc.com
2024-06-25 16:37:38,485 - alembic.runtime.migration [INFO] - Context impl PostgresqlImpl.
2024-06-25 16:37:38,485 - alembic.runtime.migration [INFO] - Will assume transactional DDL.
2024-06-25 16:37:38,502 - alembic.runtime.migration [INFO] - Context impl PostgresqlImpl.
2024-06-25 16:37:38,502 - alembic.runtime.migration [INFO] - Will assume transactional DDL.
2024-06-25 16:37:39,485 - root [INFO] - Current block 20167548, target block 20137200, last synced block 20137199, blocks to sync 1
2024-06-25 16:37:39,486 - ProgressLogger [INFO] - Started work. Items to process: 1.
2024-06-25 16:37:40,267 - ProgressLogger [INFO] - 1 items processed. Progress is 100%.
2024-06-25 16:37:40,268 - ProgressLogger [INFO] - Finished work. Total items processed: 1. Took 0:00:00.782177.
2024-06-25 16:37:40,283 - exporters.postgres_item_exporter [INFO] - Exporting items to table block_ts_mapper, blocks end, Item count: 2, Took 0:00:00.014799
2024-06-25 16:37:40,283 - ProgressLogger [INFO] - Started work.