# Introduction

Learn about Hemera Protocol, the programmable semantic data layer, powered by the first Account-centric Indexing protocol.

### Introducing Hemera <a href="#our-mission" id="our-mission"></a>

Hemera Protocol turns crypto data into semantic values, thus providing utilities powering the decentralized AI future. The key to decentralizing AI resides in first democratizing data ownership and access, which is a given yet underutilized feature of web3. The biggest blocker of combining Crypto with AI lies in the absence of a public goods data infrastructure dedicated to streamlining semantic on-chain data processing.

Hemera creates account-centric indexing protocol to abstract away low-level on-chain data structures and smart contract logics for developers, while allowing them to express new assets and contracts in the form of User-defined Functions (UDFs) and share them with the protocol network. Powered by high performance indexing network, UDFs are updated continuously in real-time, allowing developers to build a wide range of Dapps. Hemera Protocol Network saves a small data team (from 1 to 5 people) for each end-user facing web3 team.

{% embed url="<https://www.youtube.com/watch?v=zCGTcaMX-5s>" %}

### The "Why" <a href="#who-we-are" id="who-we-are"></a>

While blockchains excel at storing transaction data, today's AI systems depend heavily on account-level (user-level) semantic data to deliver personalized experiences. Features such as recommendation engines, search algorithms, and personalized AI assistants rely on individual user preferences and needs. However, the absence of infrastructure facilitating account-level semantic data retrieval poses a significant barrier in meeting these requirements.

Today the crypto world is already paying heavy prices for account-level data retrieval challenges:

* **New blockchains:** every new chain needs a block explorer, yet there’s no high performance and affordable options in the market now. The leading block explorer costs millions of dollars per year, leaving the majority of new L1s and rollups out of options. Even more pressing is when high performance EVM chains are online, existing block explorer pricing model won't scale to the needs.&#x20;
* **Dapp builders, including wallets, gamefi, socialfi etc.:** all consumer Dapps need access to user on-chain profiles, including asset holding history, social graph etc., yet building in-house functionalities can be as expensive as USD 50k per months or more including labor and server costs ([see costs estimations here](/welcome/account-centric-indexing-protocol/why-account-centric-indexing)).
* **End users have high barriers and costs to access on-chain intelligence:**  end users are constantly looking across chain ecosystems for new assets to trade on, new communities to join etc. Yet accessing those information at scale is time consuming, requires coding capabilities sometimes, and may require expensive subscription to 3rd party softwares.

### *The "How"*

Hemera proposes the first [Account-centric indexing](broken://pages/Rp0V0xjhXC1ruxBTvblx) protocol, which creates a programmable and portable crypto data network that is not supported by traditional indexing protocols and providers.

### Our Mission <a href="#our-mission" id="our-mission"></a>

Hemera is on a mission to “accelerate the advent of data and AI democratization”.

For too long, user data have been confined within the walled gardens of Internet giants, without us owning them. With Web3, user data is being democratized from these incumbents for the first time, and our goal is to accelerate this paradigm shift.

### **Connect with Us** <a href="#connect-with-us" id="connect-with-us"></a>

Join the conversation on our social media channels and be part of the Hemera community:

**Twitter (Hemera)**: <https://twitter.com/HemeraProtocol>

**Twitter (SocialScan)**: <https://twitter.com/socialscan_io>

**Discord:** [https://discord.gg/](https://discord.gg/6zYzU9v8uY)[socialscan](https://t.co/SdwJmd7lVU)

### Get Started <a href="#get-started" id="get-started"></a>

Unlock your potential with the tools and resources offered by Hemera Protocol. From simplifying data interoperability to offering a decentralized infrastructure, Hemera provides a solid foundation for creating innovative decentralized applications.

For developers ready to dive in, check out these quick start resources:

* **Explore Account-Centric Indexing (ACI):** [ACI Quick Start](/welcome/account-centric-indexing-protocol/what-is-account-centric-indexing)
* **Developing User-Defined Functions (UDFs):** [UDF Guide](/udfs-user-defined-functions/building-user-defined-functions-udf)

<br>


# Quick Start

The Hemera Protocol offers a streamlined approach for developers to build data-driven dApps. The Hemera Indexer, a core component of this protocol, facilitates parallel execution of **User-Defined Functions** (UDFs) alongside essential indexing tasks, enhancing performance and simplifying complex data workflows.

To get started with Hemera Indexer and UDFs, you can follow this guide:&#x20;

## **Set Up Your Development Environment**

* **Clone the Repository**:

```bash
git clone https://github.com/HemeraProtocol/hemera-indexer
```

* **Install Python 3.9**:

  Ensure Python 3.9 is installed on your system. You can download it from the official Python website.<br>

{% hint style="info" %}
Building from Source is recommended when getting started with UDFs.&#x20;
{% endhint %}

## **Building from Source**

* **Install Dependencies and Activate Environment** :

  Navigate to the cloned repository and run:

  ```bash
  make development
  source <your-environment>
  ```

  Replace `<your-environment>` with the name of your virtual environment.<br>
* **Switch Branches**:

  Depending on your development needs, you may switch between branches:

  ```bash
  git checkout <branch-name>
  ```

  Replace `<branch-name>` with the desired branch.

## **Writing a UDF**

User-Defined Functions (UDFs) in Hemera are designed to streamline the extraction, transformation, and analysis of blockchain data. Here’s how you can create a UDF by defining a `dataclass`, integrating a `model`, and writing the `job logic`.

### **Define a Dataclass**

A `dataclass` is used to structure the data you will process. It defines the schema for the blockchain data extracted by the Hemera Indexer.

```python
from dataclasses import dataclass

@dataclass
class TransactionData:
    block_number: int
    transaction_hash: str
    sender: str
    receiver: str
    value: float
```

This example defines a schema for transactions with attributes like block number, transaction hash, sender, receiver, and value.

Learn more about **Dataclass** in the UDF: [Components of UDFs](/udfs-user-defined-functions/components-of-udfs)

***

### **Create a Model**

A `model` encapsulates the processing logic to transform or compute derived values from the raw data.

```python
class TransactionModel:
    def __init__(self, data: TransactionData):
        self.data = data

    def compute_transaction_fee(self, gas_price: int, gas_used: int) -> float:
        """Calculate transaction fee."""
        return gas_price * gas_used / 1e9

    def to_dict(self):
        """Convert the data to a dictionary for database insertion."""
        return {
            "block_number": self.data.block_number,
            "transaction_hash": self.data.transaction_hash,
            "sender": self.data.sender,
            "receiver": self.data.receiver,
            "value": self.data.value,
        }
```

This `TransactionModel` processes the `TransactionData` and includes additional logic like calculating transaction fees.

Learn more about Model in the UDF section: [Components of UDFs](/udfs-user-defined-functions/components-of-udfs)

***

### **Write the Job Logic**

The `job logic` is where the UDF interacts with the Hemera Indexer to fetch and process blockchain data.

```python
import json
from hemera.udf.base import HemeraJob

class ProcessTransactionsJob(HemeraJob):
    def run(self, block_data):
        """Process each block's transactions."""
        processed_transactions = []
        for transaction in block_data["transactions"]:
            # Map raw transaction data to dataclass
            tx_data = TransactionData(
                block_number=block_data["block_number"],
                transaction_hash=transaction["hash"],
                sender=transaction["from"],
                receiver=transaction["to"],
                value=float(transaction["value"]) / 1e18,  # Convert wei to ETH
            )

            # Process data using model
            model = TransactionModel(tx_data)
            processed_transactions.append(model.to_dict())

        # Output data (to database, file, or another pipeline stage)
        self.output_results(processed_transactions)

    def output_results(self, transactions):
        """Mock function to send results to a database or log output."""
        print(json.dumps(transactions, indent=2))
```

Here’s what’s happening:

* The `run` method processes each block’s transactions.
* Raw blockchain data is mapped to the `TransactionData` dataclass.
* The `TransactionModel` performs transformations, and the results are formatted for output

Learn more about Job Logic: [Components of UDFs](/udfs-user-defined-functions/components-of-udfs)

***

## Running the Indexer with UDFs

Once your UDF is ready, it can be deployed and executed alongside the Hemera Indexer. The following example demonstrates how to run the indexer with a demo job:

**Command to Execute a Demo Job with UDF**

```bash
python3 hemera.py stream \
  --provider-uri https://ethereum.publicnode.com \
  --start-block 20804100 \
  --end-block 20804486 \
  --output-types demo_job \
  --block-batch-size 10000 \
  --postgres-url postgresql://uniswap:123asd@localhost:30432/uniswapv3 \
  --output postgres \
  --config-file config/indexer-config-template.yaml
```

**CLI Parameters:**

* `--provider-uri`: The Ethereum node URL used to fetch blockchain data.
* `--start-block` and `--end-block`: The block range for processing.
* `--output-types`: Specifies the job type, here `demo_job`, to indicate the UDF logic applied.
* `--block-batch-size`: Number of blocks processed in each batch for efficient indexing.
* `--postgres-url`: PostgreSQL connection URL to store processed data.
* `--output`: Specifies the output format, such as `postgres`.
* `--config-file`: Path to the Hemera configuration file.

Learn more about the CLI Parameters: [Configurations](/hemera-indexer/configurations)

***

**Leveraging SocialScan Developer APIs**

Hemera offers APIs to build advanced blockchain explorers with features like real-time transactions, token activity, and customizable wallet activity focus.

* **API Documentation**:

  Access detailed [API documentation](https://api-docs.thehemera.com/socialscan-explorer-api/introduction) to integrate these features into your applications.

#### **Example Applications**

Developers can utilize Hemera for various applications, including:

* **DeFi Analytics**:

  Track liquidity pools and yield metrics.
* **NFT Monitoring**:

  Visualize collections, sales, and holder activity.
* **Social Graphs**:

  Map on-chain interactions and relationships.


# Account Centric Indexing Protocol

Learn about the Account-Centric Indexing Protocol, a system designed to streamline data retrieval by focusing on account-specific information.

#### The Hemera Network

Explore The Hemera Network, a cutting-edge network designed for enhanced connectivity and efficiency.

#### Use Cases for Hemera

Discover various use cases for Hemera, such as:

* **Decentralized Applications:** Building apps with increased security and scalability.
* **Data Analytics:** Efficient data processing for better insights.
* **Blockchain Integration:** Seamless integration with existing blockchain systems.

{% content-ref url="/pages/Rp5r5AdSrdzW47T7IWOs" %}
[What is Account-Centric Indexing?](/welcome/account-centric-indexing-protocol/what-is-account-centric-indexing)
{% endcontent-ref %}

{% content-ref url="/pages/puOm8wwRNL0sXqg1YYdq" %}
[The Hemera Network](/welcome/account-centric-indexing-protocol/the-hemera-network)
{% endcontent-ref %}

{% content-ref url="/pages/msWqkjZdtjMwazFlEXPb" %}
[Example Hemera use cases](/welcome/account-centric-indexing-protocol/example-hemera-use-cases)
{% endcontent-ref %}


# What is Account-Centric Indexing?

### Background

Smart contracts on blockchains allow the freedom of creating arbitrary softwares, allowing end users to record data to blockchains. Such data include users’ financial activities including assets and trading actions, and users’ consumer behaviors including gaming activities, NFT collections & social networks etc. As GameFi, SocialFi, Crypto AI and other consumer-facing Dapps thrive, users’ consumer behaviors are expected to become more prevalent on-chain in future. While users’ data are valuable assets and crucial to any potential AI & ML use cases, crypto data's lacks essential semantics for both human-readable needs and AI use cases. The major challenges are listed below:

1. Data resides in silos across different Dapps & blockchains
2. Raw crypto data lacks essential semantics&#x20;

Here we propose Account-centric Indexing, a programmable semantic indexing protocol to tackle the above challenges.<br>

### Data Model

Account-centric Indexing mainly focuses on resolving accounts' on-chain profiles, and keeps the profiles updated in real-time. Here we model users’ on-chain profiles through Accounts & Features, each Account is described by a Feature set (see Figure1). On-chain transactions are programmed into Triggers, which in turn trigger Accounts’ Features to update via a functional interface defined by User-Defined Functions(UDFs).&#x20;

<figure><img src="https://lh7-us.googleusercontent.com/docsz/AD_4nXeMaosSDxO6zOR3JYkVmvXOsswhQbg9Y6c5qqxyVdQepJN6LCdlLcMDlcfPbh04pvjD2E57YI0X4h6eJBsh6aDiIf57T3MMjzcTNieCJPopyAH_xTm6y7d18RCjIokpkarWwm5CmQmwgHiD7ZvIftgNnOr9?key=WtjgM7yt84sY1v8s8nJy8w" alt=""><figcaption><p>Figure 1. Comparing EVM’s account data model with Account-Centric Indexing’s data model. EVM accounts contain 4 fields, the nonce, current ether balance, contract code (if any), account’s storage (empty by default). The left pink box shows examples of 4 different EVM accounts in light pink rectangles. The 1st and 4th accounts are externally owned accounts (EOA) and do not have contract code nor storage. The 2rd &#x26; 3rd accounts are smart contract accounts, as such both contain a code section, and a list storing contract related current values. The right gray box is the example of one Account object in Account-Centric Indexing. Account has an ID which is the account address, and a set of Features (white rectangles). Each Feature is recognized by a human-readable name, a unique ID, and a list of values and corresponding block number. </p></figcaption></figure>

<br>

<figure><img src="https://lh7-us.googleusercontent.com/docsz/AD_4nXfi7IGi9TLWiD9gcYhV7HV_RY4qTnRcKtix7pKy_hBZzAt6DySu9SV0E6t5LNMnqfgqwhSTM3AjDD7KHN0A0dC8ZG5rqmU_wIywS21afRlJzznA0Gu5-w94Qnpijaa7l58eRtW65XjBLuRciNBa8Al4JOg?key=WtjgM7yt84sY1v8s8nJy8w" alt=""><figcaption><p>Figure 2. Account-Centric Indexing data models in action. Each Account is described by a set of Features, which are updated through UDFs. In the figure, Account 0xA (gray box on the left) has 4 features (blue circles and white boxes on the left), each of which has a unique ID, a human-readable name, and a list of values. UDF (blue diamond)  is defined by Trigger, UDF code (gray box on the right) and listens (red arrow) to new events and updates the impacted feature.</p></figcaption></figure>

Each Account is described by a set of Features, some of which are constant attributes (e.g age of an account), and others are dynamically changing values (e.g. total trading volume). A Feature could be a user's ETH balance, liquidity in a pool, recent 30 days DEX transaction volume, NFT holdings, MEME trading P\&Ls, social followings & followers etc. Essentially, it refers to any semantic value describing a user’s on-chain activities. When developers utilize Account-Centric Indexing, they can access Accounts and their associated Features directly through interfaces provided by the Hemera Protocol Network.&#x20;

A Feature can be defined by any developers, and created and updated programmatically during the Account-centric Indexing process. Each feature has a unique identifier (ID), a human-readable feature name, and a feature type, i.e. a Feature named “USDT balance” is an “ERC20 token balance” typed Feature. For feature types already supported by the protocol, i.e.  “ERC20 token balance”, Account-Centric Indexer automatically creates new Features when needed and updates existing Feature values. For example,  “ERC20 token balance” is a built-in feature type, and when a new ERC20 smart contract with token ticker $ABC generates transactions, a new Feature instance named “abc\_balance” with type “ERC20 token balance” is created and updated for all Accounts who traded $ABC token.&#x20;

\ <br>

<figure><img src="https://lh7-us.googleusercontent.com/docsz/AD_4nXcajkYrMnMi_vsR_HtEDES9kGi6fuBDCFn4p2eLSQVZpD2TKb0f2vC8rSKvXJ0ztM5eZiJ9yL_oWpmxqn-0CYkltZZGD9XelAOxoiUzGx81pFYM4O6hnr-szLggGGxVOqV26RvaEBlvU9ZhimTXsG4t7Lb6?key=WtjgM7yt84sY1v8s8nJy8w" alt=""><figcaption><p>Figure 3 Account-centric Indexing updates Account Features when a new transaction occurs. A new transaction occurs from account 0xABC to swap 0.72 ether for 221M $PEPE on Uniswap (light pink box on the left). Account-Centric Indexer listens to the new transaction, and generates two triggers (blue rectangles with rounded corners) which in turn updates three different features for Account 0xABC (gray box on the right)</p></figcaption></figure>

Feature values are updated through UDFs (see Figure 3). A UDF consists of two elements: Triggers that listen to on-chain changes and the UDF code. A Trigger is essentially any action that causes on-chain state updates, which can be a transaction or an internal transaction. Developers can define Triggers through configuration files, and allow Account-centric indexers to create and manage Triggers. Each Trigger object is directly linked to one or more UDFs and when new Trigger instances are created, impacted UDFs keep Features up to date. UDFs take in standard data structures from EVM blockchains including transactions, logs, and traces, and in addition a list of developer-defined inputs allowing developers to express new Feature types when needed.&#x20;

Finally, each Account’s Feature values are updated and historical value changes are stored by the Hemera Protocol Network. Each feature value is tied to a block number, and a new entry will be created when the feature value of a later block number differs from that of the preceding block number. With the data model, users can easily get current feature values and compute historical changes.

<br>


# Why "account-centric" indexing?

Retrieving and computing semantics out of crypto data is repetitive, complicated and expensive. Hemera Protocol abstracts away the complexity and makes the task simple.

## Challenges

To have a full grasp of the complications behind programming for account-level semantic data, it is crucial to first dissect the blockchain data structures. Ethereum blockchain data can be categorized into two types of non-overlapping datasets: state, needed to build and validate new blocks; and history, necessary for syncing a node from Genesis to the latest block. State is composed of contract bytecode, contract storage, account balance (i.e. Ether balance) and account nonces. History is composed of blocks and transactions.&#x20;

From these definitions, when it comes to programming for account-level semantic data, i.e. assets tracking, social graph etc., developers need to perform at least the following three types of operations touching both states and history data:

1. Account to contract address mapping: for each account, iterate through all historical transactions and record the respective contract addresses it has transacted with, and keep this mapping updated in real-time. Since transactions are stored chronically, developers will have to first search for the raw data transaction by transaction using Remote Procedure Call (RPC) services, which is time consuming and expensive. This mapping enables operations in type 2 & 3 operations.
2. Retrieve asset balances: each asset’s (except for Ether) current balance is stored in contract storage. If asset balance history is needed, developers need to replay and store the balance change history by iterating through transactions. Further complications exist when considering the differences in assets types and respective implications.&#x20;
3. interpret transactions' semantic meaning: this task is most time consuming and repetitive. It requires developers to have access to the deployed smart contract source code as well as traces and logs in order to interpret the functionality of each smart contract. Developers can access smart contract source code via blockchain explorers, which host the compiled and verified smart contract code alongside the contract bytecode on-chain. Third-party developers would invest additional time and efforts to parse smart contracts, as they need to delve into trace and log data. One could imagine that blockchain explorers transform into interfaces where contract developers submit adapters to help third party developers interpret their contracts. This is crucial, as without this capability, basic metrics such as portfolio net worth or profit and loss (P\&L) may be inaccurately calculated.

In today’s modular blockchain ecosystem, with the emergence of more rollups, high-performance chains, and Danksharding to continuously scale the EVM ecosystem, account-level data becomes fragmented across multiple blockchains. As a result, Dapp developers often find themselves reinventing the wheel, compelled to build programmable indexers from scratch in-house.&#x20;

## What use cases is account-centric indexing addressing?

Developers who are building applications that need access to end users' on-chain activities need account-centric indexing. The use cases can be broadly categorized as follows:

1. Real-time (or as fast as blockchains' block time is) access to on-chain activities
2. Semantic understanding of user's on-chain activities

### Real-time access to on-chain activities

User-facing applications often build product features based on users' on-chain activities.

Wallet applications need to provide essential portfolio management features for better user experience; Web3 social applications rely on on-chain records to foster trust among users; Quest-based applications incentivize users' on-chain actions both in-app and outside of app for calculated rewards.&#x20;

### Semantic understanding of user's on-chain activities

Web3 user-facing builders often find themselves wondering about their community, figuring and rewarding the most loyal users, performing the airdrops properly etc. Yet none of this is possible without being able to extract semantics beyond raw on-chain transaction records.&#x20;


# The advantages of Account-Centric Indexing

Account-Centric Indexing is a novel indexing protocol tackling account-level semantic data computation and retrieval. Its User-Defined Function (UDF) design allows anyone to express semantic values through a simple functional interface.

Using Account-Centric Indexing, developers can access account state changes across smart contracts and blockchains in real-time. Developers can also configure to easily index entire blockchains, or designate the wallets-of-interests and index their transactions across all EVM chains.&#x20;

Unlike traditional 'contract-centric' indexing found in Subgraphs which mainly addresses in-Dapp data retrieval, Account-Centric Indexing focuses on resolving account profiles across Dapps and chains, and thus tackles the following needs directly:

1. Configurable indexing on accounts of interests
2. Multi-chain data coverage
3. Covers use-level specific use cases

Thus account-centric indexing provides the following main advantages.:

1. Real-time access to the account-level states and history
2. Full account-level states and history coverage, including all smart contracts & chains the account has footprints on.
3. Horizontally scalable

####


# What's next for account-centric indexing?

Account-centric indexing is highly modular, and evolves with on-chain diversity

As new smart contracts and data protocols emerge, account-centric indexing evolves accordingly, extending support to include new assets, social protocols, credentials, and other developments. We are committed to enhance account-centric indexing's capabilities from the following two dimensions:

1. **Modularity:** Standardizing and modularizing the adapter design provides a consistent and scalable approach to supporting new contracts. It serves as a venue for contract developers to publish their work to end users, who then have easy access to new contracts and protocols added and supported across the network.&#x20;
2. **Horizontally scalability**: our protocol network is highly composable, allowing for on-chain data indexing units to be as small as hosting a single account or as large as hosting an entire blockchain’s data. This design ensures horizontal scalability and decentralization without compromising performance. As an example, devices as small as personal cell phones hosting one account, effectively acting as a “personal data pack”, or as large as a distributed database cluster hosting full Ethereum mainnet data (currently \~10TB in total as of end of 2023), can be seamlessly supported to join the Hemera network.


# Why create a new protocol here?

**TL;DR:** A protocol brings the following fundamental benefits:

1. Bootstrap network effects to maximize innovation
2. Properly aligned interests
3. Significantly reduce development & service costs for all developers.

A protocol brings the most value when more developers build on top of it, creating a network effect which benefits all network participants.

From a technical perspective, when considering on-chain data retrieval and the use cases built on top of it, many technical components can be standardized due to the uniform formats of underlying on-chain data. Therefore, it makes the most sense to create a protocol network that addresses these demands which also evolves over time, benefiting the entire ecosystem. Traditionally, building decentralized indexing protocol often involves performance compromises. However, Hemera Protocol's  Account-Centric Indexing architecture makes it feasible to develop such a protocol without sacrificing performance.&#x20;

From a business perspective, opting to make Hemera an open protocol presents the most logical choice for several reasons:

1. Alignment of ecosystem interests: Hemera Protocol enables the support of valuable proprietary intellectual properties, mitigating the risks associated with Web2 models where enterprises monopolize these assets and profit solely from their corporate networks. This approach ensures that Hemera Protocol can maximize its value from inception。
2. Significant cost reduction for all protocol users:
   * **Reduce development costs**: standardize the account-level data retrieval and provide it as a protocol, removing the needs for Dapp developers to build everything from scratch in house.
   * **Reduce the service costs:** the protocol distributes the cost-intensive indexing workload across the Hemera Network, which encompasses participants such as miners, developers, and individual users, ensuring computational coverage across all geographical locations.


# The Hemera Network

Hemera Protocol builds a decentralized, omni-chain covering, modular indexing protocol powering open AI innovation.


# Network Roles

The Hemera Network mainly consists of Indexers, Validators, as well as proprietary models & knowledge owners, along with proprietary data owners and labellers.&#x20;

Indexers, Validators and Inference Nodes serve as the primary computing resource providers, offering crucial data and inference access to developers and servicing the network.&#x20;

Proprietary models & knowledge and proprietary data & labels are network artifacts contributed by AI scientists, domain experts, data owners and label providers. They empower the Hemera Network with AI & Machine Learning capabilities.&#x20;

Consumers encompass both end users, who utilize Dapps, and developers, who build them. Both groups consume the data and intelligence provided by Hemera.

End users engage directly with the Hemera Network via user-defined AI agents available in the SocialScan Agent Store. Through the use of $DAY tokens, users acquire on-chain intelligence from these AI agents. The Agent Store offers a toolkit that enables anyone to configure and personalize AI agents with their specific expertise in on-chain intelligence. This capability enables end users to meet the majority of their requirements for accessing on-chain intelligence.

Hemera Network provides performant on-chain data services that developers require to build a variety of applications, spanning from essential tools such as block explorers, oracles, and data APIs, to more advanced functionalities enabled by AI models.

To ensure the economic security of the Hemera Network, maintain the integrity of queried data and intelligence, and distribute rewards for network contributions, participants stake and utilize Hemera Tokens ($DAY). DAY serves as a utility token powering the network.

Active Indexers, Validators, Inference node clients, proprietary models & knowledge owners, and proprietary data owner & labeller can earn rewards from the network. See Figure below

<figure><img src="/files/HhYXA4A40wPlg0XzqHaB" alt=""><figcaption></figcaption></figure>


# Indexers

Indexers are the foundational node operators providing account-centric indexing capabilities to the network.  Indexers earn $DAY token rewards in the following ways:

1. **Data accessibility:** indexers keep the indexed data available to requests at all time, and as such earn rewards for maintaining accessibility given SLA requirements. To join the network as indexers, $DAY tokens are staked to secure the network, and subject to slashing if serving malicious or incorrect data to applications, or if SLA conditions aren't met.&#x20;
2. **Query rewards:** by serving queries requests from validators, indexers earn token rewards for the work.&#x20;
3. **Indexing bonus:** network consumers create demands for sought-after data sets, indexers earn one-time bonus for providing indexing services to meet such demands. &#x20;

Indexers play a crucial role in providing data accessibility within the account-centric indexing network. Their evolution occurs in three phases, with later stages offering more fine-grained indexing capabilities to address needs of network consumers.&#x20;

In the first phase of the Hemera network, indexers will assist in establishing data coverage for prominent chains that are in demand by consumers. They will be allocated indexing tasks based on geographically locations and network latency to ensure that Hemera can provide global network coverage and data accessibility.&#x20;

**In future phases of indexer releases, indexer operators can configure the node to:**&#x20;

1\. optimize indexing earnings by assigning tasks based on network demands;&#x20;

2\. configure wallets-of-interests and chains to cover, catering to Dapp developers who prioritize their own data needs over earnings.


# Validators

Validators are node operators responsible for securing the network.  They fulfill the following responsibilities:

1. Serves as the API gateway of the network: manages the full life cycle of data & intelligence requests to the Hemera Network
2. Validates network responses: validate the responses provided by indexer and inference nodes
3. Validates the availabilities of indexers and inference nodes: validators maintain a lookup table called the "Address-Name-Service" (ANS) for locating the accounts and indexers who hold the accounts data. As such, validators periodically check and verify the availability of the indexed data. See Figure below

<figure><img src="/files/PQTGR6ftKGndbL6KisRX" alt=""><figcaption><p>Figure  Validator nodes (API Gateway nodes) hosting Address-Name-Service (ANS), a distributed lookup table maintained to locate the respective indexer nodes holding states for individual account addresses. The key information of ANS is the mapping of account addresses to indexer nodes’ network details. The API gateway nodes can also act as validator nodes to validate the indexer nodes and create Proof-of-Indexing when needed.</p></figcaption></figure>


# Proprietary models & knowledge

AI scientists and domain experts can contribute to Hemera Network through attributable proprietary models and knowledges

Proprietary models & knowledge are artifacts that generate on-chain intelligence for the Hemera network consumers. They are intellectual properties (IPs) in the form of machine learning algorithms, AI models, heuristic rules, or domain expertise in general that serves a certain intelligence needs. These IPs are closed source, encrypted, and operated within a trusted execution environment (TEE). Ownership is attributed to contributors' designated wallet addresses for collecting token rewards. Contributors earn rewards in the following form:

1. One-time bounty rewards: for each interested problem or issues raised by the Hemera community, a one-time incentive in the form of $DAY tokens is provided to incentivize the research and development of proprietary models and knowledge. Initially, incentives are likely to be provided from the Hemera treasury in a form similar to ecosystem grants. As the network matures, incentives may also come from the wider Web3 community. This could involve community members proposing interesting problems to be solved by the Hemera community, with rewards offered for successfully addressing these challenges.&#x20;
2. Continued usage revenue share: each artifact on Hemera protocol are paid to use, as such accumulating revenues in the form of $DAY tokens. Owners of the artifacts earn a predefined percentage of the tokens.

See below artifacts examples built on top of the Hemera protocol.&#x20;

#### **User-defined AI agents:**&#x20;

Using tools provided by SocialScan Agent Store, individual traders can define and customize agents that showcase their proprietary asset tracking and trading strategies. End users can pay $DAY tokens to access these agents. The agent creator can earn the $DAY tokens paid by users, with a small hairbut charged by the Hemera treasury to compensate indexers and validators for providing data access.


# Proprietary data and labels

Data is the fuel for AI, and data residing outside of specific AI use cases are of much less value. The demand for proprietary data and labels usually arises from the needs to develop AI & ML models. This often involves the requirement for off-chain proprietary data & labels in addition to on-chain data.

In model training scenarios, web3 end users are incentivized to contribute the requested data in a secure, verifiable and attributable manner. They earn both one-time incentives and continued reward shares when the developed model is utilized by consumers.&#x20;

After models are deployed in production, users' usage data can be securely recorded to aid in model improvement, thereby attributing thier contributions to the enhancement of the models.

To achieve this goal, Hemera Network can integrate with 3rd party protocols, such as zero-knowledge proofs for data privacy and verification, incentive attribution protocols for tracking rewards, and and proprietary data and labels residing on Hemera Network as special artifacts providing value to consumers. &#x20;


# Smart Contracts

### Key Components of the Hemera Protocol

#### Indexer registry

Indexers stake and register to the network. After registration, validators will distribute queries to the indexer. Indexers should also submit proof of indexing records periodically and broadcast them to the whole network.&#x20;

Indexers capable of real-time indexing should set a flag in their metadata. The flag status is broadcasted to indicate real-time indexing capability and should be updated regularly.

#### Query book

Validators upload all the queries from end users. Validators submit records of any indexer failures, including inactive indexers and incorrect query results.

The query book should run settlements regularly. Following settlement, the balances for indexers and end users should be updated accordingly.

#### User account

End users must deposit a specified amount of $DAY tokens as a stake. A key will be generated for the user. Users should be able to check overall usage and manage account balances (deposit or withdraw $DAY tokens).

### Key Functions for Protocol Participants

#### Indexers&#x20;

* Stake $DAY token to the smart contract and register to the network
* Submit "proof of indexing" to the smart contract
* Broadcast real-time indexing status

#### Validators

* Check the indexing status through the smart contract
* Record usage history by batch (usage from end users & query served by indexers)
* Submit indexer failure (including inactive indexer and wrong query result) records

#### Administrator

* &#x20;Triggers settlement for indexer/validator/end user

#### End users/developers&#x20;

* Stake $DAY tokens and get valid keys to access the data API


# Key roadmap items

Key timeline:

Testnet --- June, 2024

Validators and indexers available for decentralized Hemera Network bootrapping.&#x20;

Mainnet --- Aug/Sep. 2024

Hemera Network production ready, supporting application layer, e.g. blockchain explorers, AI agents etc. in production.&#x20;

Key technical items:

1. Customizable account-centric indexing
2. History expiry optimization
3. Next-gen indexing architecture
4. Network demand auto balancing
5. Web3 fine tuned LLM
6. Configurable on-chain AI agents


# Supported blockchains

As of May 2024, the following blockchains are supported:

Ethereum mainnet, Optimism, Arbitrum, Taiko, Blast, Base, Avalanche, Polygon PoS, Polygon zkEVM, Linea, Mantle, Manta-pacific, Cyber, Cyber-testnet, Story-testnet, Mint-testnet,  GM-testnet, Merlin, Taiko-testnet, Berachain-testnet (coming soon), Celestia DA, EigenDA (coming soon)


# Example Hemera use cases


# SocialScan Explorers

Blockchain explorer is an essential developer interface for every blockchain. To support a high performance blockchain explorer, a highly optimized indexing backend is needed to index on-chain transactions in near real-time, and service in particular account-level SQL queries efficiently. Also the indexing requires the coverage of full state and transaction history, along with essential data including trace, log, calldata (blob) etc. for each blockchain. Supporting blockchain explorer use case allows Hemera Network to: 1. provide 100% on-chain data coverage; 2. optimize its network performance.

On the other hand, the blockchain explorer space has been dominated by industry monopoly who charges as much as seven figures in US dollars annually (a whopping gross margin of 99% and higher!), hindering modular ecosystem development and further scaling solutions in the EVM space.&#x20;

Thus SocialScan Explorers, built on top of Hemera Protocol, provides the perfect showcase application  to bootstrap Hemera Network, all while resolving a major industry pain point.

To access examples of SocialScan explorers, see URLs below:

Manta Pacific: <https://manta.socialscan.io/>

Linea: <https://linea.socialscan.io/>

Taiko: <https://taikoscan.net/>

If you're a blockchain network interested in having your own SocialScan Explorer, reach out to us at <support@thehemera.com>&#x20;

<br>


# Anti-sybil UML algorithm

Existing Machine Learning and AI algorithms can potentially be readapted to tackle similar issues in web3, leveraging on-chain data instead of web2 inputs.&#x20;

Here we were able to develop a showcase Proprietary algorithm which was battle hardened in web2 anti-sybil use cases, and readapted for web3 use cases leverage account-level on-chain data provided by Hemera Network.&#x20;

For more details of the approach, see the blog post here:

<https://medium.com/hemera-protocol/an-airdrop-cookbook-by-hemera-and-tokentable-ed42127d9004>


# Ethereum long term DA

The introduction of blobs in the [Dencun](https://blog.ethereum.org/2024/02/27/dencun-mainnet-announcement) hard fork has allowed L2s to post their transaction data using “data blobs”, significantly [reducing gas fees.](https://medium.com/hemera-protocol/post-eip-4844-the-primary-factors-impacting-gas-fees-on-layer-2-blockchains-df64799d28ec) However, blobs are only stored by Ethereum nodes for \~18 days before being expired and deleted to save on data storage costs. Yet, while blobs are not needed to reconstruct Ethereum mainnet transactions, they are necessary for each L2 to reconstruct their own history (e.g. for verification purposes).

Any L2s using the Ethereum mainnet as their [Data Availability layer](https://l2beat.com/scaling/data-availability) need transaction data to be available when a new full node is set up and joins the L2 network without P2P sync. Existing L2s mainly sync transaction history from the L2 network itself, which has centralization risks (e.g. in the rare case that peer nodes are corrupted and transaction history is lost).

Thus, the Ethereum ecosystem needs an alternative form of blob preservation to ensure L2s have access to blob data whenever needed, in turn keeping the Data Availability guarantee. As discussed in [Paradigm’s recent blog](https://www.paradigm.xyz/2024/05/how-to-raise-the-gas-limit-2) , a robust blob storage infrastructure will also apply to L1 history data storage, providing a viable solution to the question raised by [EIP-4444](https://eips.ethereum.org/EIPS/eip-4444) in the future.

Since Hemera Network continously indexes and stores on-chain data, including calldata & blobs, providing a BlobArchive service is a natural use case.&#x20;

**Checkout our blog here for more details:** <https://medium.com/hemera-protocol/hemera-blobarchive-for-ethereum-permanent-data-availability-for-layer-2-11c289b20d45>


# EVM chain history preservation

{% hint style="warning" %}
Note: This is a roadmap item that is not currently available. We are documenting it here to give developers an idea of potential use cases that can be supported by Hemera Network.
{% endhint %}

Ethereum's history growth is gradually increasing the hardware requirements for running full nodes. For long, Ethereum community has been working on [EIP-4444](https://eips.ethereum.org/EIPS/eip-4444), which requires a solution for history preservation of Ethereum's full nodes. Similar needs will also arise, likely a lot fast, in high performance EVM scenarios, where history grows at 100 \~ 10,000X faster speed.&#x20;

As such, an EVM history preservation service designed in a similar architecture as in[ ](/welcome/account-centric-indexing-protocol/example-hemera-use-cases/ethereum-long-term-da)[the previous section](/welcome/account-centric-indexing-protocol/example-hemera-use-cases/ethereum-long-term-da) can be expanded to meet such needs.&#x20;


# Ecosystem AI Agents

Our vision for future consumer-facing web3 UX

Hemera Protocol adds a web3-optimized Large Language Model (LLM) as one of the query engine options, allowing users to query on-chain data using natural language. Hemera LLM evolves its capabilities by optimizing the latest AI innovations and building on top of the web2 foundational model. The LLM capability, coupled with the account-centric indexing layer, empowers third party developers from chains, protocols, and Dapps to easily define and configure AI agents tailored to their specific needs.&#x20;

As a showcase of Hemera Protocol's user-defined AI agents capabilities, a series of ecosystem AI agents are developed, allowing end users to discover the dynamic activities within each blockchain ecosystem using AI. The AI agents serve as personal junior crypto analysts to help end users monitor and discover assets and communities of interests.&#x20;

For example ecosystem agents on Linea, Taiko (coming soon), BeraChain (coming soon), ZetaChain (coming soon), head to <https://socialscan.io/>&#x20;

<br>


# User-defined AI Agents

{% hint style="warning" %}
Note: This is a roadmap item that is not yet available but under active development.
{% endhint %}

As a public goods protocol, Hemera's true strength lies in standardizing data access and Web3 LLM capabilities and providing them to empower the web3 ecosystem. Here user-defined AI agents here allows anyone to configure and fine tune agents according to their own domain knowledge, which can be insights on personal MEME coin trading strategy, or heuristics on identifying new smart money addresses, or personal favorite NFT assets. Once configured, the user-defined AI agents are attributed to their creators, and when used, creators can earn a percentage of the token spent. &#x20;


# Smart Contract Developers

Smart contract developers often need to analyze their data. Assuming you are a developer from EigenLayer, and your community is requesting a comprehensive dashboard on how each wallet addresses stake with EigenLayer, here are the steps you can follow to accomplish this using the Hemera Protocol.

#### Create your data model

To store the result, we can design a simple data model

<pre><code>class DailyWalletTokenStaking(BaseModel):
<strong>    date = Column(Date)
</strong>    wallet_address = Column(String)
    token_address = Column(String)
    balance = Column(Numeric)
</code></pre>

#### Understand your smart contract data

You must include all the transactions/logs/traces, which may change your output data. For your use case, you need to identify all the staking balance changes.&#x20;

If you understand your contract well, you should figure out that this transaction will trigger a user-staking balance change.&#x20;

Method: depositIntoStrategy: <https://etherscan.io/tx/0x7269c2cf90779d82b3b53f6c997cabd25243cb60f6f136e8ce5a1e6d510d478e>

#### Design trigger event

To summarize, whenever you see a transaction that interacts with this specific address (EigenLayer: Strategy Manager) and calls this specific function (depositIntoStrategy) from your smart contract, you should update the user staking balance.&#x20;

You can define this customized trigger within the indexer using this code.

<pre><code><strong># 0xe7a050aa is refering to the depositIntoStrategy
</strong>0x858646372CC42E1A627fcE94aa7A7033e7CF075A
(EigenLayer: Strategy Manager)
<strong>if transaction.to_address == "" and transaction.method_id == "0xe7a050aa":
</strong><strong>    # call update_daily_wallet_token_staking
</strong><strong>    update_daily_wallet_token_staking(transaction, logs, traces)
</strong>
</code></pre>

The trigger can be a method in transactions,  topics from logs, or even an action from the trace.

#### Write customize code

Now we can add our code and update our data model when an event is triggered. The event will pass transaction data/logs/traces to you.

You can write your customized code and update the data model you design.

```python
def update_daily_wallet_token_staking(
    transaction: Transaction, logs: Log[], traces: Trace[]
    ):
    # Get balance change from logs
    
    # Update model
    
    
```

Learn more about User Defined Functions: [Building User Defined Functions(UDF)](/udfs-user-defined-functions/building-user-defined-functions-udf)


# EVM-compatible chains

Hemera Protocol provides extensive supports for all EVM-compatible chains.


# Blockchain explorers

SocialScan explorer is a high-performance blockchain explorer built for the modular ecosystem.

Currently we have hosted solution available for any interested EVM-compatible chains.&#x20;

To access examples of SocialScan explorers, check out the explorer section:&#x20;

{% content-ref url="/pages/cvpy6outBy9VznYofPC2" %}
[Hemera Powered Explorers](/about-us/hemera-powered-explorers)
{% endcontent-ref %}

In the meantime, if you are interested in our hosted explorer solution, you can conveniently do so by [visiting our Calendly link](https://calendly.com/socialscan/intro?month=2024-02\&back=1). If you prefer written communication or have detailed queries, we welcome you to reach out to us via email at `contact@thehemera.com`


# Ecosystem AI Agents

Hemera's Ecosystem AI Agents are agents that interact with end users through LLM, allowing them to discover the latest trends within each ecosystem with Hemera's real-time on-chain intelligence.

To create an Ecosystem AI agent on Hemera, creators can follow these steps:

* Select a chain that is currently supported by Hemera.
* Submit project deployed addresses within the ecosystem.
* Submit contract addresses of launched assets.

Access the agent on <https://socialscan.io>


# Dapp developers

Decentralized application (dApp) developers rely on data pipelines to build applications. The **Hemera Indexer** is designed to streamline this process by enabling parallel execution of **User-Defined Functions (UDFs)** alongside the core indexing tasks.

By running the Hemera Indexer, developers can utilize the power of ACI (Account-Centric Indexing) to efficiently organize blockchain data by deploying custom UDFs to extract, transform, and analyze data tailored to their specific use cases.&#x20;

This parallel architecture not only optimizes performance but also simplifies complex data workflows, empowering developers to focus on building next-generation Web3 applications.

***

<details>

<summary>Core Concepts</summary>

#### [**Account-Centric Indexing**](broken://pages/Rp0V0xjhXC1ruxBTvblx)

#### [**User-Defined Functions (UDFs)**](/udfs-user-defined-functions/introduction)

</details>

***

## **Getting Started**

Refer to the [Quick Start guide ](/welcome/quick-start)to explore on getting started with Hemera Indexer.&#x20;

***

#### **SocialScan Developer APIs**

Leverage APIs to build advanced blockchain explorers:

* Real-time transactions.
* Token activity.
* Customizable wallet activity focus.

[📖 **Details**: SocialScan API Documentation](https://api-docs.thehemera.com/socialscan-explorer-api/introduction)

***

### **Example Applications**

* **DeFi Analytics**: Track liquidity pools and yield metrics.
* **NFT Monitoring**: Visualize collections, sales, and holder activity.
* **Social Graphs**: Map on-chain interactions and relationships.

📖 **Explore Use Cases**: Example Applications

***

### **Community Collaboration**

Join Hemera’s developer ecosystem to:

* **Hackathons**: Innovate with the community.
* **Open Source**: Contribute to infrastructure and documentation.
* **Support**: Engage on Discord.

***

### **Roadmap and Enhancements**

Stay ahead with upcoming features:

* Configurable on-chain AI agents.
* Web3-optimized LLM integrations.
* Mainnet launch (Q3 2024).

📖 **Roadmap**: Future Developments


# User-defined Agent creators

This application is under active development. We expect to have our early access testnet launch in Q3, 2024.  In the meantime, you can read about the application design below.&#x20;

To stay updated for early access, please follow our official twitter account for the latest announcements:

Hemera Protocol: <https://x.com/HemeraProtocol>

1. Creating an AI Agent

To create a user-defined AI agent on Hemera, creators can follow these steps:

* Accessing the Platform: Creators can sign up for an account on Hemera and log in to access the platform.
* Setting up a New AI Agent Project: Upon logging in, creators can initiate a new AI agent project and define its objectives, parameters, and customization options.
* Defining Objectives and Parameters: Creators specify the tasks and functions their AI agent will perform, along with any configurable parameters and settings.

2. Customization Interface

Hemera provides an intuitive customization interface for creators to tailor their AI agents:

* Drag-and-Drop Interface: Creators can use a user-friendly drag-and-drop interface to configure the behavior and logic of their AI agents.
* Configuration Wizards:

3. Data Indexing

Creators can seamlessly integrate data sources into their AI agents on Hemera:

* Connecting to Hemera Indexing Network: Hemera supports agent integration with our open indexing protocol, allowing creators to access blockchain data directly within their AI agents.
* Importing External Data? Creators can import data from external sources, such as APIs, databases, or Web3 data feeds, to enrich the capabilities of their AI agents.

4. Algorithm Selection

Hemera offers a range of options for selecting and implementing algorithms within AI agents:

* Pre-built Algorithm Libraries: Creators can choose from a library of pre-built algorithms optimized for common tasks and use cases.
* Custom Algorithm Development: For advanced users, Hemera provides tools for developing custom algorithms tailored to specific requirements and objectives.

5. Testing and Validation

Creators can test and validate their AI agents on Hemera to ensure optimal performance:

* Simulation and Evaluation: Creators can simulate different scenarios and evaluate the performance of their AI agents using predefined metrics and benchmarks.
* Iterative Improvement? Hemera supports an iterative development process, allowing creators to iterate on their designs based on testing and validation results.


# AVS Operator

This documentation gives an in-depth guide for setting up, registering, configuring, and running an AVS (Actively Validating Services) Operator in the Hemera ecosystem. \
\
As an AVS Operator, your role is central to maintaining data transparency and accuracy across the Hemera Protocol. By validating and aggregating on-chain data, the AVS Operator ensures that all Hemera network participants have access to reliable, verifiable information. This essentially involves validating blocks and transactions, comparing data across multiple sources, and contributing to Hemera’s unique history transparency.&#x20;

<figure><img src="/files/f14o8DZh4uPKtnNwPRwd" alt=""><figcaption></figcaption></figure>

AVS Operators are critical for Hemera’s **decentralized data verification structure** which brings enhanced trust and security to the platform’s users and data-driven applications.

***

## Step-by-Step Guide to Setting Up an AVS Operator

### **1. Key Generation and Wallet Funding**

Before you can register as an AVS Operator, you need to set up your keys and wallet.

1. **Install EigenLayer CLI:** Follow [EigenLayer’s Installation Guide](https://docs.eigenlayer.xyz/docs/getting-started/installation) to set up the EigenLayer CLI on your machine.
2. **Generate Keys:** Create both ECDSA and BLS keypairs using the following commands:

   ```shell
   shellCopy codeeigenlayer operator keys create --key-type ecdsa [keyname]
   eigenlayer operator keys create --key-type bls [keyname]
   ```

   \
   This will create an ECDSA and BLS keypair for secure, verifiable operations on EigenLayer and Hemera. **Remember to back up your private keys** securely. The encrypted keys are stored by default at `~/.eigenlayer/operator_keys/`.

### **2. Registering as an Operator on EigenLayer**

{% hint style="info" %}
**Note:** Skip this section if you are already a registered operator on the EigenLayer testnet and mainnet. Registration is required separately for testnet and mainnet.
{% endhint %}

1. **Create Configuration Files:** Use the command below to generate the necessary operator registration files (`operator.yaml` and `metadata.json`).

   ```shell
   shellCopy codeeigenlayer operator config create
   ```

   Follow the prompts to complete the configuration setup.<br>
2. **Edit `metadata.json`:** Fill in your operator’s details. This metadata will be publicly accessible and includes your operator name, website, description, and social links.

```json
jsonCopy code{
  "name": "Example Operator",
  "website": "https://example.com/",
  "description": "Example description",
  "logo": "https://example.com/logo.png",
  "twitter": "https://twitter.com/example"
}
```

1. **Upload `metadata.json` to a Public URL:** You can use services like [GitHub Gist](https://gist.github.com/) to host the metadata file. Once hosted, update the `operator.yaml` file with this public `metadata_url`.<br>
2. **Register Operator:** To finalize registration, run:

   ```shell
   shellCopy codeeigenlayer operator register operator.yaml
   ```

   A success message will confirm the operator registration: `✅ Operator is registered successfully to EigenLayer`.<br>
3. **Check Registration Status:** Verify the registration by checking the EigenLayer operator page or using the CLI command below:

   ```shell
   shellCopy codeeigenlayer operator status operator.yaml
   ```

### 3. Joining Hemera AVS

After registering on EigenLayer, the next step is to integrate your operator with Hemera AVS:

1. **Whitelist Registration:** Contact Hemera support to add your operator address to the AVS whitelist.
2. **Run Operator Registration Script:** Follow [alt-research’s guide](https://github.com/alt-research/mach-avs/blob/m2-dev/scripts/README.md) to register using `register_operator.sh`.

### 4. Configuring the Operator for Hemera AVS

Edit the `indexer-config-avs-holesky.yaml` file, located in the configuration directory, to include the necessary details for running as an AVS operator:

* Update the following items in the configuration file:
  * `operator_ecdsa_key_file`: Path to your ECDSA key file.
  * `operator_bls_key_file`: Path to your BLS key file.

These keys are required to sign and validate data with the Hemera AVS protocol.

### 5. Running the AVS Operator

You can run the AVS Operator either directly from the source code or by using Docker.

#### **Running from Source Code**

1. Navigate to the Hemera indexer directory. \
   \
   Checkout this guide on how to setup hemera indexer: [Installation](/hemera-indexer/installation)
2. Run the following command to start the AVS Operator, specifying the configuration file and database output details:

   ```shell
   shellCopy codecd hemera_indexer
   python3 hemera.py stream --provider-uri https://holesky.drpc.org --debug-provider-uri https://holesky.drpc.org -O hemera_history_transparency --output postgres --postgres-url postgresql://postgres:123456@localhost:5432/hemera_indexer --config-file ./config/indexer-config-avs-holesky.yaml
   ```

This command initializes the operator, establishing connections to RPC and data validation sources.

#### **Running with Docker**

1. Navigate to the Docker Compose directory and update environment variables in `avs.env` to suit your setup.
2. Start the AVS Operator in Docker by running:

   ```shell
   shellCopy codecd hemera_indexer
   cd docker-compose
   sudo docker compose -f avs-operator.yaml up
   ```

Docker is ideal for streamlined deployments and scalable operation setups.

***

<details>

<summary><strong>Operator Code Walkthrough</strong></summary>

Link to the code: <https://github.com/HemeraProtocol/hemera-indexer/pull/212/files#diff-3527683c538c929e553ef048358b8de50f5cecd7144a62d38e890de0f7573d4d>

The `operator.py` script facilitates data validation, transaction filtering, and event aggregation. Below is an outline of its primary components:

**Key Functionalities**

1. **Data Collection and Validation (`_verify_logs`):**
   * The operator collects transaction logs from specified blocks, verifies data using both API and RPC sources, and checks block hashes and transaction counts for consistency.
   * This process ensures that transaction data aligns with Hemera’s transparency requirements, which is critical for maintaining accurate history records.
2. **Operator Registration Verification (`_check_operator_is_registered`):**
   * Before starting, the operator checks if it is registered on EigenLayer by querying the registry coordinator contract.
   * The function returns `True` if the operator is registered, preventing unregistered operators from interacting with the network.
3. **Aggregation and Alert Creation (`_confirm_task`):**
   * Upon log verification, the operator aggregates the data and creates an alert task through the aggregator client, which assists in real-time monitoring and alerting.
4. **Configuration Handling:**
   * Configuration details for the RPC and API endpoints are managed through the YAML file, allowing flexible setup adjustments.

</details>

***

<details>

<summary>Troubleshooting</summary>

This section addresses common issues you may encounter when setting up or running the AVS Operator, with solutions and debugging tips to help you resolve them quickly.

1. **Operator Registration Failed**
   * **Issue:** Registration attempts result in an error.
   * **Solution:** Double-check the configuration files (`operator.yaml` and `metadata.json`). Ensure all required fields are filled in, especially the `metadata_url` field pointing to a publicly accessible location.
   * **Tip:** Use the `eigenlayer operator status operator.yaml` command to verify registration status.
2. **Key File Access Error**
   * **Issue:** The operator is unable to access the ECDSA or BLS key files.
   * **Solution:** Confirm that the key files are located at the paths specified in `indexer-config-avs-holesky.yaml` (`operator_ecdsa_key_file` and `operator_bls_key_file`). Check file permissions to ensure they are readable by the operator service.
3. **Block/Transaction Count Mismatch**
   * **Issue:** Block or transaction counts differ between the API and RPC sources.
   * **Solution:** Verify the synchronization of your RPC provider. If issues persist, consult the RPC provider or consider using an alternative to ensure data consistency.
4. **Docker Container Not Starting**
   * **Issue:** The AVS Operator Docker container fails to start.
   * **Solution:** Check the Docker Compose file `avs-operator.yaml` for any misconfiguration. Review `avs.env` to confirm that all environment variables are correctly set. Run `docker logs <container_id>` to get more details on startup errors.
5. **Aggregated Data Not Appearing in Database**
   * **Issue:** Data does not populate in the PostgreSQL database.
   * **Solution:** Ensure that the `postgres-url` provided in the run command is accurate and that the database service is running. Double-check network connectivity and the PostgreSQL configuration.

</details>

***

## Frequently Asked Questions (FAQs)

1. **What is the role of an AVS Operator in Hemera?**\
   An AVS Operator in Hemera is responsible for validating and aggregating on-chain data, ensuring transparency and accuracy. This helps maintain a reliable data layer that Hemera and its users can trust for decentralized applications.
2. **How do I update the operator metadata after registration?**\
   Update the `metadata.json` file with the new information, host it publicly, and edit the `operator.yaml` file to reflect the updated `metadata_url`. Run the following command to apply the update:

   ```shell
   eigenlayer operator update operator.yaml
   ```
3. **Can I run the AVS Operator on multiple machines?**\
   Yes, you can deploy the AVS Operator on multiple instances to improve redundancy and handle higher data volumes. Make sure each instance is configured with unique credentials and environment variables.
4. **How can I verify that my AVS Operator is running correctly?**\
   Use the `eigenlayer operator status operator.yaml` command to check the operator's registration status. Additionally, monitor logs for any errors and confirm that data is being populated in the configured database.
5. **What happens if my keys are compromised?**\
   In case of a security breach, generate new keys immediately using the EigenLayer CLI, update your configurations, and re-register if necessary. Secure backups of all private keys are strongly recommended.

***

## Contact and Support

For additional support, join our Discord community or contact us via email:

* **Discord:** [Join Hemera’s Community](https://discord.com/invite/hemera)
* **Email:** <contact@thehemera.com>


# Introduction

### About Hemera Indexer

As the foundation of the Hemera Protocol, the blockchain indexer plays a crucial role. It is the primary component that enables efficient and organized access to blockchain data.

Initially inspired by open-source projects like Ethereum ETL, we expanded its capabilities as the Ethereum ecosystem evolved, with the emergence of more Layer 2 chains and new ERC standards. Recognizing the need for a robust solution, we decided to develop our own indexer as the first step in building the Hemera Protocol Network.

As of July 5, 2024, the initial open-source version of the Hemera Indexer offers comprehensive functionality, allowing for the indexing of any EVM-compatible chains and providing all necessary data for a basic blockchain explorer. In the coming weeks, we plan to incorporate additional features from our in-house version into the open-source version.

### Features Offered

**Export the following entities**

* Blocks
* Transactions
* Logs
* ERC20 / ERC721 / ERC1155 tokens
* ERC20 / ERC721 / ERC1155 Token transfers
* ERC20 / ERC721 / ERC1155 Token balance & holders
* Contracts
* Traces / Internal transactions
* L1 -> L2 Transactions (Coming Soon)
* L2 -> L1 Transactions (Coming Soon)
* Rollup Batches (Coming Soon)
* DA Transactions (Coming Soon)
* User Operations (Coming Soon)

**Into the following formats**

* Postgresql SQL
* JSONL
* CSV

**Additional features**

* Ability to select arbitrary block ranges for more flexible data indexing
* Option to choose any entities for targeted data extraction
* Automated reorg detection process to ensure data consistency and integrity


# Installation

To run the Hemera Indexer, you need two things

* VM Instance (or your laptop/desktop)
* RPC Node of your EVM-compatible blockchain

For more information, check this page

{% content-ref url="/pages/IFDBCqDmKmyANqmMPAQv" %}
[Prerequisites](/hemera-indexer/installation/prerequisites)
{% endcontent-ref %}

Then, you can check out our Github repo <https://github.com/HemeraProtocol/hemera-indexer> and start to run. You can either run from docker or build from source. Check this page for more detail

{% content-ref url="/pages/vP5abnJgbLig0vkC1CGL" %}
[Install & Run](/hemera-indexer/installation/install-and-run)
{% endcontent-ref %}


# Prerequisites

### VM or Your Local Machine

To run the indexer, you need a VM environment or a local machine where you can install `python` and `postgresql`. Ensure you have sufficient disk space and computational resources, especially when indexing a large chain with extensive historical data.

### Python 3.9

Python 3.9 is supported and required to run the Hemera Indexer for the latest version. We will update this section when we support a higher Python version.

### RPC Endpoints

To get started, consider using free RPC providers. For large requests or when indexing historical data, a dedicated RPC URL is recommended. An indexer will require many RPC requests to index an entire blockchain; therefore, ensure your RPC endpoint is robust and fast. Often, the RPC endpoint is the indexer's bottleneck.


# Install & Run

Follow these steps to get started and run the Hemera Indexer:&#x20;

{% embed url="<https://youtu.be/Mv7HEBIsURY>" %}

## Clone the Repository

```
git clone https://github.com/HemeraProtocol/hemera-indexer.git
```

## Building and Running the Hemera Indexer

To operate the Hemera Indexer, you essentially have two approaches to choose from, each suited to different levels of customization and control.

**1.  Use Docker**

**2.Build from Source code**

## Run from Docker

#### **1. Install Docker & Docker Compose**

If you have trouble running the following commands, consider referring to the [official docker installation guide](https://docs.docker.com/engine/install/ubuntu/#install-using-the-repository) for the latest instructions.

```
# Add Docker's official GPG key:
sudo apt-get update
sudo apt-get install ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc

# Add the repository to Apt sources:
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
  $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
```

```
# Install docker and docker compose
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
```

#### **2. Run Docker Compose**

```
cd hemera_indexer
cd docker-compose
```

Alternatively, you might want to edit environment variables in `docker-compose.yaml`. Please check out the [configuration manual](/hemera-indexer/configurations) on how to configure the environment variables.

```
vim docker-compose.yaml
```

Now, run the following command to spin up the containers.

```
sudo docker compose up
```

You should be able to see similar logs from your console that indicate Hemera Indexer is running properly.

```
[+] Running 2/0
 ✔ Container postgresql  Created                                                                                                                0.0s
 ✔ Container hemera      Created                                                                                                                0.0s
Attaching to hemera, postgresql
postgresql  |
postgresql  | PostgreSQL Database directory appears to contain a database; Skipping initialization
postgresql  |
postgresql  | 2024-06-24 08:18:48.547 UTC [1] LOG:  starting PostgreSQL 15.7 (Debian 15.7-1.pgdg120+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit
postgresql  | 2024-06-24 08:18:48.548 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
postgresql  | 2024-06-24 08:18:48.549 UTC [1] LOG:  listening on IPv6 address "::", port 5432
postgresql  | 2024-06-24 08:18:48.554 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
postgresql  | 2024-06-24 08:18:48.564 UTC [27] LOG:  database system was shut down at 2024-06-24 06:44:23 UTC
postgresql  | 2024-06-24 08:18:48.575 UTC [1] LOG:  database system is ready to accept connections
hemera      | 2024-06-24 08:18:54,953 - root [INFO] - Using provider https://eth.llamarpc.com
hemera      | 2024-06-24 08:18:54,953 - root [INFO] - Using debug provider https://eth.llamarpc.com
hemera      | 2024-06-24 08:18:55,229 - alembic.runtime.migration [INFO] - Context impl PostgresqlImpl.
hemera      | 2024-06-24 08:18:55,230 - alembic.runtime.migration [INFO] - Will assume transactional DDL.
hemera      | 2024-06-24 08:18:55,278 - alembic.runtime.migration [INFO] - Context impl PostgresqlImpl.
hemera      | 2024-06-24 08:18:55,278 - alembic.runtime.migration [INFO] - Will assume transactional DDL.
hemera      | 2024-06-24 08:18:56,169 - root [INFO] - Current block 20160303, target block 20137200, last synced block 20137199, blocks to sync 1
hemera      | 2024-06-24 08:18:56,170 - ProgressLogger [INFO] - Started work. Items to process: 1.
hemera      | 2024-06-24 08:18:57,505 - ProgressLogger [INFO] - 1 items processed. Progress is 100%.
hemera      | 2024-06-24 08:18:57,506 - ProgressLogger [INFO] - Finished work. Total items processed: 1. Took 0:00:01.336310.
hemera      | 2024-06-24 08:18:57,529 - exporters.postgres_item_exporter [INFO] - Exporting items to table block_ts_mapper, blocks end, Item count: 2, Took 0:00:00.022562
hemera      | 2024-06-24 08:18:57,530 - ProgressLogger [INFO] - Started work.
```

## Run From Source Code

#### **1. Install Python3 and pip**

Skip this step if you already have both installed.

```
sudo apt update
sudo apt install python3
sudo apt install python3-pip
```

#### **2. Initiate Python VENV**

Skip this step if you don't want to have a dedicated python venv for Hemera Indexer.

```
sudo apt install python3-venv
python3 -m venv ./venv
```

#### **3. Install pip Dependencies**

```
source ./venv/bin/activate
sudo apt install libpq-dev
pip install -e .

```

{% hint style="info" %}

#### Running using Make script

Another option is to set up and build from source using the make script

You can use the command: `make development`&#x20;

{% endhint %}

#### &#x20;**Prepare Your PostgreSQL Instance**

Hemera Indexer requires a PostgreSQL database to store all indexed data. You may skip this step if you already have a PostgreSQL set up.

#### **5. Setup PostgreSQL**

Follow the instructions about how to set up a PostgreSQL database here: [Setup PostgreSQL on Ubuntu](https://www.cherryservers.com/blog/how-to-install-and-setup-postgresql-server-on-ubuntu-20-04).

#### **6. Configure**

Configure the `OUTPUT` or `--output` parameter according to your PostgreSQL role information. Check out [Configure Hemera Indexer](/hemera-indexer/configurations) for details.

E.g. `postgresql+psycopg2://${YOUR_USER}:${YOUR_PASSWORD}@${YOUR_HOST}:5432/${YOUR_DATABASE}`.

#### **7. Run**

Please check out [Configure Hemera Indexer](/hemera-indexer/configurations) on how to configure the indexer.

```
python hemera.py stream \
    --provider-uri https://eth.llamarpc.com \
    --debug-provider-uri https://eth.llamarpc.com \
    --postgres-url postgresql+psycopg2://devuser:devpassword@localhost:5432/hemera_indexer \
    --output jsonfile://output/eth_blocks_20000001_20010000/json,csvfile://output/hemera_indexer/csv,postgresql+psycopg2://devuser:devpassword@localhost:5432/eth_blocks_20000001_20010000 \
    --start-block 20000001 \
    --end-block 20010000 \
    --output-types block,transaction,log \
    --block-batch-size 200 \
    --batch-size 50 \
    --max-workers 8
```

Once you have successfully bootstrapped Hemera Indexer, you should be able to view similar logs as below.

```
2024-06-25 16:37:38,456 - root [INFO] - Using provider https://eth.llamarpc.com
2024-06-25 16:37:38,456 - root [INFO] - Using debug provider https://eth.llamarpc.com
2024-06-25 16:37:38,485 - alembic.runtime.migration [INFO] - Context impl PostgresqlImpl.
2024-06-25 16:37:38,485 - alembic.runtime.migration [INFO] - Will assume transactional DDL.
2024-06-25 16:37:38,502 - alembic.runtime.migration [INFO] - Context impl PostgresqlImpl.
2024-06-25 16:37:38,502 - alembic.runtime.migration [INFO] - Will assume transactional DDL.
2024-06-25 16:37:39,485 - root [INFO] - Current block 20167548, target block 20137200, last synced block 20137199, blocks to sync 1
2024-06-25 16:37:39,486 - ProgressLogger [INFO] - Started work. Items to process: 1.
2024-06-25 16:37:40,267 - ProgressLogger [INFO] - 1 items processed. Progress is 100%.
2024-06-25 16:37:40,268 - ProgressLogger [INFO] - Finished work. Total items processed: 1. Took 0:00:00.782177.
2024-06-25 16:37:40,283 - exporters.postgres_item_exporter [INFO] - Exporting items to table block_ts_mapper, blocks end, Item count: 2, Took 0:00:00.014799
2024-06-25 16:37:40,283 - ProgressLogger [INFO] - Started work.
```


# Export Result

Hemera Indexer allows you to export the blockchain data to a PostgreSQL database, or to JSON/CSV files.

### Export From PostgreSQL Database

#### **Connect to Your Postgresql Instance**

Use any PostgreSQL client to connect to your PostgreSQL instance, please make sure the `user`, `password`, and `port` is the same as your configuration.

#### **Run In Docker**

By default, the PostgreSQL port is open on and mapped to port 5432 of your ec2 instance, you can verify or change it in the PostgreSQL section of the `docker-compose.yaml`.

#### **Configure Your Network**

If you are using any cloud services, make sure the PostgreSQL port is accessible by updating the network rules.

If you are using AWS and EC2, you can check out [this post](https://www.intelligentdiscovery.io/controls/ec2/aws-ec2-postgresql-open) on how to configure the security group.

### Export To Output Files

#### **Run In Docker**

By default, the `docker-compose.yaml` mounts the `output` folder to `docker-compose/output`, assuming that you are running from `docker-compose` folder. You can find exported results in `docker-compose/output`.

#### **Run From Source Code**

The database and exported file locations are the same as what you configured in `OUTPUT` or `--output` parameter.

E.g., If you specify the `OUTPUT` or `--output` parameter as below

```bash
# Command line parameter
python hemera.py stream \
    --provider-uri https://eth.llamarpc.com \
    --debug-provider-uri https://eth.llamarpc.com \
    --postgres-url postgresql+psycopg2://devuser:devpassword@localhost:5432/hemera_indexer \
    --output jsonfile://output/eth_blocks_20000001_20010000/json,csvfile://output/hemera_indexer/csv,postgresql+psycopg2://devuser:devpassword@localhost:5432/eth_blocks_20000001_20010000 \
    --start-block 20000001 \
    --end-block 20010000 \
    --output-types block,transaction,log \
    --block-batch-size 200 \
    --batch-size 50 \
    --max-workers 8

# Or using environment variable
export OUTPUT = postgresql+psycopg2://user:password@localhost:5432/hemera_indexer,jsonfile://output/json, csvfile://output/csv
```

You will be able to find those results in the `output` folder of your current location.


# Configurations

Hemera indexer can read configuration from cmd line arguments or environment variables.

* If you run Hemera Indexer in Docker, then the environment variable is easier to configure.
* If you prefer running from Source Code, command line arguments are more intuitive.

Run with `python hemera.py stream --help` to get the latest instructions for arguments.

### Parameters

* If the name of the parameter is in `UPPER_CASE` then it's an environment variable.
* If the name of the parameter starts with `--` then it's a command line argument.

Avoid specifying the same parameter from both the environment variable and the command line argument.

**`PROVIDER_URI` or `--provider-uri`**

\[**Default**: `https://mainnet.infura.io`]

The URI of the web3 rpc provider, e.g. `file://$HOME/Library/Ethereum/geth.ipc` or `https://mainnet.infura.io`.

**`DEBUG_PROVIDER_URI` or `--debug-provider-uri`**

\[**Default**: `https://mainnet.infura.io`]

The URI of the web3 debug rpc provider, e.g. `file://$HOME/Library/Ethereum/geth.ipc` or `https://mainnet.infura.io`.

**`POSTGRES_URL` or `--postgres-url`**

\[**Required**] The PostgreSQL connection URL that the Hemera Indexer used to maintain its state. e.g. `postgresql+psycopg2://user:password@127.0.0.1:5432/postgres`.

**`OUTPUT` or `--output` or `-o`**

\[**Required**] You may specify the output parameter so Hemera Indexer will export the data to CSV or JSON files. If not specified the data will be printed to the console.

If you have multiple outputs, use "," to concat the files. The file location will be relative to your current location if you run from source code, or the `output` folder as configured in `docker-compose.yaml`.

e.g.

* `postgresql+psycopg2://user:password@localhost:5432/hemera_indexer`: Output will be exported to your postgres.
* `jsonfile://output/json`: Json files will be exported to folder `output/json`
* `csvfile://output/csv`: Csv files will be exported to folder `output/csv`
* `console,jsonfile://output/json,csvfile://output/csv`: Multiple destinations are supported.

**`ENTITY_TYPES` or `--entity-types` or `-E`**

\[**Default**: `EXPLORER_BASE,EXPLORER_TOKEN`] The list of entity types to export. e.g. `EXPLORER_BASE`, `EXPLORER_TOKEN`, `EXPLORER_TRACE`.

**`OUTPUT_TYPES` or `--output-types` or `-O`**

The list of output types to export, corresponding to more detailed data models. Specifying this option will prioritize these settings over the entity types specified in -E. Available options include: block, transaction, log, token, address\_token\_balance, erc20\_token\_transfer, erc721\_token\_transfer, erc1155\_token\_transfer, trace, contract, coin\_balance.

You may spawn up multiple Hemera Indexer processes, each of them specifying different output types to accelerate the indexing process. For example, indexing `trace` data may take much longer than other entities, you may want to run a separate process to index `trace` data. Checkout `docker-compose/docker-compose.yaml` for examples.

**`DB_VERSION` or `--db-version`**

\[**Default**: `head`] The database version to initialize the database. Using the Alembic script's revision ID to specify a version.\
e.g. `head`, indicates the latest version.\
Or `base`, indicates the empty database without any table.\
Default value: `head`

**`START_BLOCK` or `--start-block`**

The block number to start from, e.g. `0`, `1000`, etc. If you don't specify this, Hemera Indexer will read the last synced block from the PostgreSQL database and resume from it.

**`END_BLOCK` or `--end-block`**

The block number that ends at, e.g. `10000`, `20000`, etc.

**`PARTITION_SIZE` or `--partition-size`**

\[**Default**: `50000`] The number of records to write to each file.

**`PERIOD_SECONDS` or `--period-seconds`**

\[**Default**: `10`] Seconds to sleep between each sync with the latest blockchain state.

**`BATCH_SIZE` or `--batch-size`**

\[**Default**: `10`] The number of non-debug rpc requests to batch in a single request.

**`DEBUG_BATCH_SIZE` or `--debug-batch-size`**

\[**Default**: `1`] The number of debug rpc to batch in a single request.

**`BLOCK_BATCH_SIZE` or `--block-batch-size`**

\[**Default**: `1`] The number of blocks to batch in a single sync round.

**`MAX_WORKERS` or `--max-workers`**

\[**Default**: `5`] The number of workers, e.g. `4`, `5`, etc.

**`LOG_FILE` or `--log-file`**

The log file to use. e.g. `path/to/logfile.log`.


# Benchmark

Indexing full blockchain data can be a resource-heavy task. Ethereum mainnet has more than 20B transactions now.

We run a series of benchmark tests using the Hemera Indexer. Be aware of the potential time frame and resources you need to index your blockchain. Also, make sure yourtune parameters before you start your production run.


# Data Class


# Raw Data Tables


# Blocks

Blocks are the building blocks of blockchains and rollups. A block contains transactions that will alter the state of an EVM system incrementally. Transactions within a block can only be executed one after the other, not in parallel.

These tables are useful for identifying block activity and transaction changes over time.

## ETL data class

| Column Name                   | Data Type | Note        |
| ----------------------------- | --------- | ----------- |
| timestamp                     | timestamp |             |
| number                        | int       |             |
| hash                          | str       | primary key |
| parent\_hash                  | str       |             |
| nonce                         | str       |             |
| sha3\_uncles                  | str       |             |
| transactions\_root            | str       |             |
| state\_root                   | str       |             |
| receipts\_root                | str       |             |
| miner                         | str       |             |
| difficulty                    | int       |             |
| total\_difficulty             | int       |             |
| size                          | int       |             |
| extra\_data                   | str       |             |
| gas\_limit                    | int       |             |
| gas\_used                     | int       |             |
| transactions\_count           | int       |             |
| base\_fee\_per\_gas           | int       |             |
| withdrawals\_root             | str       |             |
| blob\_gas\_used               | int       |             |
| excess\_blob\_gas             | int       |             |
| traces\_count                 | int       |             |
| internal\_transactions\_count | int       |             |

Examples

```json
{
  "timestamp": "2023-10-29T03:34:19.000Z",
  "number": 435530,
  "hash": "0x68189ad6461faff3837f22571813eb507de3bdca2d1c9b596f233bd1fc8ff810",
  "parent_hash": "0x04db8bfd20912bba895e7697f48997c38ebe95a7392e58f7b88777764555a50c",
  "nonce": "0",
  "sha3_uncles": "0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347",
  "transactions_root": "0x5c704be9d382467cb8ee1e12f0ab04e30592d3fb9acb8fe7fda7e2a09bc788b0",
  "state_root": "0xeb14acf5feb30868d3ed1a29d4e1347beca86b5086a8e516041e7d237e7c0f7a",
  "receipts_root": "0x40c0b3b1cb04517954a19c2d24b5a30042fc1d0281e3b9a8ffffae13a0aa9269",
  "miner": "0x4200000000000000000000000000000000000011",
  "difficulty": 0,
  "total_difficulty": 0,
  "size": 3047,
  "extra_data": "0x",
  "gas_limit": 30000000,
  "gas_used": 965239,
  "transactions_count": 7,
  "base_fee_per_gas": 50
}
```

| Column Name                   | Data Type    | Note          |
| ----------------------------- | ------------ | ------------- |
| hash                          | bytea        | primary key   |
| number                        | bigint       |               |
| timestamp                     | timestamp    |               |
| parent\_hash                  | bytea        |               |
| nonce                         | bytea        |               |
| gas\_limit                    | numeric(100) |               |
| gas\_used                     | numeric(100) |               |
| base\_fee\_per\_gas           | numeric(100) |               |
| difficulty                    | numeric(38)  |               |
| total\_difficulty             | numeric(38)  |               |
| size                          | bigint       |               |
| miner                         | bytea        |               |
| sha3\_uncles                  | bytea        |               |
| transactions\_root            | bytea        |               |
| transactions\_count           | bigint       |               |
| state\_root                   | bytea        |               |
| receipts\_root                | bytea        |               |
| extra\_data                   | bytea        |               |
| withdrawals\_root             | bytea        |               |
| create\_time                  | timestamp    | default now() |
| update\_time                  | timestamp    | default now() |
| reorg                         | boolean      |               |
| blob\_gas\_used               | numeric(100) |               |
| excess\_blob\_gas             | numeric(100) |               |
| traces\_count                 | bigint       |               |
| internal\_transactions\_count | bigint       |               |

Examples

```json
{
  "hash": "E'\\\\x68189AD6461FAFF3837F22571813EB507DE3BDCA2D1C9B596F233BD1FC8FF810'",
  "number": 435530,
  "timestamp": "2023-10-29 03:34:19.000000",
  "parent_hash": "E'\\\\x04DB8BFD20912BBA895E7697F48997C38EBE95A7392E58F7B88777764555A50C'",
  "nonce": "E'\\\\x0000000000000000'",
  "gas_limit": 30000000,
  "gas_used": 965239,
  "base_fee_per_gas": 50,
  "difficulty": 0,
  "total_difficulty": 0,
  "size": 3047,
  "miner": "E'\\\\x4200000000000000000000000000000000000011'",
  "sha3_uncles": "E'\\\\x1DCC4DE8DEC75D7AAB85B567B6CCD41AD312451B948A7413F0A142FD40D49347'",
  "transactions_root": "E'\\\\x5C704BE9D382467CB8EE1E12F0AB04E30592D3FB9ACB8FE7FDA7E2A09BC788B0'",
  "transactions_count": 7,
  "state_root": "E'\\\\xEB14ACF5FEB30868D3ED1A29D4E1347BECA86B5086A8E516041E7D237E7C0F7A'",
  "receipts_root": "E'\\\\x40C0B3B1CB04517954A19C2D24B5A30042FC1D0281E3B9A8FFFFAE13A0AA9269'",
  "extra_data": "E'\\\\x'",
  "withdrawals_root": null,
  "create_time": "2024-08-12 13:38:46.949956",
  "update_time": null,
  "reorg": false,
  "blob_gas_used": null,
  "excess_blob_gas": null,
  "traces_count": 15,
  "internal_transactions_count": 3
}
```

```csv
0x68189AD6461FAFF3837F22571813EB507DE3BDCA2D1C9B596F233BD1FC8FF810,435530,2023-10-29 03:34:19.000000,0x04DB8BFD20912BBA895E7697F48997C38EBE95A7392E58F7B88777764555A50C,0x0000000000000000,30000000,965239,50,0,0,3047,0x4200000000000000000000000000000000000011,0x1DCC4DE8DEC75D7AAB85B567B6CCD41AD312451B948A7413F0A142FD40D49347,0x5C704BE9D382467CB8EE1E12F0AB04E30592D3FB9ACB8FE7FDA7E2A09BC788B0,7,0xEB14ACF5FEB30868D3ED1A29D4E1347BECA86B5086A8E516041E7D237E7C0F7A,0x40C0B3B1CB04517954A19C2D24B5A30042FC1D0281E3B9A8FFFFAE13A0AA9269,0x,null,2024-08-12 13:38:46.949956,null,false,null,null,15,3
```


# Transactions

Transactions are cryptographically signed instructions from accounts. An account will initiate a transaction to update the state of the Ethereum network. Transactions will always originate from externally owned accounts, a smart contract cannot initiate a transaction.

Transactions need to be broadcast to the whole network. Any node can broadcast a request for a transaction to be executed on the EVM; after this happens, a miner will execute the transaction and propagate the resulting state change to the rest of the network.

## ETL data class

| Column Name                    | Data Type | Note |
| ------------------------------ | --------- | ---- |
| hash                           | str       |      |
| nonce                          | int       |      |
| transaction\_index             | int       |      |
| from\_address                  | str       |      |
| to\_address                    | str       |      |
| value                          | int       |      |
| gas                            | int       |      |
| gas\_price                     | int       |      |
| input                          | str       |      |
| receipt\_cumulative\_gas\_used | int       |      |
| receipt\_gas\_used             | int       |      |
| receipt\_contract\_address     | str       |      |
| receipt\_root                  | str       |      |
| receipt\_status                | int       |      |
| block\_timestamp               | timestamp |      |
| block\_number                  | int       |      |
| block\_hash                    | str       |      |
| max\_fee\_per\_gas             | int       |      |
| max\_priority\_fee\_per\_gas   | int       |      |
| transaction\_type              | int       |      |
| receipt\_effective\_gas\_price | int       |      |
| receipt\_l1\_fee               | int       |      |
| receipt\_l1\_gas\_used         | int       |      |
| receipt\_l1\_gas\_price        | int       |      |
| receipt\_l1\_fee\_scalar       | float     |      |
| receipt\_blob\_gas\_used       | int       |      |
| receipt\_blob\_gas\_price      | int       |      |
| blob\_versioned\_hashes        | list      |      |
| exist\_error                   | bool      |      |
| error                          | str       |      |
| revert\_reason                 | str       |      |

Examples

```json
{
  "hash": "0x35c6f6ec2ddb2530a8f01a6dc05666da248b6c1d7124a3f7c80d5e557596e9a7",
  "nonce": 154,
  "transaction_index": 4,
  "from_address": "0x634de605fea9f9c3245034129ea30c2cee399972",
  "to_address": "0xa44155ffbce68c9c848f8ea6f28c40311085125e",
  "value": 0,
  "gas": 1219916,
  "gas_price": 84899924,
  "input": "0x733f53280000000000000000000000000000000000000000000000000000000000000040000000000000000000000000000000000000000000000000000000000000016000000000000000000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000000006d00000000000000000000000000000000000000000000000000000000000000a00fd0ba716e314ca66bd0c6fb8d27f6da149ab7808057bf36616f42be663d8cce00000000000000000000000000000000000000000000000001593477e4ffffff0000000000000000000000000000000000000000000000000000018bc9c817120000000000000000000000000000000000000000000000000000000000000004302e303200000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000000200000000000000000000000000000000000000000000000000000000000000041c3c950926b7e0bf4e7ba1d5c1f367c05ba3bb7036020c2ca1bd65747ac597e0c034b42e85aab0bbebaaa7610e20a676b64b8858af2d1a4437a53a07abeacc6eb1b00000000000000000000000000000000000000000000000000000000000000",
  "receipt_cumulative_gas_used": 2368079,
  "receipt_gas_used": 1102725,
  "receipt_contract_address": null,
  "receipt_root": null,
  "receipt_status": 1,
  "block_timestamp": "2023-11-13T09:43:49.000Z",
  "block_number": 567347,
  "block_hash": "0xf06ba757ac30de0d45c3d3e68c697c39246747009e93ebb528b818297f0c9438",
  "max_fee_per_gas": 1671037270,
  "max_priority_fee_per_gas": 100000,
  "transaction_type": 2,
  "receipt_effective_gas_price": 84899924,
  "receipt_l1_fee": 698092851650339,
  "receipt_l1_gas_used": 7676,
  "receipt_l1_gas_price": 69957595267,
  "receipt_l1_fee_scalar": 1.3
}
```

## Model in the Database

| Column Name                    | Data Type        | Note                                                                          |
| ------------------------------ | ---------------- | ----------------------------------------------------------------------------- |
| hash                           | bytea            | primary key                                                                   |
| transaction\_index             | integer          |                                                                               |
| from\_address                  | bytea            |                                                                               |
| to\_address                    | bytea            |                                                                               |
| value                          | numeric(100)     |                                                                               |
| transaction\_type              | integer          |                                                                               |
| input                          | bytea            |                                                                               |
| nonce                          | integer          |                                                                               |
| block\_hash                    | bytea            |                                                                               |
| block\_number                  | bigint           |                                                                               |
| block\_timestamp               | timestamp        |                                                                               |
| gas                            | numeric(100)     |                                                                               |
| gas\_price                     | numeric(100)     |                                                                               |
| max\_fee\_per\_gas             | numeric(100)     |                                                                               |
| max\_priority\_fee\_per\_gas   | numeric(100)     |                                                                               |
| receipt\_root                  | bytea            |                                                                               |
| receipt\_status                | integer          |                                                                               |
| receipt\_gas\_used             | numeric(100)     |                                                                               |
| receipt\_cumulative\_gas\_used | numeric(100)     |                                                                               |
| receipt\_effective\_gas\_price | numeric(100)     |                                                                               |
| receipt\_l1\_fee               | numeric(100)     |                                                                               |
| receipt\_l1\_fee\_scalar       | numeric(100, 18) |                                                                               |
| receipt\_l1\_gas\_used         | numeric(100)     |                                                                               |
| receipt\_l1\_gas\_price        | numeric(100)     |                                                                               |
| receipt\_blob\_gas\_used       | numeric(100)     |                                                                               |
| receipt\_blob\_gas\_price      | numeric(100)     |                                                                               |
| blob\_versioned\_hashes        | bytea\[]         |                                                                               |
| receipt\_contract\_address     | bytea            |                                                                               |
| exist\_error                   | boolean          |                                                                               |
| error                          | text             |                                                                               |
| revert\_reason                 | text             |                                                                               |
| create\_time                   | timestamp        | default now()                                                                 |
| update\_time                   | timestamp        | default now()                                                                 |
| reorg                          | boolean          |                                                                               |
| method\_id                     | varchar          | generated always as (substr(((input)::character varying)::text, 3, 8)) stored |

Examples

```json
{
  "hash": "E'\\\\x35C6F6EC2DDB2530A8F01A6DC05666DA248B6C1D7124A3F7C80D5E557596E9A7'",
  "transaction_index": 0,
  "from_address": "E'\\\\x634DE605FEA9F9C3245034129EA30C2CEE399972'",
  "to_address": "E'\\\\xA44155FFBCE68C9C848F8EA6F28C40311085125E'",
  "value": 0,
  "transaction_type": 2,
  "input": "E'\\\\x733F53280000000000000000000000000000000000000000000000000000000000000040000000000000000000000000000000000000000000000000000000000000016000000000000000000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000000006D00000000000000000000000000000000000000000000000000000000000000A00FD0BA716E314CA66BD0C6FB8D27F6DA149AB7808057BF36616F42BE663D8CCE00000000000000000000000000000000000000000000000001593477E4FFFFFF0000000000000000000000000000000000000000000000000000018BC9C817120000000000000000000000000000000000000000000000000000000000000004302E303200000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000000200000000000000000000000000000000000000000000000000000000000000041C3C950926B7E0BF4E7BA1D5C1F367C05BA3BB7036020C2CA1BD65747AC597E0C034B42E85AAB0BBEBAAA7610E20A676B64B8858AF2D1A4437A53A07ABEACC6EB1B00000000000000000000000000000000000000000000000000000000000000'",
  "nonce": 154,
  "block_hash": "E'\\\\xF06BA757AC30DE0D45C3D3E68C697C39246747009E93EBB528B818297F0C9438'",
  "block_number": 567347,
  "block_timestamp": "2023-11-13 09:43:49.000000",
  "gas": 1219916,
  "gas_price": 84899924,
  "max_fee_per_gas": 1671037270,
  "max_priority_fee_per_gas": 100000,
  "receipt_root": null,
  "receipt_status": 1,
  "receipt_gas_used": 1102725,
  "receipt_cumulative_gas_used": 2368079,
  "receipt_effective_gas_price": 84899924,
  "receipt_l1_fee": 698092851650339,
  "receipt_l1_fee_scalar": 1.3,
  "receipt_l1_gas_used": 7676,
  "receipt_l1_gas_price": 69957595267,
  "receipt_blob_gas_used": null,
  "receipt_blob_gas_price": null,
  "blob_versioned_hashes": null,
  "receipt_contract_address": null,
  "exist_error": false,
  "error": null,
  "revert_reason": null,
  "create_time": "2024-08-12 13:38:46.949956",
  "update_time": null,
  "reorg": false,
  "method_id": "733f5328"
}
```

```csv
0x35C6F6EC2DDB2530A8F01A6DC05666DA248B6C1D7124A3F7C80D5E557596E9A7,0,0x634DE605FEA9F9C3245034129EA30C2CEE399972,0xA44155FFBCE68C9C848F8EA6F28C40311085125E,0,2,0x733F53280000000000000000000000000000000000000000000000000000000000000040000000000000000000000000000000000000000000000000000000000000016000000000000000000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000000006D00000000000000000000000000000000000000000000000000000000000000A00FD0BA716E314CA66BD0C6FB8D27F6DA149AB7808057BF36616F42BE663D8CCE00000000000000000000000000000000000000000000000001593477E4FFFFFF0000000000000000000000000000000000000000000000000000018BC9C817120000000000000000000000000000000000000000000000000000000000000004302E303200000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000000200000000000000000000000000000000000000000000000000000000000000041C3C950926B7E0BF4E7BA1D5C1F367C05BA3BB7036020C2CA1BD65747AC597E0C034B42E85AAB0BBEBAAA7610E20A676B64B8858AF2D1A4437A53A07ABEACC6EB1B00000000000000000000000000000000000000000000000000000000000000,154,0xF06BA757AC30DE0D45C3D3E68C697C39246747009E93EBB528B818297F0C9438,567347,2023-11-13 09:43:49.000000,1219916,84899924,1671037270,100000,null,1,1102725,2368079,84899924,698092851650339,1.3,7676,69957595267,null,null,null,null,false,null,null,2024-08-12 13:38:46.949956,null,false,733f5328
```


# Logs

L1 to L2 Transactions represent the transfer of assets or data from the Layer 1 (L1) blockchain (typically Ethereum mainnet) to the Layer 2 (L2) blockchain. These transactions are crucial for users to move their assets to the L2 network, enabling faster and cheaper transactions while maintaining the security guarantees of the L1 network.

## ETL data class

| Column Name           | Data Type | Note |
| --------------------- | --------- | ---- |
| index                 | int       |      |
| l1\_block\_number     | int       |      |
| l1\_block\_timestamp  | timestamp |      |
| l1\_block\_hash       | str       |      |
| l1\_transaction\_hash | str       |      |
| l1\_from\_address     | str       |      |
| l1\_to\_address       | str       |      |
| l2\_block\_number     | int       |      |
| l2\_block\_timestamp  | timestamp |      |
| l2\_block\_hash       | str       |      |
| l2\_transaction\_hash | str       |      |
| l2\_from\_address     | str       |      |
| l2\_to\_address       | str       |      |
| amount                | decimal   |      |
| from\_address         | str       |      |
| to\_address           | str       |      |
| l1\_token\_address    | str       |      |
| l2\_token\_address    | str       |      |
| extra\_info           | json      |      |
| \_type                | int       |      |
| deposit\_hash         | str       |      |

Examples

```json
{
  "index": 218,
  "l1_block_number": 18126236,
  "l1_block_timestamp": "2023-09-13T00:36:23.000Z",
  "l1_block_hash": null,
  "l1_transaction_hash": "0xca4f846d51cac9cf70369b01818fc1af0221dbd5dd8e011fdb00c7a161ee3027",
  "l1_from_address": "0xa5f146cbd3ee13f482315dd0f873c2bfbbc5f2c4",
  "l1_to_address": "0x3b95bc951ee0f553ba487327278cac44f29715e5",
  "l2_block_number": 37037,
  "l2_block_timestamp": "2023-09-13T00:38:49.000Z",
  "l2_block_hash": null,
  "l2_transaction_hash": "0xbfb147979b46b4caf21c7f7851eac6191dfdd4bf59e78c274c0e461752e9a65e",
  "l2_from_address": "0x746ca609680c55c3bdd0b3627b4c5db21b13d421",
  "l2_to_address": "0x4200000000000000000000000000000000000007",
  "amount": 5000000000000000,
  "from_address": "0xa5f146cbd3ee13f482315dd0f873c2bfbbc5f2c4",
  "to_address": "0xa5f146cbd3ee13f482315dd0f873c2bfbbc5f2c4",
  "l1_token_address": "0x0000000000000000000000000000000000000000",
  "l2_token_address": "0xdeaddeaddeaddeaddeaddeaddeaddeaddead1111",
  "extra_info": {
    "sender": "0x3b95bc951ee0f553ba487327278cac44f29715e5",
    "target": "0x4200000000000000000000000000000000000010",
    "gas_limit": 200000,
    "tx_origin": "0x746ca609680c55c3bdd0b3627b4c5db21b13d421"
  },
  "_type": null,
  "deposit_hash": null
}
```

```csv
218,18126236,2023-09-13T00:36:23.000Z,,0xca4f846d51cac9cf70369b01818fc1af0221dbd5dd8e011fdb00c7a161ee3027,0xa5f146cbd3ee13f482315dd0f873c2bfbbc5f2c4,0x3b95bc951ee0f553ba487327278cac44f29715e5,37037,2023-09-13T00:38:49.000Z,,0xbfb147979b46b4caf21c7f7851eac6191dfdd4bf59e78c274c0e461752e9a65e,0x746ca609680c55c3bdd0b3627b4c5db21b13d421,0x4200000000000000000000000000000000000007,5000000000000000,0xa5f146cbd3ee13f482315dd0f873c2bfbbc5f2c4,0xa5f146cbd3ee13f482315dd0f873c2bfbbc5f2c4,0x0000000000000000000000000000000000000000,0xdeaddeaddeaddeaddeaddeaddeaddeaddead1111,"{""sender"": ""0x3b95bc951ee0f553ba487327278cac44f29715e5"", ""target"": ""0x4200000000000000000000000000000000000010"", ""gas_limit"": 200000, ""tx_origin"": ""0x746ca609680c55c3bdd0b3627b4c5db21b13d421""}",,,
```

## Model in the Database

| Column Name           | Data Type   | Note |
| --------------------- | ----------- | ---- |
| index                 | bigint      |      |
| l1\_block\_number     | bigint      |      |
| l1\_block\_timestamp  | timestamp   |      |
| l1\_block\_hash       | varchar(66) |      |
| l1\_transaction\_hash | varchar(66) |      |
| l1\_from\_address     | varchar(42) |      |
| l1\_to\_address       | varchar(42) |      |
| l2\_block\_number     | bigint      |      |
| l2\_block\_timestamp  | timestamp   |      |
| l2\_block\_hash       | varchar(66) |      |
| l2\_transaction\_hash | varchar(66) |      |
| l2\_from\_address     | varchar(42) |      |
| l2\_to\_address       | varchar(42) |      |
| amount                | numeric(78) |      |
| from\_address         | varchar(42) |      |
| to\_address           | varchar(42) |      |
| l1\_token\_address    | varchar(42) |      |
| l2\_token\_address    | varchar(42) |      |
| extra\_info           | jsonb       |      |
| \_type                | integer     |      |
| deposit\_hash         | varchar     |      |

```json
{
  "index": 218,
  "l1_block_number": 18126236,
  "l1_block_timestamp": "2023-09-13 00:36:23.000000",
  "l1_block_hash": null,
  "l1_transaction_hash": "0xca4f846d51cac9cf70369b01818fc1af0221dbd5dd8e011fdb00c7a161ee3027",
  "l1_from_address": "0xa5f146cbd3ee13f482315dd0f873c2bfbbc5f2c4",
  "l1_to_address": "0x3b95bc951ee0f553ba487327278cac44f29715e5",
  "l2_block_number": 37037,
  "l2_block_timestamp": "2023-09-13 00:38:49.000000",
  "l2_block_hash": null,
  "l2_transaction_hash": "0xbfb147979b46b4caf21c7f7851eac6191dfdd4bf59e78c274c0e461752e9a65e",
  "l2_from_address": "0x746ca609680c55c3bdd0b3627b4c5db21b13d421",
  "l2_to_address": "0x4200000000000000000000000000000000000007",
  "amount": 5000000000000000,
  "from_address": "0xa5f146cbd3ee13f482315dd0f873c2bfbbc5f2c4",
  "to_address": "0xa5f146cbd3ee13f482315dd0f873c2bfbbc5f2c4",
  "l1_token_address": "0x0000000000000000000000000000000000000000",
  "l2_token_address": "0xdeaddeaddeaddeaddeaddeaddeaddeaddead1111",
  "extra_info": {
    "sender": "0x3b95bc951ee0f553ba487327278cac44f29715e5",
    "target": "0x4200000000000000000000000000000000000010",
    "gas_limit": 200000,
    "tx_origin": "0x746ca609680c55c3bdd0b3627b4c5db21b13d421"
  },
  "_type": null,
  "deposit_hash": null
}
```

```csv
218,18126236,2023-09-13 00:36:23.000000,,0xca4f846d51cac9cf70369b01818fc1af0221dbd5dd8e011fdb00c7a161ee3027,0xa5f146cbd3ee13f482315dd0f873c2bfbbc5f2c4,0x3b95bc951ee0f553ba487327278cac44f29715e5,37037,2023-09-13 00:38:49.000000,,0xbfb147979b46b4caf21c7f7851eac6191dfdd4bf59e78c274c0e461752e9a65e,0x746ca609680c55c3bdd0b3627b4c5db21b13d421,0x4200000000000000000000000000000000000007,5000000000000000,0xa5f146cbd3ee13f482315dd0f873c2bfbbc5f2c4,0xa5f146cbd3ee13f482315dd0f873c2bfbbc5f2c4,0x0000000000000000000000000000000000000000,0xdeaddeaddeaddeaddeaddeaddeaddeaddead1111,"{""sender"": ""0x3b95bc951ee0f553ba487327278cac44f29715e5"", ""target"": ""0x4200000000000000000000000000000000000010"", ""gas_limit"": 200000, ""tx_origin"": ""0x746ca609680c55c3bdd0b3627b4c5db21b13d421""}",,,
```


# Traces

Transactions can trigger smaller atomic actions that modify the internal state of an Ethereum Virtual Machine. Information about the execution of these actions is logged and can be found stored as an EVM execution trace, or just a *trace*.

## ETL data class

<table><thead><tr><th width="215.70987654320987">Column Name</th><th width="193">Data Type</th><th>Note</th></tr></thead><tbody><tr><td>transaction_hash</td><td>str</td><td></td></tr><tr><td>transaction_index</td><td>int</td><td></td></tr><tr><td>from_address</td><td>str</td><td></td></tr><tr><td>to_address</td><td>str</td><td></td></tr><tr><td>value</td><td>decimal</td><td></td></tr><tr><td>input</td><td>str</td><td></td></tr><tr><td>output</td><td>str</td><td></td></tr><tr><td>trace_type</td><td>str</td><td></td></tr><tr><td>call_type</td><td>str</td><td></td></tr><tr><td>reward_type</td><td>str</td><td></td></tr><tr><td>gas</td><td>int</td><td></td></tr><tr><td>gas_used</td><td>decimal</td><td></td></tr><tr><td>subtraces</td><td>int</td><td></td></tr><tr><td>trace_address</td><td>str</td><td></td></tr><tr><td>error</td><td>str</td><td></td></tr><tr><td>status</td><td>int</td><td></td></tr><tr><td>block_timestamp</td><td>timestamp</td><td></td></tr><tr><td>block_number</td><td>int</td><td></td></tr><tr><td>block_hash</td><td>str</td><td></td></tr><tr><td>trace_id</td><td>str</td><td></td></tr></tbody></table>

Examples

```json
{
  "transaction_hash": "0xd00acf04d3164d86224c9094571c253a48cc6f3a9110a22e63e1c655acc5372f",
  "transaction_index": 0,
  "from_address": "0x1e2f91f8aa563e02e158da8870749a9bd7edfcad",
  "to_address": "0xb29caa2cb1feb7f4ccaa9dd9b8ad2022eaca6ec3",
  "value": 0,
  "input": "0x628ea1fc0000000000000000000000000a882d83a3d3160a6ef5192f03a79afc40208bde0000000000000000000000000000000000000000000000000000000000002710",
  "output": null,
  "trace_type": "call",
  "call_type": "call",
  "reward_type": null,
  "gas": 53325,
  "gas_used": 52943,
  "subtraces": 1,
  "trace_address": "{}",
  "error": null,
  "status": 1,
  "block_timestamp": "2023-07-17T18:00:01.000Z",
  "block_number": 2058,
  "block_hash": "0xdaed8fcdbd146aa3e0c63d24f9cf647dd26c855086ba5786ab96d1738ddadb63",
  "trace_id": "call_2058_0"
}
```

## Model in the Database

| Column Name        | Data Type    | Note          |
| ------------------ | ------------ | ------------- |
| trace\_id          | varchar      | primary key   |
| from\_address      | bytea        |               |
| to\_address        | bytea        |               |
| value              | numeric(100) |               |
| input              | bytea        |               |
| output             | bytea        |               |
| trace\_type        | varchar      |               |
| call\_type         | varchar      |               |
| gas                | numeric(100) |               |
| gas\_used          | numeric(100) |               |
| subtraces          | integer      |               |
| trace\_address     | integer\[]   |               |
| error              | text         |               |
| status             | integer      |               |
| block\_number      | bigint       |               |
| block\_hash        | bytea        |               |
| block\_timestamp   | timestamp    |               |
| transaction\_index | integer      |               |
| transaction\_hash  | bytea        |               |
| create\_time       | timestamp    | default now() |
| update\_time       | timestamp    | default now() |
| reorg              | boolean      |               |

Examples

```json
{
  "trace_id": "call_2058_0",
  "from_address": "E'\\\\x1E2F91F8AA563E02E158DA8870749A9BD7EDFCAD'",
  "to_address": "E'\\\\xB29CAA2CB1FEB7F4CCAA9DD9B8AD2022EACA6EC3'",
  "value": 0,
  "input": "E'\\\\x628EA1FC0000000000000000000000000A882D83A3D3160A6EF5192F03A79AFC40208BDE0000000000000000000000000000000000000000000000000000000000002710'",
  "output": null,
  "trace_type": "call",
  "call_type": "call",
  "gas": 53325,
  "gas_used": 52943,
  "subtraces": 1,
  "trace_address": "{0}",
  "error": null,
  "status": 1,
  "block_number": 2058,
  "block_hash": "E'\\\\xDAED8FCDBD146AA3E0C63D24F9CF647DD26C855086BA5786AB96D1738DDADB63'",
  "block_timestamp": "2023-07-17 18:00:01.000000",
  "transaction_index": 0,
  "transaction_hash": "E'\\\\xD00ACF04D3164D86224C9094571C253A48CC6F3A9110A22E63E1C655ACC5372F'",
  "create_time": "2024-08-12 13:38:46.949956",
  "update_time": null,
  "reorg": false
}
```

```csv
call_2058_0,0x1E2F91F8AA563E02E158DA8870749A9BD7EDFCAD,0xB29CAA2CB1FEB7F4CCAA9DD9B8AD2022EACA6EC3,0,0x628EA1FC0000000000000000000000000A882D83A3D3160A6EF5192F03A79AFC40208BDE0000000000000000000000000000000000000000000000000000000000002710,null,call,call,53325,52943,1,"{0}",null,1,2058,0xDAED8FCDBD146AA3E0C63D24F9CF647DD26C855086BA5786AB96D1738DDADB63,2023-07-17 18:00:01.000000,0,0xD00ACF04D3164D86224C9094571C253A48CC6F3A9110A22E63E1C655ACC5372F,2024-08-12 13:38:46.949956,null,false
```


# Generated Tables


# Contract Internal Transactions

The Contract Internal Transactions table is used to store data extracted from transaction traces. It specifically focuses on traces related to contract creation or traces that involve a transfer of value.

## ETL data class

| Column Name        | Data Type | Note        |
| ------------------ | --------- | ----------- |
| transaction\_hash  | str       | length 66   |
| transaction\_index | int       |             |
| from\_address      | str       | length 42   |
| to\_address        | str       | length 42   |
| value              | int       |             |
| trace\_type        | str       | length 16   |
| call\_type         | str       | length 16   |
| gas                | int       |             |
| gas\_used          | int       |             |
| subtraces          | int       |             |
| trace\_address     | str       | length 8192 |
| error              | str       |             |
| status             | int       |             |
| block\_timestamp   | timestamp |             |
| block\_number      | int       |             |
| block\_hash        | str       | length 66   |
| trace\_id          | str       |             |

Examples

```json
{
  "transaction_hash": "0xd5b40b43483e3aee69f781c3547ff14895244823e766f6f0a696518d3ec05fac",
  "transaction_index": 4,
  "from_address": "0xd06f95a78553fd94b05281aa97e8afd87af48cef",
  "to_address": "0x5dac7efe0b4cd2cabdd350efc0c69bcaa81e76c6",
  "value": 6499999999997945,
  "trace_type": "call",
  "call_type": "call",
  "gas": 371201,
  "gas_used": 312121,
  "subtraces": null,
  "trace_address": "{}",
  "error": null,
  "status": 1,
  "block_timestamp": "2023-10-29T03:34:19.000Z",
  "block_number": 435530,
  "block_hash": "0x68189ad6461faff3837f22571813eb507de3bdca2d1c9b596f233bd1fc8ff810",
  "trace_id": "435530_4_0"
}
```

```csv
0xd5b40b43483e3aee69f781c3547ff14895244823e766f6f0a696518d3ec05fac,4,0xd06f95a78553fd94b05281aa97e8afd87af48cef,0x5dac7efe0b4cd2cabdd350efc0c69bcaa81e76c6,6499999999997945,call,call,371201,312121,,{},,,2023-10-29T03:34:19.000Z,435530,0x68189ad6461faff3837f22571813eb507de3bdca2d1c9b596f233bd1fc8ff810,435530_4_0
```

## Model in the Database

| Column Name        | Data Type   | Note |
| ------------------ | ----------- | ---- |
| transaction\_hash  | bytea       |      |
| transaction\_index | bigint      |      |
| from\_address      | bytea       |      |
| to\_address        | bytea       |      |
| value              | numeric(78) |      |
| trace\_type        | varchar     |      |
| call\_type         | varchar     |      |
| gas                | numeric(38) |      |
| gas\_used          | numeric(38) |      |
| subtraces          | bigint      |      |
| trace\_address     | varchar     |      |
| error              | text        |      |
| status             | integer     |      |
| block\_timestamp   | timestamp   |      |
| block\_number      | bigint      |      |
| block\_hash        | bytea       |      |
| trace\_id          | text        |      |

Examples

```json
{
  "transaction_hash": "E'\\\\xd5b40b43483e3aee69f781c3547ff14895244823e766f6f0a696518d3ec05fac'",
  "transaction_index": 4,
  "from_address": "E'\\\\xd06f95a78553fd94b05281aa97e8afd87af48cef'",
  "to_address": "E'\\\\x5dac7efe0b4cd2cabdd350efc0c69bcaa81e76c6'",
  "value": 6499999999997945,
  "trace_type": "call",
  "call_type": "call",
  "gas": 371201,
  "gas_used": 312121,
  "subtraces": null,
  "trace_address": "{}",
  "error": null,
  "status": 1,
  "block_timestamp": "2023-10-29 03:34:19.000000",
  "block_number": 435530,
  "block_hash": "E'\\\\x68189ad6461faff3837f22571813eb507de3bdca2d1c9b596f233bd1fc8ff810'",
  "trace_id": "435530_4_0"
}
```

```csv
\\xd5b40b43483e3aee69f781c3547ff14895244823e766f6f0a696518d3ec05fac,4,\\xd06f95a78553fd94b05281aa97e8afd87af48cef,\\x5dac7efe0b4cd2cabdd350efc0c69bcaa81e76c6,6499999999997945,call,call,371201,312121,,{},,,1,2023-10-29 03:34:19.000000,435530,\\x68189ad6461faff3837f22571813eb507de3bdca2d1c9b596f233bd1fc8ff810,435530_4_0
```


# ERC20 Token Transfers

ERC20 Token Transfers table records all the transfer events of ERC20 tokens. Each row represents a single token transfer, including the token address, sender (from\_address), recipient (to\_address), amount (value), and the transaction and block details associated with the transfer.

## ETL data class

| Column Name       | Data Type | Note |
| ----------------- | --------- | ---- |
| token\_address    | str       |      |
| from\_address     | str       |      |
| to\_address       | str       |      |
| value             | int       |      |
| transaction\_hash | str       |      |
| log\_index        | int       |      |
| block\_timestamp  | timestamp |      |
| block\_number     | int       |      |
| block\_hash       | str       |      |

Examples

```json
{
  "token_address": "0x80137510979822322193fc997d400d5a6c747bf7",
  "wallet_address": "0xcaab1bd9025a576cee4757506d81b32e4724f9ef",
  "balance_of": 9834981273804,
  "transaction_hash": null,
  "block_timestamp": "2023-11-16T01:06:59.000Z",
  "block_number": 590166,
  "block_hash": null
}
```

```csv
0xf417f5a458ec102b90352f697d6e2ac3a3d2851f,0xdc5aabac622b5f472ba377c62522e97836a37a4f,0xbad4ccc91ef0dfffbcab1402c519601fbaf244ef,10872268,0x83c246999b005ba42738cdbf3b585d5c212ab9726b2cdf4dc9b67f2e93df4390,10,2023-10-29T03:34:19.000Z,435530,0x68189ad6461faff3837f22571813eb507de3bdca2d1c9b596f233bd1fc8ff810
```

## Model in the Database

| Column Name       | Data Type   | Note |
| ----------------- | ----------- | ---- |
| token\_address    | bytea       |      |
| from\_address     | bytea       |      |
| to\_address       | bytea       |      |
| value             | numeric(78) |      |
| transaction\_hash | bytea       |      |
| log\_index        | bigint      |      |
| block\_timestamp  | timestamp   |      |
| block\_number     | bigint      |      |
| block\_hash       | bytea       |      |

Examples

```json
{
  "token_address": "\\xf417f5a458ec102b90352f697d6e2ac3a3d2851f",
  "from_address": "\\xdc5aabac622b5f472ba377c62522e97836a37a4f",
  "to_address": "\\xbad4ccc91ef0dfffbcab1402c519601fbaf244ef",
  "value": 10872268,
  "transaction_hash": "\\x83c246999b005ba42738cdbf3b585d5c212ab9726b2cdf4dc9b67f2e93df4390",
  "log_index": 10,
  "block_timestamp": "2023-10-29 03:34:19.000000",
  "block_number": 435530,
  "block_hash": "\\x68189ad6461faff3837f22571813eb507de3bdca2d1c9b596f233bd1fc8ff810"
}
```

```csv
\\xf417f5a458ec102b90352f697d6e2ac3a3d2851f,\\xdc5aabac622b5f472ba377c62522e97836a37a4f,\\xbad4ccc91ef0dfffbcab1402c519601fbaf244ef,10872268,\\x83c246999b005ba42738cdbf3b585d5c212ab9726b2cdf4dc9b67f2e93df4390,10,2023-10-29 03:34:19.000000,435530,\\x68189ad6461faff3837f22571813eb507de3bdca2d1c9b596f233bd1fc8ff810
```


# Tokens

Tokens are digital assets that can be created and managed on blockchain platforms like Ethereum. They represent a wide range of assets, from cryptocurrencies to digital collectibles. Currently, tokens are primarily categorized into three main types:

1. ERC20: The most common type of token, used for cryptocurrencies and utility tokens.
2. ERC721: Non-fungible tokens (NFTs) that represent unique digital assets.
3. ERC1155: Multi-token standard that can represent both fungible and non-fungible tokens.

ERC721 and ERC1155 are commonly used for NFTs (Non-Fungible Tokens), which represent unique digital assets such as art, collectibles, or in-game items.

## ETL data class

| Column Name   | Data Type | Note                      |
| ------------- | --------- | ------------------------- |
| address       | str       |                           |
| token\_type   | str       | ERC20, ERC721, or ERC1155 |
| name          | str       |                           |
| symbol        | str       |                           |
| decimals      | int       |                           |
| block\_number | int       |                           |
| total\_supply | int       |                           |

Examples

```json

{
  "address": "0x0DC808ADCE2099A9F62AA87D9670745ABA741746",
  "token_type": "ERC20",
  "name": "Wrapped Ether",
  "symbol": "WETH",
  "decimals": 18,
  "block_number": 19025325,
  "total_supply": 5133665930482359665722
}

```

```json
  {
  "address": "0xd5cb4a5009e9d9cdd58ef093182729d78c5d5a6e",
  "token_type": "ERC721",
  "name": "Manta Fest",
  "symbol": "Manta",
  "decimals": null,
  "block_number": 1153515,
  "total_supply": 115591
}

```

```csv
0x0DC808ADCE2099A9F62AA87D9670745ABA741746,ERC20,Wrapped Ether,WETH,18,19025325,5133665930482359665722
0xd5cb4a5009e9d9cdd58ef093182729d78c5d5a6e,ERC721,Manta Fest,Manta,,1153515,115591
```

## Model in the Database

| Column Name            | Data Type      | Note                      |
| ---------------------- | -------------- | ------------------------- |
| address                | bytea          | primary key               |
| name                   | varchar        |                           |
| symbol                 | varchar        |                           |
| total\_supply          | numeric(100)   |                           |
| decimals               | numeric(100)   |                           |
| token\_type            | varchar        | ERC20, ERC721, or ERC1155 |
| holder\_count          | integer        |                           |
| transfer\_count        | integer        |                           |
| icon\_url              | varchar        |                           |
| urls                   | jsonb          |                           |
| volume\_24h            | numeric(38, 2) |                           |
| price                  | numeric(38, 6) |                           |
| previous\_price        | numeric(38, 6) |                           |
| market\_cap            | numeric(38, 2) |                           |
| on\_chain\_market\_cap | numeric(38, 2) |                           |
| is\_verified           | boolean        |                           |
| cmc\_id                | integer        | CoinMarketCap ID          |
| cmc\_slug              | varchar        | CoinMarketCap slug        |
| gecko\_id              | varchar        | CoinGecko ID              |
| description            | varchar        |                           |
| create\_time           | timestamp      | default now()             |
| update\_time           | timestamp      | default now()             |
| block\_number          | bigint         |                           |

Examples

```json
{
  "address": "E'\\\\x0DC808ADCE2099A9F62AA87D9670745ABA741746'",
  "name": "Wrapped Ether",
  "symbol": "WETH",
  "total_supply": 5133665930482359665722,
  "decimals": 18,
  "token_type": "ERC20",
  "holder_count": 32961,
  "transfer_count": 1161654,
  "icon_url": "https://s2.coinmarketcap.com/static/img/coins/64x64/2396.png",
  "urls": {
    "website": [
      "https://weth.io/"
    ],
    "explorer": [
      "https://etherscan.io/token/0xc02aaa39b223fe8d0a0e5c4f27ead9083c756cc2"
    ]
  },
  "volume_24h": 1487479300.00,
  "price": 2274.550500,
  "previous_price": 2274.550500,
  "market_cap": 0.00,
  "on_chain_market_cap": 11676782.41,
  "is_verified": true,
  "cmc_id": 2396,
  "cmc_slug": "weth",
  "gecko_id": "weth",
  "description": "WETH (WETH) is a cryptocurrency and operates on the Ethereum platform. WETH is a wrapped version of Ether, allowing for easier integration with DeFi protocols.",
  "create_time": "2024-01-04 15:41:00.000000",
  "update_time": "2024-01-04 15:41:00.000000",
  "block_number": 19025325
}
```

```json
{
  "address": "E'\\\\xD5CB4A5009E9D9CDD58EF093182729D78C5D5A6E'",
  "name": "Manta Fest",
  "symbol": "Manta",
  "total_supply": 115591,
  "decimals": null,
  "token_type": "ERC721",
  "holder_count": 53660,
  "transfer_count": 116398,
  "icon_url": null,
  "urls": null,
  "volume_24h": null,
  "price": null,
  "previous_price": null,
  "market_cap": null,
  "on_chain_market_cap": null,
  "is_verified": null,
  "cmc_id": null,
  "cmc_slug": null,
  "gecko_id": null,
  "description": null,
  "create_time": "2024-01-20 05:58:29.000000",
  "update_time": "2024-01-20 05:58:29.000000",
  "block_number": 1153515
}
```

```csv
0xD5CB4A5009E9D9CDD58EF093182729D78C5D5A6E,Manta Fest,Manta,115591,,ERC721,53660,116398,,,,,,,,,,,,,,2024-01-20 05:58:29.000000,2024-01-20 05:58:29.000000,1153515
0x0DC808ADCE2099A9F62AA87D9670745ABA741746,Wrapped Ether,WETH,5133665930482359665722,18,ERC20,32961,1161654,https://s2.coinmarketcap.com/static/img/coins/64x64/2396.png,"{""website"": [""https://weth.io/""], ""explorer"": [""https://etherscan.io/token/0xc02aaa39b223fe8d0a0e5c4f27ead9083c756cc2""]}",1487479300.00,2274.550500,2274.550500,0.00,11676782.41,true,2396,weth,weth,"WETH (WETH) is a cryptocurrency and operates on the Ethereum platform. WETH is a wrapped version of Ether, allowing for easier integration with DeFi protocols.",2024-01-04 15:41:00.000000,2024-01-04 15:41:00.000000,19025325
```


# ERC20 Token Holders

ERC20 Token Holders table records the balance of each token holder at a given block. Each row represents a holder's balance for a specific token, including the token address, wallet address, balance (balance\_of), and the associated block details.

## ETL data class

| Column Name       | Data Type | Note |
| ----------------- | --------- | ---- |
| token\_address    | str       |      |
| wallet\_address   | str       |      |
| balance\_of       | int       |      |
| transaction\_hash | str       |      |
| block\_timestamp  | timestamp |      |
| block\_number     | int       |      |
| block\_hash       | str       |      |

Examples

```json
{
  "token_address": "0x80137510979822322193fc997d400d5a6c747bf7",
  "wallet_address": "0xcaab1bd9025a576cee4757506d81b32e4724f9ef",
  "balance_of": 9834981273804,
  "transaction_hash": null,
  "block_timestamp": "2023-11-16T01:06:59.000Z",
  "block_number": 590166,
  "block_hash": null
}
```

```csv
0x80137510979822322193fc997d400d5a6c747bf7,0xcaab1bd9025a576cee4757506d81b32e4724f9ef,9834981273804,,,2023-11-16T01:06:59.000Z,590166,
```

## Model in the Database

| Column Name       | Data Type    | Note |
| ----------------- | ------------ | ---- |
| token\_address    | bytea        |      |
| wallet\_address   | bytea        |      |
| balance\_of       | numeric(100) |      |
| transaction\_hash | bytea        |      |
| block\_timestamp  | timestamp    |      |
| block\_number     | bigint       |      |
| block\_hash       | bytea        |      |

```json
{
  "token_address": "\\x80137510979822322193fc997d400d5a6c747bf7",
  "wallet_address": "\\xcaab1bd9025a576cee4757506d81b32e4724f9ef",
  "balance_of": 9834981273804,
  "transaction_hash": null,
  "block_timestamp": "2023-11-16 01:06:59.000000",
  "block_number": 590166,
  "block_hash": null
}
```

```csv
\\x80137510979822322193fc997d400d5a6c747bf7,\\xcaab1bd9025a576cee4757506d81b32e4724f9ef,9834981273804,,,2023-11-16 01:06:59.000000,590166,
```


# ERC721 Token Transfers

ERC721 Token Transfers table records all the transfer events of ERC721 tokens. Each row represents a single token transfer, including the token address, token ID, sender (from\_address), recipient (to\_address) and the transaction and block details associated with the transfer.

## ETL data class

| Column Name       | Data Type | Note |
| ----------------- | --------- | ---- |
| token\_address    | str       |      |
| token\_id         | int       |      |
| from\_address     | str       |      |
| to\_address       | str       |      |
| transaction\_hash | str       |      |
| log\_index        | int       |      |
| block\_timestamp  | timestamp |      |
| block\_number     | int       |      |
| block\_hash       | str       |      |

Examples

```json
{
  "token_address": "0x5dac7efe0b4cd2cabdd350efc0c69bcaa81e76c6",
  "token_id": 7228,
  "from_address": "0x0000000000000000000000000000000000000000",
  "to_address": "0xfbabacd8ef4deefa572cdc1a685781c2b551490e",
  "transaction_hash": "0xde75fa94443a7e3035a0ad4ed1bfb762aa9ec1cdf283cf86878246f0911e5a0d",
  "log_index": 3,
  "block_timestamp": "2023-10-29T03:34:19.000Z",
  "block_number": 435530,
  "block_hash": "0x68189ad6461faff3837f22571813eb507de3bdca2d1c9b596f233bd1fc8ff810"
}
```

```csv
0x5dac7efe0b4cd2cabdd350efc0c69bcaa81e76c6,7228,0x0000000000000000000000000000000000000000,0xfbabacd8ef4deefa572cdc1a685781c2b551490e,0xde75fa94443a7e3035a0ad4ed1bfb762aa9ec1cdf283cf86878246f0911e5a0d,3,2023-10-29T03:34:19.000Z,435530,0x68189ad6461faff3837f22571813eb507de3bdca2d1c9b596f233bd1fc8ff810,
```

## Model in the Database

| Column Name       | Data Type   | Note |
| ----------------- | ----------- | ---- |
| token\_address    | bytea       |      |
| token\_id         | numeric(78) |      |
| from\_address     | bytea       |      |
| to\_address       | bytea       |      |
| transaction\_hash | bytea       |      |
| log\_index        | bigint      |      |
| block\_timestamp  | timestamp   |      |
| block\_number     | bigint      |      |
| block\_hash       | bytea       |      |

Examples

```json
{
  "token_address": "\\x5dac7efe0b4cd2cabdd350efc0c69bcaa81e76c6",
  "token_id": 7228,
  "from_address": "\\x0000000000000000000000000000000000000000",
  "to_address": "\\xfbabacd8ef4deefa572cdc1a685781c2b551490e",
  "transaction_hash": "\\xde75fa94443a7e3035a0ad4ed1bfb762aa9ec1cdf283cf86878246f0911e5a0d",
  "log_index": 3,
  "block_timestamp": "2023-10-29 03:34:19.000000",
  "block_number": 435530,
  "block_hash": "\\x68189ad6461faff3837f22571813eb507de3bdca2d1c9b596f233bd1fc8ff810",
  "token_uri": null
}
```

```csv
\\x5dac7efe0b4cd2cabdd350efc0c69bcaa81e76c6,7228,\\x0000000000000000000000000000000000000000,\\xfbabacd8ef4deefa572cdc1a685781c2b551490e,\\xde75fa94443a7e3035a0ad4ed1bfb762aa9ec1cdf283cf86878246f0911e5a0d,3,2023-10-29 03:34:19.000000,435530,\\x68189ad6461faff3837f22571813eb507de3bdca2d1c9b596f233bd1fc8ff810,
```


# ERC721 Token Holders

ERC721 Token Holders table records the balance of each token holder at a given block. Each row represents a holder's balance for a specific ERC721 token, including the token address, wallet address, balance (balance\_of), and the associated block details.

## ETL data class

| Column Name       | Data Type | Note |
| ----------------- | --------- | ---- |
| token\_address    | str       |      |
| wallet\_address   | str       |      |
| balance\_of       | int       |      |
| transaction\_hash | str       |      |
| block\_timestamp  | timestamp |      |
| block\_number     | int       |      |
| block\_hash       | str       |      |

Examples

```json
{
  "token_address": "0xfb3fdf272cf5bc4ca115e6d8767e99b70a767da7",
  "wallet_address": "0xa096e581472070ebca6e5d28df4e981ed3d9c5a1",
  "balance_of": 100,
  "transaction_hash": null,
  "block_timestamp": "2023-10-29T03:38:19.000Z",
  "block_number": 435554,
  "block_hash": null
}
```

```csv
0xfb3fdf272cf5bc4ca115e6d8767e99b70a767da7,0xa096e581472070ebca6e5d28df4e981ed3d9c5a1,100,,,2023-10-29T03:38:19.000Z,435554,
```

## Model in the Database

| Column Name       | Data Type    | Note |
| ----------------- | ------------ | ---- |
| token\_address    | bytea        |      |
| wallet\_address   | bytea        |      |
| balance\_of       | numeric(100) |      |
| transaction\_hash | bytea        |      |
| block\_timestamp  | timestamp    |      |
| block\_number     | bigint       |      |
| block\_hash       | bytea        |      |

```json
{
  "token_address": "\\xfb3fdf272cf5bc4ca115e6d8767e99b70a767da7",
  "wallet_address": "\\xa096e581472070ebca6e5d28df4e981ed3d9c5a1",
  "balance_of": 100,
  "transaction_hash": null,
  "block_timestamp": "2023-10-29 03:38:19.000000",
  "block_number": 435554,
  "block_hash": null
}
```

```csv
\\xfb3fdf272cf5bc4ca115e6d8767e99b70a767da7,\\xa096e581472070ebca6e5d28df4e981ed3d9c5a1,100,,,2023-10-29 03:38:19.000000,435554,
```


# ERC1155 Token Transfers


# ERC1155 Token Holders


# Address Coin Balances

Address Coin Balances table records the history of native token balance changes for each address. This table is updated whenever there is a balance change.

## ETL data class

| Column Name      | Data Type | Note |
| ---------------- | --------- | ---- |
| address          | str       |      |
| block\_number    | int       |      |
| balance          | int       |      |
| block\_timestamp | timestamp |      |

Examples

```json
{
  "address": "0x4200000000000000000000000000000000000011",
  "block_number": 664,
  "balance": 289870500000000,
  "block_timestamp": "2023-09-08T19:36:39.000Z"
}
```

```csv
0x4200000000000000000000000000000000000011,664,289870500000000,2023-09-08T19:36:39.000Z
```

## Model in the Database

| Column Name      | Data Type    | Note        |
| ---------------- | ------------ | ----------- |
| address          | bytea        | primary key |
| block\_number    | bigint       |             |
| balance          | numeric(100) |             |
| block\_timestamp | timestamp    |             |

Examples

```json
{
  "address": "E'\\\\x4200000000000000000000000000000000000011'",
  "block_number": 664,
  "balance": 289870500000000,
  "block_timestamp": "2023-09-08 19:36:39.000000"
}
```

```csv
\\x4200000000000000000000000000000000000011,664,289870500000000,2023-09-08 19:36:39.000000
```


# Address Token Balances

History of token balance change for each address - token\_address pair. This table is updated whenever there is a balance change.

## ETL data class

| Column Name      | Data Type | Note |
| ---------------- | --------- | ---- |
| address          | str       |      |
| block\_number    | int       |      |
| token\_address   | str       |      |
| token\_type      | str       |      |
| token\_id        | int       |      |
| token\_balance   | int       |      |
| block\_timestamp | timestamp |      |

Examples

```json
{
  "address": "0x66c0232d3c5f0c25914785606e73bd41b788b9ac",
  "block_number": 559924,
  "token_address": "0x0dc808adce2099a9f62aa87d9670745aba741746",
  "token_type": "ERC20",
  "token_id": -1,
  "token_balance": 32644185039418274000,
  "block_timestamp": "2023-11-12T13:06:39.000Z"
}

```

```csv
0x66c0232d3c5f0c25914785606e73bd41b788b9ac,559924,0x0dc808adce2099a9f62aa87d9670745aba741746,ERC20,-1,32644185039418274000,2023-11-12T13:06:39.000Z
```

## Model in the Database

| Column Name      | Data Type    | Note |
| ---------------- | ------------ | ---- |
| address          | bytea        |      |
| block\_number    | bigint       |      |
| token\_address   | bytea        |      |
| token\_type      | varchar      |      |
| token\_id        | numeric(78)  |      |
| token\_balance   | numeric(100) |      |
| block\_timestamp | timestamp    |      |

Examples

```json
{
  "address": "E'\\\\x66c0232d3c5f0c25914785606e73bd41b788b9ac'",
  "block_number": 559924,
  "token_address": "E'\\\\x0dc808adce2099a9f62aa87d9670745aba741746'",
  "token_type": "ERC20",
  "token_id": -1,
  "token_balance": 32644185039418274000,
  "block_timestamp": "2023-11-12 13:06:39.000000"
}
```

```
\\x66c0232d3c5f0c25914785606e73bd41b788b9ac,559924,\\x0dc808adce2099a9f62aa87d9670745aba741746,ERC20,-1,32644185039418274000,2023-11-12 13:06:39.000000
```


# Address Current Token Balances


# Daily Wallet Address Stats

The Daily Wallet Address Stats table provides daily statistics for each wallet address. This table is updated on a daily basis and includes information about the number and value of transactions (incoming, outgoing, and self), internal transactions, and token transfers (ERC20, ERC721, and ERC1155) associated with each address.

| Column Name                 | Data Type | Note      |
| --------------------------- | --------- | --------- |
| address                     | str       | length 42 |
| block\_date                 | date      |           |
| txn\_in\_cnt                | int       |           |
| txn\_out\_cnt               | int       |           |
| txn\_self\_cnt              | int       |           |
| txn\_in\_error\_cnt         | int       |           |
| txn\_out\_error\_cnt        | int       |           |
| txn\_self\_error\_cnt       | int       |           |
| txn\_in\_value              | int       |           |
| txn\_out\_value             | int       |           |
| internal\_txn\_in\_cnt      | int       |           |
| internal\_txn\_out\_cnt     | int       |           |
| internal\_txn\_in\_value    | int       |           |
| internal\_txn\_out\_value   | int       |           |
| erc20\_transfer\_in\_cnt    | int       |           |
| erc721\_transfer\_in\_cnt   | int       |           |
| erc1155\_transfer\_in\_cnt  | int       |           |
| erc20\_transfer\_out\_cnt   | int       |           |
| erc721\_transfer\_out\_cnt  | int       |           |
| erc1155\_transfer\_out\_cnt | int       |           |
| internal\_txn\_cnt          | int       |           |
| erc20\_transfer\_cnt        | int       |           |
| erc721\_transfer\_cnt       | int       |           |
| erc1155\_transfer\_cnt      | int       |           |
| txn\_cnt                    | int       |           |

Examples

```json
{
  "address": "0x2821023289920dd815ca1f5adb0a33328afb3bfe",
  "block_date": "2023-10-25",
  "txn_in_cnt": 4,
  "txn_out_cnt": 44,
  "txn_self_cnt": 0,
  "txn_in_error_cnt": 0,
  "txn_out_error_cnt": 1,
  "txn_self_error_cnt": 0,
  "txn_in_value": 45655000000001070,
  "txn_out_value": 49278943858313640,
  "internal_txn_in_cnt": 0,
  "internal_txn_out_cnt": 0,
  "internal_txn_in_value": null,
  "internal_txn_out_value": null,
  "erc20_transfer_in_cnt": 13,
  "erc721_transfer_in_cnt": 13,
  "erc1155_transfer_in_cnt": 3,
  "erc20_transfer_out_cnt": 9,
  "erc721_transfer_out_cnt": 1,
  "erc1155_transfer_out_cnt": 1,
  "internal_txn_cnt": 0,
  "erc20_transfer_cnt": 22,
  "erc721_transfer_cnt": 14,
  "erc1155_transfer_cnt": 4,
  "txn_cnt": 48
}
```

```csv
0x2821023289920dd815ca1f5adb0a33328afb3bfe,2023-10-25,4,44,0,0,1,0,45655000000001070,49278943858313640,0,0,,,13,13,3,9,1,1,0,22,14,4,48
```

## Model in the Database

| Column Name                 | Data Type   | Note |
| --------------------------- | ----------- | ---- |
| address                     | bytea       |      |
| block\_date                 | date        |      |
| txn\_in\_cnt                | integer     |      |
| txn\_out\_cnt               | integer     |      |
| txn\_self\_cnt              | integer     |      |
| txn\_in\_error\_cnt         | integer     |      |
| txn\_out\_error\_cnt        | integer     |      |
| txn\_self\_error\_cnt       | integer     |      |
| txn\_in\_value              | numeric(78) |      |
| txn\_out\_value             | numeric(78) |      |
| internal\_txn\_in\_cnt      | integer     |      |
| internal\_txn\_out\_cnt     | integer     |      |
| internal\_txn\_in\_value    | numeric(78) |      |
| internal\_txn\_out\_value   | numeric(78) |      |
| erc20\_transfer\_in\_cnt    | integer     |      |
| erc721\_transfer\_in\_cnt   | integer     |      |
| erc1155\_transfer\_in\_cnt  | integer     |      |
| erc20\_transfer\_out\_cnt   | integer     |      |
| erc721\_transfer\_out\_cnt  | integer     |      |
| erc1155\_transfer\_out\_cnt | integer     |      |
| internal\_txn\_cnt          | integer     |      |
| erc20\_transfer\_cnt        | integer     |      |
| erc721\_transfer\_cnt       | integer     |      |
| erc1155\_transfer\_cnt      | integer     |      |
| txn\_cnt                    | integer     |      |

```json
{
  "address": "\\x2821023289920dd815ca1f5adb0a33328afb3bfe",
  "block_date": "2023-10-25",
  "txn_in_cnt": 4,
  "txn_out_cnt": 44,
  "txn_self_cnt": 0,
  "txn_in_error_cnt": 0,
  "txn_out_error_cnt": 1,
  "txn_self_error_cnt": 0,
  "txn_in_value": 45655000000001070,
  "txn_out_value": 49278943858313640,
  "internal_txn_in_cnt": 0,
  "internal_txn_out_cnt": 0,
  "internal_txn_in_value": null,
  "internal_txn_out_value": null,
  "erc20_transfer_in_cnt": 13,
  "erc721_transfer_in_cnt": 13,
  "erc1155_transfer_in_cnt": 3,
  "erc20_transfer_out_cnt": 9,
  "erc721_transfer_out_cnt": 1,
  "erc1155_transfer_out_cnt": 1,
  "internal_txn_cnt": 0,
  "erc20_transfer_cnt": 22,
  "erc721_transfer_cnt": 14,
  "erc1155_transfer_cnt": 4,
  "txn_cnt": 48
}

```

```csv
\\x2821023289920dd815ca1f5adb0a33328afb3bfe,2023-10-25,4,44,0,0,1,0,45655000000001070,49278943858313640,0,0,,,13,13,3,9,1,1,0,22,14,4,48
```


# Contracts

Contracts table records all the smart contracts deployed on the blockchain. Each row represents a unique contract and contains information about its address, creator, creation code, deployed code, and the transaction and block details associated with its deployment.

## ETL data class

| Column Name                | Data Type | Note |
| -------------------------- | --------- | ---- |
| address                    | str       |      |
| name                       | str       |      |
| contract\_creator          | str       |      |
| creation\_code             | str       |      |
| deployed\_code             | str       |      |
| block\_number              | int       |      |
| block\_hash                | str       |      |
| block\_timestamp           | timestamp |      |
| transaction\_index         | int       |      |
| transaction\_hash          | str       |      |
| transaction\_from\_address | str       |      |

Examples

```json

{
  "address": "0x7c6ba614a28d505ef254cb888db872763d6f5b85",
  "name": null,
  "contract_creator": "0xb3b41365f9179ac0711727ade3e0879c0aaa273b",
  "creation_code": null,
  "deployed_code": null,
  "block_number": 849291,
  "block_hash": "0xbd334c7a4dcd83d6f85b7cacef214433131cc8098f5014b050b15336fd00d535",
  "block_timestamp": "2023-11-13T09:39:42.000Z",
  "transaction_index": 0,
  "transaction_hash": "0x0a16c572c35fc45e65e8daf28c3d58358566d5b69feb0557272812e22a75e7cc",
  "transaction_from_address": "0xb3b41365f9179ac0711727ade3e0879c0aaa273b"
}
```

```csv
0x7c6ba614a28d505ef254cb888db872763d6f5b85,,0xb3b41365f9179ac0711727ade3e0879c0aaa273b,,,849291,0xbd334c7a4dcd83d6f85b7cacef214433131cc8098f5014b050b15336fd00d535,2023-11-13T09:39:42.000Z,0,0x0a16c572c35fc45e65e8daf28c3d58358566d5b69feb0557272812e22a75e7cc,0xb3b41365f9179ac0711727ade3e0879c0aaa273b
```

## Model in the Database

| Column Name                | Data Type | Note          |
| -------------------------- | --------- | ------------- |
| address                    | bytea     | primary key   |
| name                       | varchar   |               |
| contract\_creator          | bytea     |               |
| creation\_code             | bytea     |               |
| deployed\_code             | bytea     |               |
| block\_number              | bigint    |               |
| block\_hash                | bytea     |               |
| block\_timestamp           | timestamp |               |
| transaction\_index         | integer   |               |
| transaction\_hash          | bytea     |               |
| transaction\_from\_address | bytea     |               |
| create\_time               | timestamp | default now() |
| update\_time               | timestamp | default now() |

```csv
\\x7c6ba614a28d505ef254cb888db872763d6f5b85,,\\xb3b41365f9179ac0711727ade3e0879c0aaa273b,,,849291,\\xbd334c7a4dcd83d6f85b7cacef214433131cc8098f5014b050b15336fd00d535,2023-11-13 09:39:42.000000,0,\\x0a16c572c35fc45e65e8daf28c3d58358566d5b69feb0557272812e22a75e7cc,\\xb3b41365f9179ac0711727ade3e0879c0aaa273b,2023-11-13 09:39:42.000000,2023-11-13 09:39:42.000000
```


# Other Tables


# Inscriptions


# Bridges


# L1 to L2 Transactions


# L2 to L1 Transactions


# Optimistic Rollup Data Availability Batches


# Optimistic Rollup State Batches


# Use Cases


# UniSwap V3

UniSwap Protocol is A suite of persistent, non-upgradable smart contracts that together create an automated market maker, a protocol that facilitates peer-to-peer market making and swapping of ERC-20 tokens on the Ethereum blockchain.

Uniswap v3 offers a more efficient and customizable trading experience compared to previous versions. It introduces concentrated liquidity, allowing liquidity providers to allocate capital within specific price ranges, improving capital efficiency and reducing slippage for traders.

Key Features:

\#of trades (in total / on each \[Protocol Name]) #of traded assets (all ecr20 tokens) Trading volume (in total / on each \[Protocol Name]) Average trade size (in total / on each \[Protocol Name]) Times of providing liquidity in \[Pool Name] LP value for \[Pool Name] Overall trading PnL in USD value (Coming Soon) LP PnL for \[Pool Name] in USD value (Coming Soon)


# Data Class

## UniswapV3Pool

| Attribute                | Description                                                           |
| ------------------------ | --------------------------------------------------------------------- |
| position\_token\_address | Address of the ERC721 token contract representing liquidity positions |
| factory\_address         | Address of the Uniswap V3 factory contract that created this pool     |
| pool\_address            | Unique address of this specific liquidity pool                        |
| token0\_address          | Contract address of the first token in the pair                       |
| token1\_address          | Contract address of the second token in the pair                      |
| fee                      | The fee tier of the pool                                              |
| tick\_spacing            | The spacing between ticks for this pool                               |
| block\_number            | The block number when this pool was created or last updated           |
| block\_timestamp         | The timestamp of the block when this pool was created or last updated |

## UniswapV3Token

| Attribute                | Description                                                               |
| ------------------------ | ------------------------------------------------------------------------- |
| position\_token\_address | Address of the ERC721 token contract representing liquidity positions     |
| token\_id                | Unique identifier for this specific liquidity position (NFT)              |
| pool\_address            | Address of the pool this position is in                                   |
| tick\_lower              | The lower tick of the price range for this position                       |
| tick\_upper              | The upper tick of the price range for this position                       |
| fee                      | The fee tier of the pool this position is in                              |
| block\_number            | The block number when this position was created or last updated           |
| block\_timestamp         | The timestamp of the block when this position was created or last updated |

## UniswapV3PoolPrice

| Attribute        | Description                                                                     |
| ---------------- | ------------------------------------------------------------------------------- |
| factory\_address | Address of the Uniswap V3 factory contract                                      |
| pool\_address    | Address of the specific pool                                                    |
| sqrt\_price\_x96 | The square root of the price ratio of the two tokens, encoded as a Q64.96 value |
| tick             | The current tick of the pool                                                    |
| block\_number    | The block number when this price update occurred                                |
| block\_timestamp | The timestamp of the block when this price update occurred                      |

## UniswapV3TokenDetail

| Attribute                | Description                                                           |
| ------------------------ | --------------------------------------------------------------------- |
| position\_token\_address | Address of the ERC721 token contract representing liquidity positions |
| token\_id                | Unique identifier for this specific liquidity position (NFT)          |
| pool\_address            | Address of the pool this position is in                               |
| wallet\_address          | Address of the wallet that owns this position                         |
| liquidity                | The amount of liquidity in this position                              |
| block\_number            | The block number when this detail was recorded or updated             |
| block\_timestamp         | The timestamp of the block when this detail was recorded or updated   |

## UniswapV3PoolCurrentPrice

| Attribute        | Description                                                                             |
| ---------------- | --------------------------------------------------------------------------------------- |
| factory\_address | Address of the Uniswap V3 factory contract                                              |
| pool\_address    | Address of the specific pool                                                            |
| sqrt\_price\_x96 | The current square root of the price ratio of the two tokens, encoded as a Q64.96 value |
| tick             | The current tick of the pool                                                            |
| block\_number    | The most recent block number for this price information                                 |
| block\_timestamp | The timestamp of the most recent block for this price information                       |

## UniswapV3SwapEvent

| Attribute                  | Description                                                                        |
| -------------------------- | ---------------------------------------------------------------------------------- |
| pool\_address              | Address of the pool where the swap occurred                                        |
| position\_token\_address   | Address of the ERC721 token contract representing liquidity positions              |
| transaction\_from\_address | Address that initiated the transaction containing this swap                        |
| sender                     | Address that sent the tokens to be swapped                                         |
| recipient                  | Address that received the output tokens from the swap                              |
| amount0                    | The amount of token0 involved in the swap (positive if received, negative if paid) |
| amount1                    | The amount of token1 involved in the swap (positive if received, negative if paid) |
| liquidity                  | The pool's liquidity after the swap                                                |
| tick                       | The pool's tick after the swap                                                     |
| sqrt\_price\_x96           | The pool's price after the swap, encoded as a Q64.96 value                         |
| token0\_address            | Contract address of token0 in the pool                                             |
| token1\_address            | Contract address of token1 in the pool                                             |
| transaction\_hash          | Hash of the transaction containing this swap event                                 |
| log\_index                 | Index of this event in the transaction logs                                        |
| block\_number              | Block number in which this swap occurred                                           |
| block\_timestamp           | Timestamp of the block in which this swap occurred                                 |

## UniswapV3TokenCurrentStatus

| Attribute                | Description                                                           |
| ------------------------ | --------------------------------------------------------------------- |
| position\_token\_address | Address of the ERC721 token contract representing liquidity positions |
| token\_id                | Unique identifier for this specific liquidity position (NFT)          |
| pool\_address            | Address of the pool this position is in                               |
| wallet\_address          | Current owner's wallet address for this position                      |
| liquidity                | Current amount of liquidity in this position                          |
| block\_number            | Most recent block number for this status update                       |
| block\_timestamp         | Timestamp of the most recent block for this status update             |

## UniswapV3TokenUpdateLiquidity

| Attribute                | Description                                                             |
| ------------------------ | ----------------------------------------------------------------------- |
| position\_token\_address | Address of the ERC721 token contract representing liquidity positions   |
| token\_id                | Unique identifier for the liquidity position (NFT) being updated        |
| owner                    | Address of the wallet owning this position                              |
| liquidity                | Amount of liquidity added or removed                                    |
| amount0                  | Amount of token0 added or removed                                       |
| amount1                  | Amount of token1 added or removed                                       |
| action\_type             | Type of action (e.g., "mint" for adding liquidity, "burn" for removing) |
| transaction\_hash        | Hash of the transaction containing this liquidity update                |
| pool\_address            | Address of the pool where liquidity was updated                         |
| token0\_address          | Contract address of token0 in the pool                                  |
| token1\_address          | Contract address of token1 in the pool                                  |
| log\_index               | Index of this event in the transaction logs                             |
| block\_number            | Block number in which this liquidity update occurred                    |
| block\_timestamp         | Timestamp of the block in which this liquidity update occurred          |

## UniswapV3TokenCollectFee

| Attribute                | Description                                                           |
| ------------------------ | --------------------------------------------------------------------- |
| position\_token\_address | Address of the ERC721 token contract representing liquidity positions |
| recipient                | Address receiving the collected fees                                  |
| owner                    | Address of the wallet owning the position                             |
| token\_id                | Unique identifier for the liquidity position (NFT) collecting fees    |
| amount0                  | Amount of token0 collected as fees                                    |
| amount1                  | Amount of token1 collected as fees                                    |
| pool\_address            | Address of the pool from which fees were collected                    |
| token0\_address          | Contract address of token0 in the pool                                |
| token1\_address          | Contract address of token1 in the pool                                |
| transaction\_hash        | Hash of the transaction in which fees were collected                  |
| log\_index               | Index of this event in the transaction logs                           |
| block\_number            | Block number in which this fee collection occurred                    |
| block\_timestamp         | Timestamp of the block in which this fee collection occurred          |


# Trigger and Function

### UniSwapV3PoolJob

#### Filter Logic

```python
def get_filter(self):
    return TransactionFilterByLogs(
        [
            TopicSpecification(addresses=[self._factory_address], topics=[self._create_pool_topic0]),
            TopicSpecification(topics=self._pool_price_topic0_list),
        ]
    )
```

This job filters for logs from the factory address with the pool creation topic, and logs with topics indicating pool price changes.

#### Key Processing Steps

1. Collecting Pool Data:

```python
def _collect_pool_batch(self, logs):
    for log in logs:
        if self._factory_address == log.address and self._create_pool_topic0 == log.topic0:
            entity = decode_pool_created(self._nft_address, self._factory_address, log)
            self._collect_item(UniswapV3Pool.type(), entity)
```

2. Collecting Swap Events:

```python
if log.topic0 == constants.UNISWAP_V3_POOL_SWAP_TOPIC0:
    # Process and collect swap event
    self._collect_item(UniswapV3SwapEvent.type(), swap_event)
```

3. Updating Pool Prices:
4. call `slot0` function.

```python
pool_prices = slot0_rpc_requests(
            self._web3,
            self._batch_web3_provider.make_request,
            requests,
            self._is_batch,
            self._abi_list,
            self._batch_size,
            self._max_worker,
        )
for data in pool_prices:
    detail = UniswapV3PoolPrice(
        factory_address=self._factory_address,
        pool_address=data["pool_address"],
        sqrt_price_x96=data["sqrtPriceX96"],
        tick=data["tick"],
        block_number=data["block_number"],
        block_timestamp=data["block_timestamp"],
    )
    self._collect_item(UniswapV3PoolPrice.type(), detail)
```

### UniswapV3TokenJob

#### Filter Logic

```python
def get_filter(self):
    return TransactionFilterByLogs(
        [
            TopicSpecification(addresses=[self._nft_address]),
        ]
    )
```

This job filters for logs from the Uniswap V3 NFT contract address.

#### Key Processing Steps

1. Collecting Token Data:

Filter token ownership transfers and liquidity adjustments for each tokenId.

```python
for log in logs:
    if log.address == self._nft_address:
        if topic0 in [constants.TRANSFER_TOPIC0, constants.UNISWAP_V3_ADD_LIQUIDITY_TOPIC0, 
                      constants.UNISWAP_V3_REMOVE_LIQUIDITY_TOPIC0]:
            # Process and collect token data
```

2. Fetching Token Ownership and Position Data:

call `owner` and `positions`

```python
owner_info = owner_rpc_requests(...)
token_infos = positions_rpc_requests(...)
```

3. Collect Token Information:

```python
for data in token_infos:
    if token_id not in self._exist_token_ids:
        # Create new token
        self._collect_item(UniswapV3Token.type(), new_token)
    
    detail = UniswapV3TokenDetail(...)
    self._collect_item(UniswapV3TokenDetail.type(), detail)
    
    # Update current status
    token_id_current_status[token_id] = create_token_status(detail)
```

4. Collect Liquidity Updates and Fee Collection Events::

```python
if topic0 in [constants.UNISWAP_V3_REMOVE_LIQUIDITY_TOPIC0, 
              constants.UNISWAP_V3_ADD_LIQUIDITY_TOPIC0]:
    self._collect_item(UniswapV3TokenUpdateLiquidity.type(), liquidity_event)
elif topic0 == constants.UNISWAP_V3_TOKEN_COLLECT_FEE_TOPIC0:
    self._collect_item(UniswapV3TokenCollectFee.type(), fee_event)
```


# Run & Query

## 1. Configuration

In the `indexer/modules/custom/uniswap_v3/config.ini` file, configure the protocol information for your specific blockchain. Use the `chainId` as the key.

## 2. Verify ABI

Ensure that the ABI matches the Uniswap V3 protocol you're working with. If it doesn't, update the ABI in the `indexer/modules/custom/uniswap_v3/constants.py` file.

## 3. Indexing Data

To start indexing the data, run the following command:

```bash
python hemera.py stream -pg postgresql://{db_user}:{db_password}@{db_url}:5432/{database} -p {chain_rpc} -s {start_block_number} -e {end_block_number} -o postgres -B 1000 -w 5 -b 100 -E {entity_type}
```

(you can add your {entity\_type} in `enumeration/entity_type.py`)

## 4. Verify Data

### Get Current Holdings for a Wallet Address

First, retrieve all tokenIds held by the wallet\_address, and then fetch the prices for the pools associated with these tokenIds.

API Path: `/v1/aci/<wallet_address>/uniswapv3/current_holding`

### Get First Liquidity Addition Time for a Wallet Address

API Path: `/v1/aci/<wallet_address>/uniswapv3/first_liquidity_time`

### Get the Number of Pools with Liquidity Added or Removed by a Wallet Address

API Path: `/v1/aci/<wallet_address>/uniswapv3/provide_liquidity_pool_count`


# ENS

The Ethereum Name Service (ENS) is a decentralized naming system built on the Ethereum blockchain, allowing users to register and manage human-readable domain names (like `yourname.eth`) that can be linked to Ethereum addresses, smart contracts, and other decentralized resources. ENS simplifies the process of interacting with blockchain addresses and resources by replacing complex alphanumeric strings with easy-to-remember names, enhancing the user experience in the Ethereum ecosystem.

## Key Features

* Current primary name
* ENS holdings (All the ENS names holds by an Address)
* Resolved ENS names (All the ENS names resolved to the Address)
* First registration time
* First primary name set time


# Data Class

To cover ENS actions and features, we created the following data classes.

* `af_ens_event`: Records all events (actions) related to ENS. One event per row, one transaction can have multiple events.
* `af_ens_node_current`: Records an ENS node's (name) related information. One node per row, only keep current state.
* `af_ens_address_current`: Records an address's current ENS related data (primary name). One address per row, only keep current state.

Besides, since each ENS name is either an ERC721 or an ERC1155 token. The ENS job will also require data from generic data class `address_token_balance.`

#### `af_ens_event`

| Column Name          | Data Type    | Note                                                                 |
| -------------------- | ------------ | -------------------------------------------------------------------- |
| transaction\_hash    | bytea        | Hash of the transaction                                              |
| log\_index           | integer      | Index of the log in the transaction                                  |
| transaction\_index   | integer      | Index of the transaction within the block                            |
| block\_number        | bigint       | Number of the block containing the transaction                       |
| block\_hash          | bytea        | Hash of the block containing the transaction                         |
| block\_timestamp     | timestamp    | Timestamp of when the block was mined                                |
| method               | varchar      | Method name associated with the event                                |
| event\_name          | varchar      | Name of the event                                                    |
| from\_address        | bytea        | Address that initiated the transaction                               |
| to\_address          | bytea        | Address that received the transaction                                |
| base\_node           | bytea        | Base node of the ENS name                                            |
| node                 | bytea        | Node hash of the ENS name                                            |
| label                | bytea        | Label hash of the ENS name                                           |
| name                 | varchar      | Human-readable ENS name                                              |
| expires              | timestamp    | Expiry time of the ENS registration                                  |
| owner                | bytea        | Owner address of the ENS name                                        |
| resolver             | bytea        | Resolver address associated with the ENS name                        |
| registrant           | bytea        | Address of the registrant of the ENS name                            |
| address              | bytea        | Address associated with the ENS name (forward resolution)            |
| reverse\_base\_node  | bytea        | Base node for reverse resolution                                     |
| reverse\_node        | bytea        | Node for reverse resolution                                          |
| reverse\_label       | bytea        | Label hash for reverse resolution                                    |
| reverse\_name        | varchar      | Human-readable name for reverse resolution                           |
| token\_id            | numeric(100) | Token ID representing the ENS name in numeric format                 |
| w\_token\_id         | numeric(100) | Wrapped token ID, if applicable                                      |
| create\_time         | timestamp    | Time when the record was created, defaults to current timestamp      |
| update\_time         | timestamp    | Time when the record was last updated, defaults to current timestamp |
| reorg                | boolean      | Flag indicating if the record was affected by a chain reorganization |
| ens\_tnx\_log\_index | constraint   | Primary key on (transaction\_hash, log\_index)                       |

#### `af_ens_address_current`:

| Column Name   | Data Type | Note                                                                     |
| ------------- | --------- | ------------------------------------------------------------------------ |
| address       | bytea     | ENS address; primary key                                                 |
| name          | varchar   | Human-readable ENS name associated with the address                      |
| reverse\_node | bytea     | Node for reverse resolution                                              |
| create\_time  | timestamp | Time when the record was created, defaults to the current timestamp      |
| update\_time  | timestamp | Time when the record was last updated, defaults to the current timestamp |
| block\_number | bigint    | Block number when the address was last updated or registered             |

#### `af_ens_node_current`

| Column Name      | Data Type    | Note                                                                     |
| ---------------- | ------------ | ------------------------------------------------------------------------ |
| node             | bytea        | Node hash of the ENS name; primary key                                   |
| token\_id        | numeric(100) | ERC721 Token ID representing the ENS name in numeric format              |
| w\_token\_id     | numeric(100) | ERC1155 Wrapped token ID, if applicable                                  |
| first\_owned\_by | bytea        | Address of the first owner of the ENS name                               |
| name             | varchar      | Human-readable ENS name                                                  |
| registration     | timestamp    | Timestamp of when the ENS name was registered                            |
| expires          | timestamp    | Expiry time of the ENS registration                                      |
| address          | bytea        | Address associated with the ENS name                                     |
| create\_time     | timestamp    | Time when the record was created, defaults to the current timestamp      |
| update\_time     | timestamp    | Time when the record was last updated, defaults to the current timestamp |

##


# Trigger and Function

The ENS job is triggered when an ENS-related transaction occurs or an event is produced. Specifically:

* `NameRegistered`: When a user registers an ENS name

![Screenshot of ENS Data](/files/PMM3c8azVLB8xSfZ3E9g)

```python
def extract(self, address, tp0, log, ens_middle, contract_object_map, event_map, prev_logs=None) -> ENSMiddleD:
    if (tp0 == RegisterExtractor.tp0_register) or (tp0 == self.tp0a):
        event_data = decode_log(log, contract_object_map, event_map)
        tmp = event_data["args"]
        ens_middle.expires = convert_str_ts(tmp.get("expires", ""))
        ens_middle.name = tmp.get("name")
        if "." in ens_middle.name:
            # not supported
            return None
        ens_middle.name = ens_middle.name + ".eth"
        ens_middle.label = tmp.get("label").lower()
        ens_middle.owner = tmp.get("owner").lower()
        ens_middle.base_node = BASE_NODE
        ens_middle.node = namehash(ens_middle.name)
        ens_middle.event_name = event_data["_event"]
        token_id = None
        w_token_id = None
        for log in prev_logs:
            if (
                log["address"] == "0x57f1887a8bf19b14fc0df6fd9b2acc9af147ea85"
                and log["topic1"] == "0x0000000000000000000000000000000000000000000000000000000000000000"
            ):
                token_id = str(int(log["topic3"], 16))
            if (
                log["address"] == "0xd4416b13d2b3a9abae7acd5d6c2bbdbe25686401"
                and log["topic0"] == "0xc3d58168c5ae7397731d063d5bbf3d657854427343f4c083240f7aacaa2d0f62"
            ):
                evd = decode_log(log, contract_object_map, event_map)
                if evd["args"].get("id"):
                    w_token_id = str(evd["args"].get("id"))
        return ENSMiddleD(
            transaction_hash=ens_middle.transaction_hash,
            log_index=ens_middle.log_index,
            transaction_index=ens_middle.transaction_index,
            block_number=ens_middle.block_number,
            block_hash=ens_middle.block_hash,
            block_timestamp=ens_middle.block_timestamp,
            from_address=ens_middle.from_address,
            to_address=ens_middle.to_address,
            expires=ens_middle.expires,
            name=ens_middle.name,
            label=ens_middle.label,
            owner=ens_middle.owner,
            base_node=ens_middle.base_node,
            node=ens_middle.node,
            event_name=ens_middle.event_name,
            method=ens_middle.method,
            token_id=token_id,
            w_token_id=w_token_id,
        )
    else:
        return None
```

The above code performs the following tasks:

1. Extracts registration information from the log
2. Extracts token\_id or wrapped token\_id from prev\_logs

* `NameChanged`: When a user sets a primary name (In some cases, there will be no log, so the transaction input needs to be decoded)

![Screenshot of ENS Data](/files/a1HXJUdAHdrFxr6oEAXx)

```python
def extract(self, address, tp0, log, ens_middle, contract_object_map, event_map, prev_logs=None) -> ENSMiddleD:
    if tp0 == self.tp0:
        event_data = decode_log(log, contract_object_map, event_map)
        tmp = event_data["args"]
        name = tmp.get("name") or ""
        if not name or len(name) - 4 != name.find("."):
            return None
        ens_middle.reverse_name = name
        ens_middle.address = ens_middle.from_address
        ens_middle.node = namehash(name)
        ens_middle.reverse_base_node = REVERSE_BASE_NODE
        ens_middle.reverse_node = str(log["topic1"]).lower()
        ens_middle.event_name = event_data["_event"]
        return ENSMiddleD(
            transaction_hash=ens_middle.transaction_hash,
            log_index=ens_middle.log_index,
            transaction_index=ens_middle.transaction_index,
            block_number=ens_middle.block_number,
            block_hash=ens_middle.block_hash,
            block_timestamp=ens_middle.block_timestamp,
            from_address=ens_middle.from_address,
            to_address=ens_middle.to_address,
            reverse_name=ens_middle.reverse_name,
            address=ens_middle.address,
            node=ens_middle.node,
            reverse_node=ens_middle.reverse_node,
            reverse_base_node=REVERSE_BASE_NODE,
            event_name=ens_middle.event_name,
            method=ens_middle.method,
        )
```

The above code performs the following tasks:

1. Decodes the log
2. Retrieves the ENS name and address

Additionally, when the log does not exist, we need to decode the transaction input:

```python
if method == "setName":
    d_tnx = self.decode_transaction(transaction)
    ens_middle = AttrDict(dic)
    ens_middle.log_index = -1
    name = None
    if d_tnx[1].get("name"):
        name = d_tnx[1]["name"]
    elif d_tnx[1].get("newName"):
        name = d_tnx[1]["newName"]
    if not name or len(name) - 4 != name.find("."):
        return []
    ens_middle.reverse_name = name

    ens_middle.node = namehash(name)
    ens_middle.address = tra["from_address"].lower()
    return [
        ENSMiddleD(
            transaction_hash=ens_middle.transaction_hash,
            log_index=ens_middle.log_index,
            transaction_index=ens_middle.transaction_index,
            block_number=ens_middle.block_number,
            block_hash=ens_middle.block_hash,
            block_timestamp=ens_middle.block_timestamp,
            from_address=ens_middle.from_address,
            to_address=ens_middle.to_address,
            reverse_name=ens_middle.reverse_name,
            address=ens_middle.address,
            node=ens_middle.node,
            reverse_node=None,
            reverse_base_node=REVERSE_BASE_NODE,
            event_name=None,
            method="setName",
        )
    ]
```

* `AddressChanged`: When a user resolves an ENS name to an address. Processing is like the same

![Screenshot of ENS Data](/files/RtV8ubULDAKN9pJFUCod)

```python
def extract(self, address, tp0, log, ens_middle, contract_object_map, event_map, prev_logs=None) -> ENSMiddleD:
    if tp0 == self.tp0:
        event_data = decode_log(log, contract_object_map, event_map)
        tmp = event_data["args"]
        coin_type = tmp["coinType"]
        if not coin_type or str(coin_type) != "60":
            return None
        ens_middle.node = tmp["node"]
        ens_middle.address = tmp["newAddress"].lower()
        ens_middle.event_name = event_data["_event"]
        return ENSMiddleD(
            transaction_hash=ens_middle.transaction_hash,
            log_index=ens_middle.log_index,
            transaction_index=ens_middle.transaction_index,
            block_number=ens_middle.block_number,
            block_hash=ens_middle.block_hash,
            block_timestamp=ens_middle.block_timestamp,
            topic0=tp0,
            from_address=ens_middle.from_address,
            to_address=ens_middle.to_address,
            node=ens_middle.node,
            address=ens_middle.address,
            event_name=ens_middle.event_name,
            method=ens_middle.method,
        )

    return None
```

`newAddress` in the event is which address the ens point to.

* `NameRenewed`: When a user renews an ENS name. Processing is like the same

![Screenshot of ENS Data](/files/hjvzWwuMNWfomNo9CrxN)

```python
def extract(self, address, tp0, log, ens_middle, contract_object_map, event_map, prev_logs=None) -> ENSMiddleD:
    if tp0 == self.tp0:
        event_data = decode_log(log, contract_object_map, event_map)

        tmp = event_data["args"]
        name = tmp.get("name")
        if "." in name:
            return None
        name = name + '.eth'
        ens_middle.name = name
        ens_middle.node = namehash(name)
        ens_middle.label = tmp.get("label").lower()
        ens_middle.expires = convert_str_ts(tmp.get("expires", ""))
        ens_middle.event_name = event_data["_event"]

        return ENSMiddleD(
            transaction_hash=ens_middle.transaction_hash,
            log_index=ens_middle.log_index,
            transaction_index=ens_middle.transaction_index,
            block_number=ens_middle.block_number,
            block_hash=ens_middle.block_hash,
            block_timestamp=ens_middle.block_timestamp,
            topic0=tp0,
            from_address=ens_middle.from_address,
            to_address=ens_middle.to_address,
            name=ens_middle.name,
            node=ens_middle.node,
            label=ens_middle.label,
            expires=ens_middle.expires,
            event_name=ens_middle.event_name,
            method=ens_middle.method,
        )
```

`expires` is the new expiration date after renewal.


# Run & Query

## How to Run

You can start the export\_ens\_job by adding the following arguments:

```shell
-O ERC721TokenTransfer,ERC1155TokenTransfer,TokenBalance,CurrentTokenBalance,UpdateERC1155TokenIdDetail,ERC1155TokenIdDetail,UpdateERC721TokenIdDetail,ERC721TokenIdDetail,ERC721TokenIdChange,ENSMiddleD,ENSRegisterD,ENSNameRenewD,ENSAddressChangeD,ENSAddressD
--force-filter-mode true
```

## Public API

### GET /v1/aci/{address}/ens/current

Get currently address level ENS related features

```json
{
    "primary_name": "vitalik.eth",
    "be_resolved_by_ens": [
        "brianshaw.eth",
        "satoshinart.eth",
        "tornado-ui.eth",
        "zuzaluventure.eth",
        "dacc.eth",
        "buything.eth",
        "tipsforcoins.eth"
    ],
    "ens_holdings": [
        "vytalik.eth",
        "dacc.eth"
    ],
    "first_register_time": "2024-08-18 04:04:59"
}
```

### GET /v1/aci/{address}/ens/detail

Get ENS related actions operated by the address

Response:

```json
[
    {
        "method": "setAddr",
        "event": "AddressChanged",
        "block_number": 8834378,
        "block_timestamp": "2019-10-29 13:47:34",
        "transaction_index": 154,
        "log_index": 170,
        "transaction_hash": "0x09922ac0caf1efcc8f68ce004f382b46732258870154d8805707a1d4b098dfd0",
        "node": "0xee6c4522aab0003e8d14cd40a6af439055fd2577951148c14b6cea9a53475835",
        "name": "vitalik.eth"
    },
    {
        "method": "renew",
        "event": "NameRenewed",
        "block_number": 10572879,
        "block_timestamp": "2020-08-01 08:47:08",
        "transaction_index": 188,
        "log_index": 202,
        "transaction_hash": "0xab2cc289b49a5ccc797a699e831f5b9672c12cc69f6077ca8075bc6c8d18e760",
        "node": "0xee6c4522aab0003e8d14cd40a6af439055fd2577951148c14b6cea9a53475835",
        "name": "vitalik.eth"
    },
    {
        "method": "setAddr",
        "event": "AddressChanged",
        "block_number": 11862656,
        "block_timestamp": "2021-02-15 17:19:09",
        "transaction_index": 113,
        "log_index": 316,
        "transaction_hash": "0x160ef4492c731ac6b59beebe1e234890cd55d4c556f8847624a0b47125fe4f84",
        "node": "0xee6c4522aab0003e8d14cd40a6af439055fd2577951148c14b6cea9a53475835",
        "name": "vitalik.eth"
    },
    {
        "method": "setAddr",
        "event": "AddressChanged",
        "block_number": 11862656,
        "block_timestamp": "2021-02-15 17:19:09",
        "transaction_index": 113,
        "log_index": 312,
        "transaction_hash": "0x160ef4492c731ac6b59beebe1e234890cd55d4c556f8847624a0b47125fe4f84",
        "node": "0xee6c4522aab0003e8d14cd40a6af439055fd2577951148c14b6cea9a53475835",
        "name": "vitalik.eth"
    },
    {
        "method": "renew",
        "event": "NameRenewed",
        "block_number": 14929873,
        "block_timestamp": "2022-06-09 01:14:52",
        "transaction_index": 82,
        "log_index": 131,
        "transaction_hash": "0x0c1e0b69e4af1815834db4fdc946e2a24f7a7625c992709cdd86c507067ff95e",
        "node": "0xa1580ef894a94c6d646e612c2a5c641af63a4db147c2d268880239fdf6150af1",
        "name": "vitalik-buterin.eth"
    },
    {
        "method": "renew",
        "event": "NameRenewed",
        "block_number": 17080759,
        "block_timestamp": "2023-04-19 12:51:23",
        "transaction_index": 83,
        "log_index": 286,
        "transaction_hash": "0x7b91d77516ee54bf9c6bc21dde9d91d75b33e6795266111c88e57d74e8552397",
        "node": "0x133a0d6e787307c1bdb6a3cde083ac5096ad9d67298908427642512fa2f6aa4f",
        "name": "vbuterin.eth"
    },
    {
        "method": "renew",
        "event": "NameRenewed",
        "block_number": 17080762,
        "block_timestamp": "2023-04-19 12:51:59",
        "transaction_index": 69,
        "log_index": 206,
        "transaction_hash": "0xfb1bce20c86df4e22ad3f9bc99843f88cd3ed10644867350a530dc4581f9bf07",
        "node": "0x117e485acf1ebff586d1382b7689201ce23f7142ae92f4c397849f4b01e6a1cc",
        "name": "vitalikbuterin.eth"
    },
    {
        "method": "setAddr",
        "event": "AddressChanged",
        "block_number": 19930268,
        "block_timestamp": "2024-05-23 04:43:11",
        "transaction_index": 259,
        "log_index": 676,
        "transaction_hash": "0x75c795c47fa6facc754652ced9ff805a886e6228064a3e3b5eac758b3324368d",
        "node": "0xee6c4522aab0003e8d14cd40a6af439055fd2577951148c14b6cea9a53475835",
        "name": "vitalik.eth"
    },
    {
        "method": "register",
        "event": "NameRegistered",
        "block_number": 20553007,
        "block_timestamp": "2024-08-18 04:04:59",
        "transaction_index": 15,
        "log_index": 70,
        "transaction_hash": "0x649999f334679cb221eceda3dee07a6364c19a84764f7a16b5587cc67204a292",
        "node": "0xe422aae15240e573e09f4f3928c96fb984b2dd25c7e0a9eb96e2f419574ee504",
        "name": "dacc.eth"
    },
    {
        "method": "register",
        "event": "AddressChanged",
        "block_number": 20553007,
        "block_timestamp": "2024-08-18 04:04:59",
        "transaction_index": 15,
        "log_index": 68,
        "transaction_hash": "0x649999f334679cb221eceda3dee07a6364c19a84764f7a16b5587cc67204a292",
        "node": "0xe422aae15240e573e09f4f3928c96fb984b2dd25c7e0a9eb96e2f419574ee504",
        "name": "dacc.eth"
    }
]
```


# OpenSea

OpenSea is a decentralized marketplace for buying, selling, and trading non-fungible tokens (NFTs). It allows users to interact with a wide variety of digital assets, including artwork, collectibles, virtual land, and in-game items, all of which are secured on the blockchain, primarily Ethereum. OpenSea is one of the largest and most popular NFT platforms, offering tools for creators to mint and sell their NFTs and for collectors to discover and trade unique digital items.

Currently, it only supports the Seaport protocol, with plans to gradually update and expand support in the future.

**Key Features:**

* NFT trading volume
* NFT trading collections
* NFT trading buy volume
* NFT trading sell volume


# Data Class

To cover OpenSea actions and features, we created the following data classes:

* `AddressOpenseaTransaction`: Records all opensea transactions for a specific address. One opensea transaction per row.
* `OpenseaOrder`: Records information about individual OpenSea orders. One order per row.

These data classes capture the essential information for tracking OpenSea activities and order details.

#### `AddressOpenseaTransaction`

| Column Name       | Data Type | Note                                                      |
| ----------------- | --------- | --------------------------------------------------------- |
| address           | str       | Address associated with the transaction                   |
| related\_address  | str       | Related address in the transaction                        |
| is\_offer         | bool      | Flag indicating if the address is an offer in transaction |
| transaction\_type | int       | Type of the transaction(buy, sell or swap)                |
| order\_hash       | str       | Hash of the associated opensea order                      |
| zone              | str       | Zone of the transaction                                   |
| offer             | dict      | Details of the offer                                      |
| consideration     | dict      | Details of the consideration                              |
| fee               | dict      | Fee details for the transaction                           |
| transaction\_hash | str       | Hash of the transaction                                   |
| block\_number     | int       | Number of the block containing the transaction            |
| log\_index        | int       | Index of the log in the transaction                       |
| block\_timestamp  | int       | Timestamp of when the block was mined                     |
| block\_hash       | str       | Hash of the block containing the transaction              |
| protocol\_version | str       | Version of the seaport protocol used                      |

#### `OpenseaOrder`

| Column Name       | Data Type | Note                                              |
| ----------------- | --------- | ------------------------------------------------- |
| order\_hash       | str       | Hash of the order                                 |
| zone              | str       | Zone of the order                                 |
| offerer           | str       | Address of the offerer                            |
| recipient         | str       | Address of the recipient                          |
| offer             | dict      | Details of the offer                              |
| consideration     | dict      | Details of the consideration                      |
| block\_timestamp  | int       | Timestamp of when the block was mined             |
| block\_hash       | str       | Hash of the block containing the order            |
| transaction\_hash | str       | Hash of the transaction associated with the order |
| log\_index        | int       | Index of the log in the transaction               |
| block\_number     | int       | Number of the block containing the order          |
| protocol\_version | str       | Version of the protocol used (default: "1.6")     |


# Trigger and Function

The Opensea job is triggered when an Opensea-related transaction occurs or an event is produced. Specifically:

ref: [Opensea Event and Errors](https://docs.opensea.io/docs/seaport-events-and-errors)

* `OrderFulfilled`: When an order is successfully fulfilled

![Screenshot of Opensea](/files/rQjOIH35627cttMLda23)

```python
def parse_opensea_transaction_order_fulfilled_event(
        transaction: Transaction, contract_address: str, protocol_version: str = 1.6, fee_addresses: List[str] = []
) -> List[OpenseaLog]:
    results = []
    logs = transaction.receipt.logs
    for log in logs:
        if (
                log.topic0 == OPENSEA_EVENT_ABI_SIGNATURE_MAPPING["ORDER_FULFILLED_EVENT"]
                and log.address == contract_address
        ):
            opensea_transaction = decode_log(OPENSEA_EVENT_ABI_MAPPING["ORDER_FULFILLED_EVENT"], log)

            results.append(
                OpenseaLog(
                    orderHash="0x" + opensea_transaction["orderHash"].hex(),
                    offerer=opensea_transaction["offerer"],
                    zone=opensea_transaction["zone"],
                    recipient=opensea_transaction["recipient"],
                    offer=opensea_transaction["offer"],
                    consideration=opensea_transaction["consideration"],
                    block_timestamp=transaction.block_timestamp,
                    block_hash=transaction.block_hash,
                    block_number=transaction.block_number,
                    transaction_hash=transaction.hash,
                    log_index=log.log_index,
                    protocol_version=protocol_version,
                    fee_addresses=fee_addresses,
                )
            )

    return results
```

and the function to extract the data from the opensea event log:

```python
def transfer(self, opensea_logs: List[OpenseaLog]):
        for opensea_log in opensea_logs:
            yield OpenseaOrder(
                order_hash=opensea_log.orderHash,
                zone=opensea_log.zone,
                offerer=opensea_log.offerer,
                recipient=opensea_log.recipient,
                offer=opensea_log.offer,
                consideration=opensea_log.consideration,
                block_timestamp=opensea_log.block_timestamp,
                block_hash=opensea_log.block_hash,
                transaction_hash=opensea_log.transaction_hash,
                block_number=opensea_log.block_number,
                log_index=opensea_log.log_index,
                protocol_version=opensea_log.protocol_version,
            )
            offer = calculate_total_amount(opensea_log.offer)
            consideration = calculate_total_amount(opensea_log.consideration)
            fee = calculate_fee_amount(opensea_log.consideration, opensea_log.fee_addresses)

            opensea_transaciton_type = get_opensea_transaction_type(opensea_log.offer, opensea_log.consideration)
            if opensea_log.offerer == opensea_log.recipient:
                continue
            yield AddressOpenseaTransaction(
                address=opensea_log.offerer,
                related_address=opensea_log.recipient,
                is_offer=True,
                transaction_type=opensea_transaciton_type.value,
                order_hash=opensea_log.orderHash,
                zone=opensea_log.zone,
                offer=offer,
                consideration=consideration,
                fee=fee,
                transaction_hash=opensea_log.transaction_hash,
                block_number=opensea_log.block_number,
                log_index=opensea_log.log_index,
                block_timestamp=opensea_log.block_timestamp,
                block_hash=opensea_log.block_hash,
                protocol_version=opensea_log.protocol_version,
            )
            yield AddressOpenseaTransaction(
                address=opensea_log.recipient,
                related_address=opensea_log.offerer,
                is_offer=False,
                transaction_type=(
                    1 - opensea_transaciton_type.value
                    if opensea_transaciton_type.value <= 1
                    else opensea_transaciton_type.value
                ),
                order_hash=opensea_log.orderHash,
                zone=opensea_log.zone,
                offer=offer,
                consideration=consideration,
                fee=fee,
                transaction_hash=opensea_log.transaction_hash,
                block_number=opensea_log.block_number,
                log_index=opensea_log.log_index,
                block_timestamp=opensea_log.block_timestamp,
                block_hash=opensea_log.block_hash,
                protocol_version=opensea_log.protocol_version,
            )
```


# Run & Query

## How to Run

You can start the export\_opensea\_job by adding the following arguments:

```shell
-E OPEN_SEA
```

or

```shell
-O AddressOpenseaTransaction,OpenseaOrder
```

## Public API

### GET /v1/aci//opensea/profile

Get currently address level OpenSea related features

```json
{
  "address": "0x6657e801c98c0ba77169e3d65642b9e930928a03",
  "buy_txn_count": 27,
  "sell_txn_count": 11,
  "swap_txn_count": 0,
  "buy_opensea_order_count": 27,
  "sell_opensea_order_count": 11,
  "swap_opensea_order_count": 0,
  "buy_volume_usd": 18678.386373040692,
  "sell_volume_usd": 30148.607791721905,
  "create_time": "2024-09-05T04:53:04+00:00",
  "update_time": "2024-09-05T09:46:10+00:00",
  "first_transaction_hash": "0xcdd12932129e756aebbc5e88310c9e60294f40e212d61ef3b48c9dc594576a59",
  "first_block_timestamp": "2024-01-01T05:54:35+00:00",
  "txn_count": 36,
  "opensea_order_count": 36,
  "volume_usd": 48826.9941647626,
  "latest_transaction_hash": "0x036327d1172b02d002bf34d6f5288e9d57db9a34d78e77247b34b367df7b3d1b",
  "latest_block_timestamp": "2024-09-09T01:35:47+00:00"
}
```

### GET /v1/aci/{address}/opensea/transactions

Get Opensea address level opensea transactions

Response:

```json
{
  "data": [
    {
      "address": "0x6657e801c98c0ba77169e3d65642b9e930928a03",
      "transaction_hash": "0x036327d1172b02d002bf34d6f5288e9d57db9a34d78e77247b34b367df7b3d1b",
      "block_number": 20709805,
      "block_timestamp": "2024-09-09T01:35:47+00:00",
      "order_hash": "0x8b58fc15c3d5cf31b015e1d73b9ced4ac9ee5a39d6515a940e044c8a3c0688f6",
      "protocol_version": "1.5",
      "zone": "0x004c00500000ad104d7dbd00e3ae0a5c00560c00",
      "transaction_type": "sell",
      "volume_usd": 69.25004968606547,
      "items": [
        {
          "token_address": "0xa98cc213495b178bc0aa690223325bbed2dbbc71",
          "token_name": "Block Queens",
          "token_type": "ERC721",
          "token_symbol": "BQ",
          "amount": 1,
          "identifier": 171
        }
      ]
    },
    {
      "address": "0x6657e801c98c0ba77169e3d65642b9e930928a03",
      "transaction_hash": "0xa57e2dca011289f67e116114d3dff6fdfdec8a44a9efc11fbc99a65bd06eac40",
      "block_number": 20705832,
      "block_timestamp": "2024-09-08T12:16:35+00:00",
      "order_hash": "0x999a648aa281851978108584e107be465bdbd0cb1182bd3e1ed2ddd3be5ad99e",
      "protocol_version": "1.6",
      "zone": "0x000056f7000000ece9003ca63978907a00ffd100",
      "transaction_type": "buy",
      "volume_usd": 36.27089092218665,
      "items": [
        {
          "token_address": "0x46ac8540d698167fcbb9e846511beb8cf8af9bd8",
          "token_name": "Quantum Art",
          "token_type": "ERC721",
          "token_symbol": "QART",
          "amount": 1,
          "identifier": 580018
        },
        {
          "token_address": "0xc02aaa39b223fe8d0a0e5c4f27ead9083c756cc2",
          "token_name": "Wrapped Ether",
          "token_type": "ERC20",
          "token_symbol": "WETH",
          "amount": 7.9e-05,
          "identifier": null
        }
      ]
    },
    {
      "address": "0x6657e801c98c0ba77169e3d65642b9e930928a03",
      "transaction_hash": "0x43082a4afbb94326e1ad86c4462002adeb2225babf1d452ea81c77f0134b1d72",
      "block_number": 20672934,
      "block_timestamp": "2024-09-03T22:06:47+00:00",
      "order_hash": "0x4b4fe235e4ef1930bcc9ca72e0ea51827176464af4cdc8e27ec52320afc78a2b",
      "protocol_version": "1.6",
      "zone": "0x000056f7000000ece9003ca63978907a00ffd100",
      "transaction_type": "buy",
      "volume_usd": 36.759912153924276,
      "items": [
        {
          "token_address": "0x46ac8540d698167fcbb9e846511beb8cf8af9bd8",
          "token_name": "Quantum Art",
          "token_type": "ERC721",
          "token_symbol": "QART",
          "amount": 1,
          "identifier": 320014
        },
        {
          "token_address": "0xc02aaa39b223fe8d0a0e5c4f27ead9083c756cc2",
          "token_name": "Wrapped Ether",
          "token_type": "ERC20",
          "token_symbol": "WETH",
          "amount": 7.5e-05,
          "identifier": null
        }
      ]
    },
    {
      "address": "0x6657e801c98c0ba77169e3d65642b9e930928a03",
      "transaction_hash": "0x70bb3290333629cbdb0e907997104c2e2ed08dfbcb9dc5dccdab7a779246f1a3",
      "block_number": 20514912,
      "block_timestamp": "2024-08-12T20:26:23+00:00",
      "order_hash": "0x556c0f09ed315b38e698caf581a6955f17eb75421a12cebac44545f5afb2e841",
      "protocol_version": "1.6",
      "zone": "0x000056f7000000ece9003ca63978907a00ffd100",
      "transaction_type": "buy",
      "volume_usd": 40.66941051987043,
      "items": [
        {
          "token_address": "0x46ac8540d698167fcbb9e846511beb8cf8af9bd8",
          "token_name": "Quantum Art",
          "token_type": "ERC721",
          "token_symbol": "QART",
          "amount": 1,
          "identifier": 290041
        },
        {
          "token_address": "0xc02aaa39b223fe8d0a0e5c4f27ead9083c756cc2",
          "token_name": "Wrapped Ether",
          "token_type": "ERC20",
          "token_symbol": "WETH",
          "amount": 7.5e-05,
          "identifier": null
        }
      ]
    },
    {
      "address": "0x6657e801c98c0ba77169e3d65642b9e930928a03",
      "transaction_hash": "0x1523e1661bdaddc8fad8003b287d02eca048c11c48e21539c6073023861206c6",
      "block_number": 20491691,
      "block_timestamp": "2024-08-09T14:38:35+00:00",
      "order_hash": "0x821b6ba99df5c3369d4817b4809266d60c00afba817411181fe24f33fbc41ced",
      "protocol_version": "1.5",
      "zone": "0x004c00500000ad104d7dbd00e3ae0a5c00560c00",
      "transaction_type": "sell",
      "volume_usd": 1040.5875659115932,
      "items": [
        {
          "token_address": "0x7a15b36cb834aea88553de69077d3777460d73ac",
          "token_name": "Feral File — Unsupervised",
          "token_type": "ERC721",
          "token_symbol": "FF009",
          "amount": 1,
          "identifier": 9654437293614983878447008954651574738102085915959744418992085055843510330616
        }
      ]
    },
    {
      "address": "0x6657e801c98c0ba77169e3d65642b9e930928a03",
      "transaction_hash": "0x4622bf1001760e6ce5b90b554b78cfb05627fe066db00fd2c95402ca5ce7efd6",
      "block_number": 20455408,
      "block_timestamp": "2024-08-04T13:11:35+00:00",
      "order_hash": "0xd62e0dbd232f7c242010edb456de9c85ae13caf7acd167374f26173926625347",
      "protocol_version": "1.6",
      "zone": "0x000056f7000000ece9003ca63978907a00ffd100",
      "transaction_type": "buy",
      "volume_usd": 50.59778781852618,
      "items": [
        {
          "token_address": "0xa98cc213495b178bc0aa690223325bbed2dbbc71",
          "token_name": "Block Queens",
          "token_type": "ERC721",
          "token_symbol": "BQ",
          "amount": 1,
          "identifier": 171
        },
        {
          "token_address": "0xc02aaa39b223fe8d0a0e5c4f27ead9083c756cc2",
          "token_name": "Wrapped Ether",
          "token_type": "ERC20",
          "token_symbol": "WETH",
          "amount": 9.3e-05,
          "identifier": null
        },
        {
          "token_address": "0xc02aaa39b223fe8d0a0e5c4f27ead9083c756cc2",
          "token_name": "Wrapped Ether",
          "token_type": "ERC20",
          "token_symbol": "WETH",
          "amount": 0.001209,
          "identifier": null
        }
      ]
    },
    {
      "address": "0x6657e801c98c0ba77169e3d65642b9e930928a03",
      "transaction_hash": "0xe6eadba507c0d0afddadc89c76a400ecd1b578745dba6680e5633ad9981dabaf",
      "block_number": 20439335,
      "block_timestamp": "2024-08-02T07:20:47+00:00",
      "order_hash": "0x6518325945a9cba8aab18143087fd36dafb7a74ebf7bb066a071dc44925dc8c9",
      "protocol_version": "1.6",
      "zone": "0x000056f7000000ece9003ca63978907a00ffd100",
      "transaction_type": "buy",
      "volume_usd": 46.04946859100522,
      "items": [
        {
          "token_address": "0x46ac8540d698167fcbb9e846511beb8cf8af9bd8",
          "token_name": "Quantum Art",
          "token_type": "ERC721",
          "token_symbol": "QART",
          "amount": 1,
          "identifier": 360013
        },
        {
          "token_address": "0xc02aaa39b223fe8d0a0e5c4f27ead9083c756cc2",
          "token_name": "Wrapped Ether",
          "token_type": "ERC20",
          "token_symbol": "WETH",
          "amount": 7.7e-05,
          "identifier": null
        }
      ]
    },
    {
      "address": "0x6657e801c98c0ba77169e3d65642b9e930928a03",
      "transaction_hash": "0x580f5e0774f5b387cab6ac704e10a2498e02f72998b0eb283f1be9c5a8d06eea",
      "block_number": 20379983,
      "block_timestamp": "2024-07-25T00:28:59+00:00",
      "order_hash": "0xe7b2d570a070f267a6aacf65c7b716e42fdb69cb0722f1b7430cf4846ddcb5dc",
      "protocol_version": "1.6",
      "zone": "0x000056f7000000ece9003ca63978907a00ffd100",
      "transaction_type": "buy",
      "volume_usd": 796.3328595622321,
      "items": [
        {
          "token_address": "0xedc3ad89f7b0963fe23d714b34185713706b815b",
          "token_name": "Project Godjira Generation 2",
          "token_type": "ERC721",
          "token_symbol": "PGG2",
          "amount": 1,
          "identifier": 100
        },
        {
          "token_address": "0xc02aaa39b223fe8d0a0e5c4f27ead9083c756cc2",
          "token_name": "Wrapped Ether",
          "token_type": "ERC20",
          "token_symbol": "WETH",
          "amount": 0.0012555,
          "identifier": null
        }
      ]
    },
    {
      "address": "0x6657e801c98c0ba77169e3d65642b9e930928a03",
      "transaction_hash": "0x20ec732997b0d15983eb77c3c5eb75eb835bea1eb6480a619f129b20e3cc69d2",
      "block_number": 20366523,
      "block_timestamp": "2024-07-23T03:24:23+00:00",
      "order_hash": "0xe5d45eeaa10bd035f899095389aa559ab4f2beaf099cbecfcdc63875862d76d4",
      "protocol_version": "1.5",
      "zone": "0x004c00500000ad104d7dbd00e3ae0a5c00560c00",
      "transaction_type": "sell",
      "volume_usd": 2074.9284452838433,
      "items": [
        {
          "token_address": "0x42069abfe407c60cf4ae4112bedead391dba1cdb",
          "token_name": "CryptoDickbutts S3",
          "token_type": "ERC721",
          "token_symbol": "CDB",
          "amount": 1,
          "identifier": 4171
        }
      ]
    },
    {
      "address": "0x6657e801c98c0ba77169e3d65642b9e930928a03",
      "transaction_hash": "0xce9d2b9fbdd2309325b9609e35286f9092273f1355c41194568f4cab7907d7fc",
      "block_number": 20348614,
      "block_timestamp": "2024-07-20T15:23:35+00:00",
      "order_hash": "0x9a8c57542a4d83728bf8b73f29eefe7a8197d2afb051a5de7d0cc76cba3a681b",
      "protocol_version": "1.6",
      "zone": "0x000056f7000000ece9003ca63978907a00ffd100",
      "transaction_type": "buy",
      "volume_usd": 699.4866481083809,
      "items": [
        {
          "token_address": "0x42069abfe407c60cf4ae4112bedead391dba1cdb",
          "token_name": "CryptoDickbutts S3",
          "token_type": "ERC721",
          "token_symbol": "CDB",
          "amount": 1,
          "identifier": 4171
        },
        {
          "token_address": "0xc02aaa39b223fe8d0a0e5c4f27ead9083c756cc2",
          "token_name": "Wrapped Ether",
          "token_type": "ERC20",
          "token_symbol": "WETH",
          "amount": 0.000994,
          "identifier": null
        }
      ]
    }
  ],
  "total": 38
}
```


# Deposit to L2

A bridge to Layer 2 (L2) is a mechanism that allows users to transfer assets and data from a Layer 1 blockchain, like Ethereum, to a Layer 2 scaling solution. These bridges facilitate faster, cheaper transactions by moving activity off the main chain while still benefiting from the security of Layer 1. Once assets are bridged to L2, users can enjoy reduced fees and quicker transaction times, with the option to return their assets to the main chain when needed.

> The `deposit_to_l2_job` indexes all transactions related to deposit to l2 operations on the Ethereum mainnet. As a result, we can tract and query deposit-related informations.

**Key Features**

* Query deposit details through different dimensions (wallet address, chain name, bridge address, token, block number)
* Calculate the number of deposits made by a wallet on a specific bridge
* Determine on which chains a wallet has made deposit operations
* Calculate the quantity of tokens deposited by a wallet on different chains through various bridges

**Destination Chain:**

* **Optimism** Deposit ETH/ERC20 <https://etherscan.io/address/0x99C9fc46f92E8a1c0deC1b1747d010903E884bE1>
* **Arbitrum**
  * Gateway to Deposit ERC20 <https://etherscan.io/address/0x72Ce9c846789fdB6fC1f34aC4AD25Dd9ef7031ef>
  * Deposit ETH <https://etherscan.io/address/0x4Dbd4fc535Ac27206064B68FfCf827b0A60BAB3f>
* **Base (OP Stack)**
  * Deposit ETH <https://etherscan.io/address/0x3154Cf16ccdb4C6d922629664174b904d80F2C35>
* **Linea**
  * Deposit ETH <https://etherscan.io/address/0xd19d4B5d358258f05D7B411E21A1460D11B0876F>
  * Deposit ERC20 <https://etherscan.io/address/0x051F1D88f0aF5763fB888eC4378b4D8B29ea3319>
  * Deposit USDC <https://etherscan.io/address/0x504A330327A089d8364C4ab3811Ee26976d388ce>
* **Other notable chains** (Mantle,Scroll,zkSync,Blast,Manta,Polygon zkEVM,Taiko)


# Data Class

There are two data classes specifically for `deposit_to_l2_job`, representing:

1. `TokenDepositTransaction`: Transaction details of deposit operations
2. `AddressTokenDeposit`: Aggregated snapshot values of deposit operations across designed dimensions

**TokenDepositTransaction**

| Column Name       | Data Type | Note                                                               |
| ----------------- | --------- | ------------------------------------------------------------------ |
| transaction\_hash | str       | Primary key, hash of the transaction.                              |
| wallet\_address   | str       | Address that initiated the wallet.                                 |
| chain\_id         | int       | pecifies the L2 chain to which the deposit operation is directed.  |
| contract\_address | str       | Represents the contract which user makes deposit operations to L2. |
| token\_address    | str       | The token used in this deposit operation.                          |
| value             | int       | Represents the amount of tokens in this deposit transaction.       |
| block\_number     | int       | Number of the block containing the transaction.                    |
| block\_timestamp  | int       | Timestamp of when the block was mined.                             |

**AddressTokenDeposit**

| Column Name       | Data Type | Note                                                                                                                   |
| ----------------- | --------- | ---------------------------------------------------------------------------------------------------------------------- |
| wallet\_address   | str       | Primary key, address that initiated the wallet.                                                                        |
| chain\_id         | int       | Primary key, pecifies the L2 chain to which the deposit operation is directed.                                         |
| contract\_address | str       | Primary key, represents the contract which user makes deposit operations to L2.                                        |
| token\_address    | str       | Primary key, the token used for deposits through this contract.                                                        |
| value             | int       | Represents the total amount of tokens deposited on the specific L2 chain through this contract up to the current time. |
| block\_number     | int       | The block number of which included the last deposit transaction.                                                       |
| block\_timestamp  | timestamp | The block timestamp of when the block was mined.                                                                       |


# Trigger and Function

The `deposit_to_l2` job is triggered when an deposit-related transaction occurs or an event is produced. Specifically:

* `bridgeETHTo`: When the transaction content involves depositing WETH to a specific L2 chain through a contract ![Screenshot of Deposit](/files/cnmzlOjdY0lCHOGIAKlD)
* `depositERC20`: When the transaction content involves depositing ERC20 to a specific L2 chain through a contract ![Screenshot of Deposit](/files/yUw7qKH9igx8hUqWD3nx)

```python
def parse_deposit_transaction_function(
    transactions: List[Transaction],
    contract_set: set,
    chain_mapping: dict,
    sig_function_mapping: dict,
    sig_parse_mapping: dict,
) -> List[TokenDepositTransaction]:
    deposit_tokens = []
    for transaction in transactions:
        if transaction.to_address in contract_set:
            input_sig = transaction.input[0:10]
            if input_sig in sig_function_mapping:
                deposit_transaction = sig_parse_mapping[input_sig](
                    transaction=transaction,
                    chain_mapping=chain_mapping,
                    function=sig_function_mapping[input_sig],
                )
                deposit_tokens.append(deposit_transaction)
    return deposit_tokens
```

and the function to extract the deposit info from the transaction input data:

```python
def eth_deposit_parse(transaction: Transaction, chain_mapping: dict, function: ABIFunction) -> TokenDepositTransaction:
    return TokenDepositTransaction(
        transaction_hash=transaction.hash,
        wallet_address=transaction.from_address,
        chain_id=chain_mapping[transaction.to_address],
        contract_address=transaction.to_address,
        token_address=ETH_ADDRESS,
        value=transaction.value,
        block_number=transaction.block_number,
        block_timestamp=transaction.block_timestamp,
    )


def usdc_deposit_parse(transaction: Transaction, chain_mapping: dict, function: ABIFunction) -> TokenDepositTransaction:
    decoded_input = decode_transaction_data(function, HexStr(transaction.input))
    return TokenDepositTransaction(
        transaction_hash=transaction.hash,
        wallet_address=transaction.from_address,
        chain_id=chain_mapping[transaction.to_address],
        contract_address=transaction.to_address,
        token_address=USDC_ADDRESS,
        value=decoded_input["_amount"],
        block_number=transaction.block_number,
        block_timestamp=transaction.block_timestamp,
    )


def token_deposit_parse(
    transaction: Transaction, chain_mapping: dict, function: ABIFunction
) -> TokenDepositTransaction:
    decoded_input = decode_transaction_data(function, HexStr(transaction.input))
    return TokenDepositTransaction(
        transaction_hash=transaction.hash,
        wallet_address=transaction.from_address,
        chain_id=chain_mapping[transaction.to_address],
        contract_address=transaction.to_address,
        token_address=decoded_input["_l1Token"],
        value=decoded_input["_amount"],
        block_number=transaction.block_number,
        block_timestamp=transaction.block_timestamp,
    )
```

Meanwhile, after deposit\_to\_l2\_job is triggered, the AddressTokenDeposit value is continuously aggregated by TokenDepositTransaction:

```python
    for deposit in pre_aggregate_deposit_in_same_block(deposit_tokens):
        cache_key = (deposit.wallet_address, deposit.chain_id, deposit.contract_address, deposit.token_address)
        cache_value = self.cache.get(cache_key)
        if cache_value and cache_value.block_number < deposit.block_number:
            # add and save 2 cache
            token_deposit = AddressTokenDeposit(
                wallet_address=deposit.wallet_address,
                chain_id=deposit.chain_id,
                contract_address=deposit.contract_address,
                token_address=deposit.token_address,
                value=deposit.value + cache_value.value,
                block_number=deposit.block_number,
                block_timestamp=deposit.block_timestamp,
            )
    
            self.cache.set(cache_key, token_deposit)
            self._collect_item(AddressTokenDeposit.type(), token_deposit)
    
        elif cache_value is None:
            # check from db and save 2 cache
            history_deposit = self.check_history_deposit_from_db(
                deposit.wallet_address, deposit.chain_id, deposit.token_address
            )
            if history_deposit is None or history_deposit.block_number < deposit.block_number:
                token_deposit = AddressTokenDeposit(
                    wallet_address=deposit.wallet_address,
                    chain_id=deposit.chain_id,
                    contract_address=deposit.contract_address,
                    token_address=deposit.token_address,
                    value=deposit.value + history_deposit.value if history_deposit else deposit.value,
                    block_number=deposit.block_number,
                    block_timestamp=deposit.block_timestamp,
                )
                self.cache.set(cache_key, token_deposit)
                self._collect_item(AddressTokenDeposit.type(), token_deposit)
    
    self._data_buff[AddressTokenDeposit.type()] = distinct_collections_by_group(
        collections=self._data_buff[AddressTokenDeposit.type()],
        group_by=["wallet_address", "chain_id", "contract_address", "token_address"],
        max_key="block_number",
    )
```


# Run & Query

## How to Run

* You can start the deposit\_to\_l2\_job by adding the following arguments:

```shell
-O TokenDepositTransaction,TokenDepositTransaction --force-filter-mode true
or
-E DEPOSIT_TO_L2 --force-filter-mode true
```

Before starting job, user needs to configure the monitored l2 chain contract.

Here's the template for the configuration file.

### Token name Configuration

The configuration table includes a {token\_type} configuration item. In the current program, this is an enumerated value with the following options:

* `eth`
* `usdc`
* `Custom field`

The custom field option matches the name field in the decode section of the abi\_function. This tells the program which field represents the deposit token after decoding. Note: eth and usdc are internal configuration values. Typically, the deposit contract methods for these two tokens do not record the token address separately in the input data.

### ABI Function Specifics

In the abi\_function, the field name representing the deposit amount after decoding should be recorded as `_amount`.

```
[chain_deposit_info]
contract_info=
    {
        "{chain_id}":[{
            "contract": "{contract_address}",
            "ABIFunction": [
            {
                "token": "{token_name}",
                "json": {full json of abi function}
            },
            ……
            ]
        }],
        ……
    }
```

Here are part of the configuration information that are currently running

```
[chain_deposit_info]
contract_info=
    {
        "8453":[{
            "contract": "0x3154Cf16ccdb4C6d922629664174b904d80F2C35",
            "ABIFunction": [
            {
                "token": "eth",
                "json": {"inputs":[{"internalType":"uint32","name":"_minGasLimit","type":"uint32"},{"internalType":"bytes","name":"_extraData","type":"bytes"}],"name":"depositETH","outputs":[],"stateMutability":"payable","type":"function"}
            },
            {
                "token": "eth",
                "json":{"inputs":[{"internalType":"address","name":"_to","type":"address"},{"internalType":"uint32","name":"_minGasLimit","type":"uint32"},{"internalType":"bytes","name":"_extraData","type":"bytes"}],"name":"bridgeETHTo","outputs":[],"stateMutability":"payable","type":"function"}
            }]
        }],
        "10":[{
            "contract": "0x99C9fc46f92E8a1c0deC1b1747d010903E884bE1",
            "ABIFunction": [
            {
                "token": "eth",
                "json": {"inputs":[{"internalType":"uint32","name":"_minGasLimit","type":"uint32"},{"internalType":"bytes","name":"_extraData","type":"bytes"}],"name":"depositETH","outputs":[],"stateMutability":"payable","type":"function"}
            },
            {
                "token": "eth",
                "json":{"inputs":[{"internalType":"address","name":"_to","type":"address"},{"internalType":"uint32","name":"_minGasLimit","type":"uint32"},{"internalType":"bytes","name":"_extraData","type":"bytes"}],"name":"bridgeETHTo","outputs":[],"stateMutability":"payable","type":"function"}
            },
            {
                "token": "_l1Token",
                "json": {"inputs":[{"internalType":"address","name":"_l1Token","type":"address"},{"internalType":"address","name":"_l2Token","type":"address"},{"internalType":"address","name":"_to","type":"address"},{"internalType":"uint256","name":"_amount","type":"uint256"},{"internalType":"uint32","name":"_minGasLimit","type":"uint32"},{"internalType":"bytes","name":"_extraData","type":"bytes"}],"name":"depositERC20To","outputs":[],"stateMutability":"nonpayable","type":"function"}
            }]
        }]
    }
```

## Public API

### GET /v1/aci/{wallet\_address}/deposit/current

Get the total amount of deposits on each monitoring l2 chain for the current address

```json
{
    "wallet_address": "0x07AE8551BE970CB1CCA11DD7A11F47AE82E70E67",
    "deposit_times": 35,
    "chain_list": [
        "arbitrum",
        "linea"
    ],
    "asset_list": [
        {
            "chain": "linea",
            "bridge": "0x051f1d88f0af5763fb888ec4378b4d8b29ea3319",
            "token": "0x2260fac5e5542a773aa44fbcfedf7c193bc2c599",
            "token_name": null,
            "token_symbol": null,
            "token_icon_url": null,
            "token_type": null,
            "amount": 113891426.0
        },
        {
            "chain": "arbitrum",
            "bridge": "0x72ce9c846789fdb6fc1f34ac4ad25dd9ef7031ef",
            "token": "0xdac17f958d2ee523a2206206994597c13d831ec7",
            "token_name": null,
            "token_symbol": null,
            "token_icon_url": null,
            "token_type": null,
            "amount": 389739663425.0
        },
        {
            "chain": "arbitrum",
            "bridge": "0x72ce9c846789fdb6fc1f34ac4ad25dd9ef7031ef",
            "token": "0xa0b86991c6218b36c1d19d4a2e9eb0ce3606eb48",
            "token_name": null,
            "token_symbol": null,
            "token_icon_url": null,
            "token_type": null,
            "amount": 632746985369.0
        },
        {
            "chain": "arbitrum",
            "bridge": "0x72ce9c846789fdb6fc1f34ac4ad25dd9ef7031ef",
            "token": "0x2260fac5e5542a773aa44fbcfedf7c193bc2c599",
            "token_name": null,
            "token_symbol": null,
            "token_icon_url": null,
            "token_type": null,
            "amount": 628399963.0
        },
        {
            "chain": "linea",
            "bridge": "0x504a330327a089d8364c4ab3811ee26976d388ce",
            "token": "0xa0b86991c6218b36c1d19d4a2e9eb0ce3606eb48",
            "token_name": null,
            "token_symbol": null,
            "token_icon_url": null,
            "token_type": null,
            "amount": 59479591975.0
        },
        {
            "chain": "arbitrum",
            "bridge": "0x72ce9c846789fdb6fc1f34ac4ad25dd9ef7031ef",
            "token": "0xc02aaa39b223fe8d0a0e5c4f27ead9083c756cc2",
            "token_name": null,
            "token_symbol": null,
            "token_icon_url": null,
            "token_type": null,
            "amount": 1.8289342516725786e+21
        },
        {
            "chain": "arbitrum",
            "bridge": "0x72ce9c846789fdb6fc1f34ac4ad25dd9ef7031ef",
            "token": "0x6b175474e89094c44da98b954eedeac495271d0f",
            "token_name": null,
            "token_symbol": null,
            "token_icon_url": null,
            "token_type": null,
            "amount": 3.095182829935051e+23
        }
    ]
}
```

### GET /v1/aci/deposit/transactions

Query the detailed data of the deposit transaction according to the filter criteria

Response:

```json
{
  "data": [
    {
      "transaction_hash": "0x4769f88dfe3a7aaafffd180837e71a10c29a9ad8d6a3b1481224de7646ab78d0",
      "wallet_address": "0x07ae8551be970cb1cca11dd7a11f47ae82e70e67",
      "chain_id": 42161,
      "contract_address": "0x72ce9c846789fdb6fc1f34ac4ad25dd9ef7031ef",
      "token_address": "0x6b175474e89094c44da98b954eedeac495271d0f",
      "value": "31856.204529119157086825",
      "block_number": 20686814,
      "block_timestamp": "2024-09-05T20:35:47+08:00",
      "chain_name": "arbitrum",
      "name": null,
      "symbol": null,
      "icon_url": null,
      "token_type": null
    },
    {
      "transaction_hash": "0x07f0757d788ae9779c1958b4f3c5dbff0f2b6f5be4023252d612bd7d01fb2b8c",
      "wallet_address": "0x07ae8551be970cb1cca11dd7a11f47ae82e70e67",
      "chain_id": 42161,
      "contract_address": "0x72ce9c846789fdb6fc1f34ac4ad25dd9ef7031ef",
      "token_address": "0xa0b86991c6218b36c1d19d4a2e9eb0ce3606eb48",
      "value": "0.000000060012277694",
      "block_number": 20686447,
      "block_timestamp": "2024-09-05T19:22:11+08:00",
      "chain_name": "arbitrum",
      "name": null,
      "symbol": null,
      "icon_url": null,
      "token_type": null
    }
  ],
  "total": 35,
  "max_display": 35,
  "page": 1,
  "size": 25
}
```


# User Profile

Key Features

* \#of txns on Ethereum
* Age of wallet (First transaction date to now)
* \#of smart contracts deployed
* Initial gas funding source (EOA address)
* \[NFT Ticker] holder
* \[Airdrop Ticker] Airdrop hunter
* Whale holder of \[Token Ticker]
* \[Token Ticker] balance
* Large \[Token Ticker] transfer volume (usd value >$10,000 at the time of transfer)


# Introduction

## What is UDF?

In Hemera Indexer, a **User-Defined Function (UDF)** is essentially a custom indexing job that:

* **Extracts specific data** from the blockchain.
* **Processes** the data according to your custom logic.
* **Stores** the results in the database.

UDFs allow you to focus on the data that matters to your application without dealing with unnecessary blockchain data. For instance, you might be interested in indexing only specific events from smart contracts, calculating address-level features, or aggregating token balances. UDFs enable you to define this custom extraction and transformation logic.

## How UDFs Work

At a high level, a UDF in Hemera Indexer:

1. **Depends on certain blockchain data**: UDFs specify dependencies on existing data classes (e.g., blocks, transactions, logs) provided by the Hemera Indexer.
2. **Processes the data**: UDFs contain custom logic that processes the input data and transforms it into the desired output format.
3. **Outputs data to the database**: The processed data is stored in the database as defined by your model.

<figure><img src="https://lh7-rt.googleusercontent.com/slidesz/AGV_vUe6FaEl1g0leLlcvX_cgyM5rjt47DR5Pe2_t74cFeHU4AicQWSsBbak8pWacdKC6c_rJA-9xi9Cuoqg9uobJIYke8CKI4nA2AlN-T5xiTmgxaacho0FoBXToITFO75Rlsi-cofyzvGoNviZ5eqZcc3twbzlyXnH=s2048?key=VbpyRI7YKJkuvjCAzsPkmQ" alt=""><figcaption></figcaption></figure>

UDFs are integrated into the indexing process of Hemera Indexer. When the indexer runs, it not only indexes the standard blockchain data but also executes your UDFs to produce custom outputs.


# Components of UDFs

User-Defined Functions (UDFs) are essential for extending and customizing the behavior of the indexing process.&#x20;

Below are the core components of a UDF and the folder structure typically used for organizing these components.

## Models (Database Tables)

* **Purpose**: Models define the structure of the data that will be stored in the database and correspond to the schema of the tables in the database.
* **Usage**: You create models for storing processed data from UDFs.
* **Location**: `modules/custom/{feature_name}/models`

## Data Classes

* **Purpose**: Data Classes provide an easily accessible in-memory data structure within your code. They are similar to models but are used in memory during data processing and transformation.
* **Usage**: Data Classes are used to represent the processed data before storing it in the database.
* **Difference from Models**: While models represent the structure of data in the database, Data Classes represent data structures used in code.
* **Location**: `modules/custom/{feature_name}/domains`

## **Filter Configuration**

**Purpose**: Filters enable developers to specify conditions for the data a UDF will capture and ensures that only relevant data is processed and stored.

* **Customizable Filter Methods**:
  * Each UDF can define its own methods for filtering data by creating `get_filter job` functions. This allows developers to set custom criteria for data selection, focusing on specific transaction types, events, or data points essential to the UDF’s purpose.
  * **Example**: A filter might select only transactions involving specific addresses, token types, or contract interactions.

## Job Logic

* **Purpose**: Job Logic encapsulates the core functionality of the UDF, handling the process of transforming the data and persisting the results into the database.
* **Key Functions**:
  * `collapse`: The core logic of the UDF, responsible for processing and transforming input data.
  * `collect_domain`: Used to save the processed data into the database.

## Folder Structure

UDF components are typically organized in the following folder structure:

```
project_root/
└── modules/
    └── custom/
        └── {feature_name}/
            ├── models/
            │   └── (model files)
            ├── domains/
            │   └── (dataclass files)
            ├── abi.py
            └── job.py
└── config/
    └── indexer-config.yaml
```

* **common/models/**: Contains the database models.
* **indexes/models/**: Contains additional index-related models.
* **customs/domains/**: Contains domain-specific data and logic.

***

## UDF Logic Flow

* **Extract Data**: Retrieve relevant data from blocks, transactions, or logs as defined by the UDF configuration and ABI.
* **Process the Data**: Apply transformations and logic to the extracted data in `collapse` within `job.py`.
* **Create Data Instances**: Instantiate Data Classes with the processed information, representing data in-memory before storing it.
* **Save to Database**: Use `collect_domain` to save the transformed data to the database tables as defined by your model.

## Types of Data Models

1. **Address-Level Data**:
   * Requires periodic updates as new transactions occur.
   * For consistency, the indexer should be run in a specific order to avoid overwriting data inconsistently
   * **Use Case**: Useful for tracking data directly tied to specific wallet addresses, such as balances and transaction counts.
2. **Event-Level Data**:
   * Event data is easier to handle as it typically has unique keys for each event.
   * These can be processed multiple times without issues, as the data is inherently immutable.
   * **Use Case**: Ideal for capturing unique, discrete events that do not require historical consistency, such as contract-specific events (e.g., transfers).

## Indexing Historical Data

To index historical data, the indexer allows you to specify a starting block number in the environment configuration. The indexer will process the blockchain data from the specified block onward.

* **Address-Level Data**:**Address-Level Data**: Running the indexer in a specific order is critical to maintain consistency when dealing with past transactions. Use the scheduler with precise block ranges (e.g., `--block=1,2,10,50-100`) to ensure sequential processing.
* **Event-Level Data**: This type of data can be processed on any block range since events are usually unique and do not require strict ordering.

## Configuration changes in UDF

Configuration changes in the UDF, such as trigger conditions, directly impact database updates. These settings determine the frequency and type of data capture, affecting how and when new information is processed and stored. This ensures that developers have control over data update intervals based on their UDF requirements.

## **Utility Methods**

The UDF ecosystem in Hemera relies on a set of utility methods for interacting with blockchain data, facilitating data transformation, encoding, and decoding.

* **ABI Utilities**:
  * Use `Web3.contract` to define contract objects and establish functions and events relevant to your UDF.
  * Define related functions and events in `modules/custom/{feature_name}/abi.py` and import these definitions within `job.py` to ensure modular consistency.
* **Encoding and Decoding**:
  * Use encoding utilities to convert data into formats suitable for storage.
  * Decode transaction inputs, logs, and other data using the decoding methods provided in Hemera’s utility package.
* **String and Hex Conversions**:
  * For consistent handling of blockchain addresses and transaction data, use utility functions for hex-string conversions.

{% content-ref url="/pages/8XO16Pkj4VaWg0eYmsxk" %}
[Building User Defined Functions(UDF)](/udfs-user-defined-functions/building-user-defined-functions-udf)
{% endcontent-ref %}


# Building User Defined Functions(UDF)

This section walks the user through the steps to define, implement, and integrating a User-Defined Function (UDF) to process blockchain data, extract specific information, and store it in a PostgreSQL database.&#x20;

We'll be following 4 primary steps:

* Setting up input and output data classes
* Defining database models
* Implementing job logic
* Integrating the job into Hemera Indexer

## Prerequisites

Before beginning, set up the development environment by following [Hemera’s setup guide](/hemera-indexer/installation) for Docker or from source​.&#x20;

***

## **Step 1: Define the Input and Output Data Classes**

### **Input Data Class**

The input data class represents the structure of the data being processed by your job. For blockchain-based UDFs, this might be a `Transaction` or `Log` object representing blockchain transactions or events.

### **Output Data Class**

The output data class represents the processed data that will be stored in the database. This class should capture relevant fields from the input data and convert them into a format suitable for saving to the database.

```python
# your_udf_data_classes.py
from dataclasses import dataclass

@dataclass
class YourCustomDataClass:
    field1: str
    field2: int
    field3: str
    # Add other fields as necessary
```

This structure ensures that you are working with typed data and can easily process it within your UDF logic.

***

## **Step 2: Define the Database Model**

The model maps the processed data to the database table schema. You will use SQLAlchemy to define your model, which ensures that your data class output is correctly persisted in the PostgreSQL database.

```python
# models/your_udf_models.py
from sqlalchemy import Column, String, Integer
from hemera.models.base import Base  # Import Base class from Hemera Indexer

class YourCustomModel(Base):
    __tablename__ = 'your_custom_table'

    field1 = Column(String, primary_key=True)
    field2 = Column(Integer)
    field3 = Column(String)
    # Add other fields matching your data class
```

Make sure that the fields defined in the model correspond to those in the data class to ensure proper mapping of the processed data.

***

## **Step 3: Implement the UDF Job**

The job logic is where your UDF processes the input data and produces output data. The job needs to:

* **Define dependencies**: Specify which data classes you depend on, such as `Transaction` or `Log`.
* **Process transactions**: Implement the logic to filter, extract, and transform data from the input.
* **Define `get_filter()`** to Specify Which Blockchain Events to Process. Filter based on criteria like addresses and topics to limit which blockchain events are processed.
* Core Logic in `_process()` Function:&#x20;
  * Iterate through each transaction.
  * Apply filtering logic to select specific transactions, such as high-value transactions.
  * Transform data from the transaction and map it to the fields in the output data class.
* **Save to database**: Convert the output into a format that matches your model and store it in the database.

```python
# your_udf_jobs.py
from hemera.jobs.base_job import BaseJob
from hemera.dataclasses import Transaction
from your_udf_data_classes import YourCustomDataClass
from your_udf_models import YourCustomModel

class YourCustomJob(BaseJob):
    """
    A custom job to process blockchain data and output custom data.
    """

    def __init__(self):
        super().__init__()
        self.dependencies = [Transaction]  # Define input data classes
        self.output_data_classes = [YourCustomDataClass]  # Define output data classes

    def process(self, transactions):
        """
        Your custom processing logic goes here.
        """
        output_data = []
        for tx in transactions:
            if self._is_target_transaction(tx):
                data = YourCustomDataClass(
                    field1=tx.hash,
                    field2=int(tx.value, 16),
                    field3=tx.from_address
                )
                output_data.append(data)
        return output_data

    def get_filter(self):
        """
        Implement any filtering logic needed for transactions.
        """
        # Example: return transactions that match a specific criterion
        return None

```

***

## **Step 4: Integrate Your UDF Job into Hemera Indexer**

{% hint style="info" %}
When you essentially put your UDFs in the above mentioned schema, you can essentially call the UDFs through the CLI.  If you want to run the UDF by default when the indexer runs, you can follow below steps:&#x20;
{% endhint %}

After defining your job, integrate it into Hemera Indexer so it can run as part of the indexing process. You may need to:

1. **Register the job**: Place your job file in the appropriate directory within the Hemera Indexer project (e.g., `hemera/jobs/custom/`).
2. **Update configuration**: Ensure that your job is included in the Hemera Indexer job registry or configuration files so that it is executed during indexing.

```python
# Example job registration (this can vary depending on the indexer setup)
job_registry.register(YourCustomJob)



```

Once updated, run the hemera indexer. For more detailed steps and deployment options, refer to the [Testing and Running UDF](/udfs-user-defined-functions/testing-and-running-udf)section.&#x20;

***

{% hint style="info" %}
We have an example UDF created below. The ERC721 Minting Job UDF is to get the mints for a particular ERC721 contract address. Check the example and implementation below :point\_down:
{% endhint %}

<details>

<summary> <strong>Example: ERC721 Minting Job</strong></summary>

Here’s an example of implementing a UDF job that processes ERC721 token minting transactions from a blockchain and saves the mint data to a PostgreSQL database.

#### **Step 1: Define Input and Output Data Classes**

**Input Data Class**: The input for this job is a `Transaction`, which is provided by the Hemera Indexer.

**Output Data Class**: The output class `ERC721TokenMint` represents the minting information for each ERC721 token.

```python
# indexer/modules/custom/erc721_token_mint/domain/erc721_mint_time.py
from dataclasses import dataclass
from indexer.domain import FilterData

@dataclass
class ERC721TokenMint(FilterData):
    token_address: str
    token_id: int
    block_number: int
    block_timestamp: int
    transaction_hash: str
```

#### **Step 2: Define the Database Model**

```python
# indexer/modules/custom/erc721_token_mint/models/erc721_token_mint.py
from sqlalchemy import Column, BIGINT, TIMESTAMP, BYTEA, NUMERIC
from hemera.models.base import Base

class ERC721TokenMint(Base):
    __tablename__ = 'erc721_token_mint'

    token_address = Column(BYTEA, primary_key=True)
    token_id = Column(NUMERIC(100), primary_key=True)
    block_number = Column(BIGINT)
    block_timestamp = Column(TIMESTAMP)
    transaction_hash = Column(BYTEA)
```

#### **Step 3: Implement the UDF Job**

```python
# indexer/modules/custom/erc721_token_mint/export_erc721_token_mint_job.py
import logging
from typing import List
from indexer.domain.log import Log
from indexer.domain.token_transfer import TokenTransfer, extract_transfer_from_log
from indexer.domain.transaction import Transaction
from indexer.jobs.base_job import FilterTransactionDataJob
from indexer.modules.custom.erc721_token_mint.domain.erc721_mint_time import ERC721TokenMint

logger = logging.getLogger(__name__)

TARGET_TOKEN_ADDRESS = ["0x144e8e2450d8660c6de415a56452b10187343ad6"]

def _filter_mint_tokens(logs: List[Log]) -> List[TokenTransfer]:
    token_transfers = []
    for log in logs:
        token_transfers += extract_transfer_from_log(log)
    return [
        transfer
        for transfer in token_transfers
        if transfer.from_address == "0x0000000000000000000000000000000000000000"
    ]

class ExportERC721MintTimeJob(FilterTransactionDataJob):
    dependency_types = [Transaction]
    output_types = [ERC721TokenMint]

    def _collect(self):
        logs = self._data_buff[Log.type()]
        mint_tokens = _filter_mint_tokens(logs)
        erc721_mint_infos = [
            ERC721TokenMint(
                token_address=token.token_address,
                token_id=token.value,
                block_timestamp=token.block_timestamp,
                block_number=token.block_number,
                transaction_hash=token.transaction_hash
            )
            for token in mint_tokens
        ]
        self._collect_domains(erc721_mint_infos)
```

#### **Step 4: Integration**

Register the `ExportERC721MintTimeJob` within the Hemera Indexer so that it becomes part of the indexing workflow.

This example demonstrates a complete workflow for a UDF job that processes blockchain transactions and stores the result in a database.

**Notes**:

* **Dependencies**: Specify the data classes your job depends on (e.g., `Transaction`).
* **Output Data Classes**: Specify the data classes your job outputs.
* **Processing Logic**: Implement your custom logic in the `process` method.
* **Database Interaction**: Use SQLAlchemy sessions to save data to the database.

**Step 4: Integrate Your UDF into Hemera Indexer**

To have your UDF executed as part of the indexing process, you need to register it within Hemera Indexer.

Depending on how Hemera Indexer is set up, you may need to:

* Place your job files in the appropriate directories (e.g., `hemera/jobs/custom/`).
* Update any job registries or configuration files to include your job.

</details>


# Testing and Running UDF

Before deploying your UDF, it's important to test it to ensure it works as expected.

## Unit Tests

Write unit tests for your processing logic:

import unittest from your\_udf\_jobs import YourCustomJob from hemera.dataclasses import Transaction

class TestYourCustomJob(unittest.TestCase):

```python
def test_process(self):
    # Setup test data
    transactions = [
        Transaction(hash='0x1', value='0xde0b6b3a7640000', from_address='0xabc'),  # 1 Ether
        Transaction(hash='0x2', value='0x1bc16d674ec80000', from_address='0xdef'),  # 2 Ether
    ]
    job = YourCustomJob()
    # Run the process method
    result = job.process(transactions)
    # Assert the results
    self.assertEqual(len(result), 1)
    self.assertEqual(result[0].field1, '0x2')
    # Add more assertions as necessary

if name == 'main': unittest.main()
```

### Integration Tests

Run the indexer with your UDF in a test environment:

* Use a test blockchain network (e.g., **Ganache**, or a testnet) to provide blockchain data.
* Use a test database to avoid affecting production data.
* Run the indexer and verify that your UDF processes data correctly and stores it in the database.

## Running the hemera indexer with UDFs

Once you've tested your UDF, you can deploy it as part of the Hemera Indexer in your production environment.

### Steps to Running

1. **Ensure Code is in the Correct Location**: Place your UDF code in the appropriate directories within the Hemera Indexer project.
2. **Update Configuration**: Make sure any configuration files or scripts include your UDF.
3. **Update Database**: Apply any new database migrations.
4. **Restart the Indexer**: Restart the Hemera Indexer to pick up the new UDF.

#### Using Docker

If you're using Docker to run Hemera Indexer:

* **Update the Docker Image**: Include your UDF code in the Docker image.
* **Rebuild the Image**:

```bash
docker-compose build
```

* **Restart the Containers**:

```
docker-compose up -d
```

#### Running from Source

1. Install development tools as specified in the Hemera documentation.
2. Prepare PostgreSQL and other dependencies.
3. Setup the enviroment:

```sh
make development
source .venv/bin/activate
```

Run the hemera Indexer through CLI:&#x20;

```bash

 python3 hemera.py stream \  --provider-uri https://ethereum.publicnode.com \  --start-block 12598072 \  --end-block 20804486 \
  --output-types UniswapV3Token \
  --block-batch-size 10000 \
  --postgres-url postgresql://hemera:123asd@localhost:5432/demo \
  --output postgres \
  --config-file config/indexer-config-template.yaml
```

Another way to test whether your UDFs are correctly configured and functioning is by checking the output folder or the PostgreSQL database. If you're using a PostgreSQL database, you can query the relevant table to verify the data.

For more detailed setup and deployment options, refer to the  [Installation](/hemera-indexer/installation) section.


# Troubleshooting and Support

This section covers some common issues you might run into while working with UDFs, along with practical steps to troubleshoot and get things back on track.

## **Common Issues and Fixes**

1. **Filter Configuration Issues**
   * **Problem**: The UDF isn’t capturing the expected data.
   * **Fix**: Double-check your `get_filter job` setup. Make sure it includes the conditions needed to capture the right data. Confirm that field names and values match what’s actually in the data source.
2. **Data Collection or Processing Errors**
   * **Problem**: Data isn’t saving to the database, or unexpected data is showing up in the logs.
   * **Fix**: Look at the `_collect` and `_process` functions to ensure they’re running in sequence. Check that data transformations in `_process` are working as expected, and verify that any dependency data is correctly fetched and formatted.
3. **Slow or Failing Dependency Data Acquisition**
   * **Problem**: The UDF is slow or failing when pulling dependency data.
   * **Fix**: Review your data acquisition setup and consider simplifying it with the standardized collection methods. Only fetch the data you absolutely need to keep things efficient.
4. **Database Model Conflicts**
   * **Problem**: Errors related to storing data in the database or issues with schema conflicts.
   * **Fix**: Confirm that all hex fields are set to store as `bytea` in your model. Make sure fields like `address` or `transaction_hash` (reserved fields) are only used as intended to avoid conflicts.
5. **Hex and String Conversion Errors**
   * **Problem**: Errors related to data format, especially with hex-string conversions.
   * **Fix**: Use the utility methods designed for hex and string conversions, especially when working with blockchain addresses and transaction hashes, to keep data consistent and avoid format issues.

## **Getting Help and Giving Feedback**

If you’ve tried these fixes but still have trouble, or if you think you’ve found a unique issue, here’s what you can do:

1. **Document Your Steps**: Take note of what you’ve tried, any error logs, and relevant configurations.
2. **Reach Out for Help**: You can open a GitHub issue or email <contact@thehemera.com> or reach us out on [discord](https://discord.gg/socialscan). Including as much detail as possible will make it easier for the team to help you quickly.


# Supported UDFs

This document outlines the supported User-Defined Functions (UDFs) for the ACI Network. You can incorporate these UDF classes into your projects to build custom UDFs using these pre-configured or pre-built components.

### 1. User Profile (on Ethereum)

| **Feature**                | **Detail**                                      |
| -------------------------- | ----------------------------------------------- |
| Account age                | Calculated in years (e.g., 6.8 years)           |
| Smart contracts deployed   | Count and details (e.g., 3 contracts)           |
| Initial gas funding source | Address of the initial funder                   |
| ERC20 balance              | Token ticker and balance for each token         |
| Token transfers            | Sent and received transactions with details     |
| Large token transfers      | Significant transfers with frequency and amount |
| Transaction count          | Total incoming and outgoing transactions        |
| Gas consumption            | Total gas used                                  |
| Token transfer count       | Total token transfers (in/out)                  |

### 2. UniswapV3 Trading

| **Feature**          | **Detail**                                   |
| -------------------- | -------------------------------------------- |
| Number of trades     | Total count of trades                        |
| Trade details        | Individual trade information (buy/sell/swap) |
| Unique traded assets | List of unique tokens traded                 |
| Total trading volume | Cumulative value of all trades in USD        |
| Average trade value  | Average size of trades in USD                |

### 3. UniswapV3 Liquidity

| **Feature**                   | **Detail**                             |
| ----------------------------- | -------------------------------------- |
| Liquidity provision frequency | Number of times liquidity was provided |
| Liquidity actions             | Details of adding/removing liquidity   |
| Number of liquidity pools     | Count of unique pools participated in  |
| Total LP value                | Total value of liquidity provided      |
| Pool details                  | Specific pool information and amounts  |
| Liquidity provision history   | First provision date and duration      |

### 4. ENS (Ethereum Name Service)

| **Feature**             | **Detail**                                   |
| ----------------------- | -------------------------------------------- |
| ENS domain holdings     | List of owned ENS domains                    |
| ENS actions             | Registration, renewal, and transfer details  |
| First registration time | Date of first ENS registration               |
| Primary ENS name        | The main ENS name for the address            |
| Resolved ENS domains    | List of ENS domains resolving to the address |

### 5. Bridge (Deposit) to L2

| **Feature**            | **Detail**                                               |
| ---------------------- | -------------------------------------------------------- |
| Bridge usage frequency | Number of times bridged to L2                            |
| Destination chains     | List of L2 chains bridged to                             |
| Bridged assets         | Details of assets bridged (chain, bridge, token, amount) |
| Bridge transactions    | Specific bridge transaction details                      |

### 6. OpenSea NFT Trading

| **Feature**            | **Detail**                               |
| ---------------------- | ---------------------------------------- |
| NFT trading count      | Total number of NFT trades               |
| NFT trading volume     | Total, buy, sell, and swap volumes       |
| NFT transactions       | Details of individual NFT trades         |
| Unique NFT collections | List of different NFT collections traded |

### 7. EigenLayer Restaking

| **Feature**            | **Detail**                                                |
| ---------------------- | --------------------------------------------------------- |
| Total restaking amount | Cumulative amount restaked                                |
| Restaking actions      | Details of restaking, withdrawal initiation, and claiming |
| First restaking time   | Date of first restaking action                            |

### 8. AAVE Lending

| **Feature**           | **Detail**                        |
| --------------------- | --------------------------------- |
| Collateral value      | Total value of collateral added   |
| Collateral actions    | Details of adding collateral      |
| First collateral time | Date of first collateral addition |
| Borrowed value        | Total value of assets borrowed    |
| Borrowing actions     | Details of borrowing transactions |
| First borrowing time  | Date of first borrowing action    |

These UDFs provide comprehensive insights into user activities, covering various aspects such as general account information, DeFi interactions, NFT trading, and cross-chain activities. They enable detailed analysis of wallet behaviors and patterns across different protocols.


# FAQs

#### **1. General Overview**

**Q: What are User-Defined Functions (UDFs) in Hemera?**\
A: UDFs in Hemera allow developers to extend and customize the behavior of the indexing process. They enable custom data handling, extraction, processing, and storage of blockchain data, allowing the creation of highly specific, feature-oriented data streams.

**Q: What is the typical lifecycle of a UDF?**\
A: A UDF lifecycle includes the following stages:

* **Definition**: Define UDF parameters, triggers, and configuration.
* **Deployment**: Deploy the UDF to listen and react to blockchain events.
* **Execution**: UDF processes and transforms data, saving results to the database.
* **Maintenance**: Update, version, or retire UDFs as needed.

***

#### **2. UDF Configuration and Components**

**Q: What is the recommended file structure for organizing UDF components?**\
A: The recommended file structure is:

* `modules/custom/{feature_name}/models` for Models.
* `modules/custom/{feature_name}/domains` for Data Classes.
* `modules/custom/{feature_name}/abi.py` for ABI files.
* `job.py` for the UDF job logic.
* Central configuration under `config/indexert-config.yaml`.

**Q: How should I structure Models and Data Classes in UDFs?**\
A: Models define database table structures and store processed data. Data Classes, used in memory, represent data during processing. Models use the `bytea` format for hex fields and include special fields like `address`, `transaction_hash`, `block_number`, and `block_timestamp` with reserved meanings.

**Q: How can I configure filters in a UDF?**\
A: Define custom filters using the `get_filter job` method. This allows specific data selection criteria based on transaction types, addresses, or events, which reduces the data load by processing only relevant data.

***

#### **3. Data Handling and Processing**

**Q: What is the purpose of consolidating `_collect` and `_process` functions?**\
A: Consolidating `_collect` and `_process` ensures a sequential data flow where `_collect` gathers data and `_process` transforms it. This setup simplifies the code structure, making it easier to maintain and reducing potential errors.

**Q: What are the standardized methods for collecting data in a UDF?**\
A: Hemera recommends four main data collection methods:

* **Block-Level Collection** for raw block data.
* **Transaction-Level Collection** for transaction-specific data.
* **Log-Based Collection** for event-related data.
* **ABI-Decoded Collection** for complex contract interactions.

**Q: How do I handle dependency data acquisition in a standardized way?**\
A: Use consistent naming and standardized utility functions for common dependency data requirements (e.g., fetching transaction details). Limit data acquisition to necessary dependencies, streamlining both code readability and resource efficiency.

***

#### **4. Database and Data Consistency**

**Q: How does database management work in UDFs?**\
A: Hemera automates database management for standard tasks, but developers configure and manage specific aspects within their UDFs. Models define how data is structured in the database, while `collect_domain` in the job logic handles data persistence.

**Q: How does data consistency work for Address-Level and Event-Level data?**\
A:

* **Address-Level Data**: Requires ordered indexing for consistency, especially for historical transactions.
* **Event-Level Data**: Unique identifiers allow flexible indexing across any block range without strict ordering.

**Q: Can I process historical data with a UDF?**\
A: Yes, you can set a starting block number in the configuration. The indexer will process blockchain data from this point onward. Address-level data requires ordered processing, while event-level data is flexible and can be processed in any range.

***

#### **5. Utility and Data Transformation Methods**

**Q: What utility methods are available for handling data in UDFs?**\
A: Hemera provides various utility methods:

* **ABI Utilities**: Define contract objects and functions using `Web3.contract`.
* **Encoding/Decoding**: Encode data for storage and decode transaction inputs or logs.
* **String and Hex Conversions**: Standardize hex and string conversions for addresses and transaction data.

**Q: How can I encode and decode data within my UDF?**\
A: Use the utility functions provided by Hemera to encode data for storage and decode transaction inputs, logs, or other data elements as required by your UDF’s processing logic.

***

#### **6. Troubleshooting and Feedback**

**Q: What should I do if I encounter bugs in my UDF?**\
A: Follow these steps:

1. Check your logs to identify error sources.
2. Review your filter configurations, triggers, and dependency data acquisition settings.
3. Submit detailed bug reports or feature suggestions via GitHub or by emailing <support@thehemera.com>.

**Q: How can I submit feedback or feature requests for UDF improvements?**\
A: You can submit feedback through the Hemera GitHub repository, or send an email to <support@thehemera.com> with a clear description of your suggestions.

***

#### **7. Version Control and Roadmap**

**Q: What happens when a UDF version is updated?**\
A: UDF version updates may introduce new fields, logic, or features. Compatibility checks are essential to ensure that updates do not disrupt existing workflows. If necessary, migration steps will be outlined for major version changes.

**Q: What is the roadmap for future UDF features and enhancements?**\
A: The roadmap includes support for additional UDFs, improved data handling capabilities, and expanding compatibility with more chains. Community feedback helps shape this roadmap, so please share any feature requests.

***

#### **8. Ecosystem and Community Support**

**Q: Where can I connect with other developers or ask questions?**\
A: Hemera’s Discord is the primary channel for developer communication, where you can ask questions, share ideas, and collaborate with the community.

**Q: Can the community contribute new UDFs or support new chains?**\
A: Yes! Community contributions are encouraged, and detailed guidelines are provided for adding UDFs and expanding chain support. Contribute via the Hemera GitHub repository or discuss with the community on Discord.

**Q: How can I submit a completed UDF?**\
A: Prepare your code following Hemera’s guidelines and submit it to the Hemera GitHub repository or email it to <support@thehemera.com>. Your submission will be reviewed, and upon approval, integrated into the ecosystem.

**Q: How can I qualify for the Hemera Builder Program?**\
A: Developers who contribute at least five UDFs and meet certain quality standards are eligible to join the Builder Program, which offers token grants and exclusive resources. Apply via the Hemera Builder Program Form.


# The story behind building Hemera

The idea originated from the learning process by a team of AI & ML experts serving hundreds of Dapps and Infra projects' growth teams for 2+ years.

Over 2 years ago, when Hemera team (fka. w3w\.ai) first came to build in web3, we were amazed by the potential of crypto data --- an ever-growing data network where each piece of the record owned by their rightful owners (the end users) in a decentralized and trustless way. Instantly we were picturing the future of internet where web3's equivalence of Facebook, TikTok, Amazon, OpenAI and Google thrive, the massive user-owned data would allow 3rd party developer to build experiences that the internet has never seen before.&#x20;

Then two main questions quickly showed up in our heads:

1. Are there enough demands in the industry for on-chain data now?
2. Any opportunities to build a durable project on top of a data set anyone can access?

For the first question regarding demands, the answer is a resounding YES. The team very quickly acquired close to 1000 projects as customers using our first application (a web3 CRM) production to help community growth and airdrops. The projects had a wide range of backgrounds, including Dapps like GameFi, NFTs, SocialFi, and infrastructures like blockchains and protocols ---- all of them needed some way to differentiate quality users on-chain from sybils, bots and fake accounts.&#x20;

Answers for the second question weren't obvious until we realized building a new infrastructure for the industry was in fact most needed: while we were trying to build the CRM application to help our customers, we in fact ended up spending up to 90% of the engineering resources building our internal Machine Learning (ML) & data infrastructures from scratch (starting from pulling raw transaction data from RPC nodes for each blockchain we were supporting). Thus we decided to turn our learnings in the infrastructure layer to a novel indexing protocol to help all web3 developers abstract out complexities involved with processing and computing semantics data out of on-chain transactions, hence the Hemera Protocol.&#x20;

Team background: Hemera's founding team built enterprise software powered by ML & Deep Learning (DL) in the Silicon Valley, and sold the it to some of the largest enterprises in both US and Asia to help them understand user behaviors: detect sybil attackers and identify potential big spenders (whales). This unique experience in web2 gave the Hemera team lots of insights in how to build large scale AI & data systems to support various ML & AI modeling use cases.&#x20;


# Partners & Backers

Our partners are the most trusted names in web3 ecosystems.

<figure><img src="/files/4AMB9oEuojKa8sv2aCDV" alt=""><figcaption></figcaption></figure>

Hemera is backed by the most prominent venture capital funds, including:

<figure><img src="/files/DlO1vbq7mlWkivCmnY7n" alt=""><figcaption></figcaption></figure>


# Partnership inquiries

If you are a developer or project team eager to explore the possibilities of integrating Hemera Protocol into your web3 application or project, we are here to support you every step of the way. Whether you are interested in leveraging our account-centric indexing for enhanced data interoperability or have questions about the potential use cases tailored to your needs, we're ready to assist.

To discuss your specific requirements and explore how Hemera can benefit your project, feel free to schedule a call with one of our team members. You can conveniently do so by [visiting our Calendly link](https://calendly.com/sammi_sg/30min?month=2024-09).

If you prefer written communication or have detailed queries, we welcome you to reach out to us via email at <contact@thehemera.com>.


# Hemera Powered Explorers

<table data-full-width="true"><thead><tr><th>Company</th><th>Environment</th><th>URL</th></tr></thead><tbody><tr><td>Polygon zkevm</td><td>Mainnet</td><td><a href="https://zkevm.socialscan.io/">https://zkevm.socialscan.io/</a></td></tr><tr><td>linea</td><td>Mainnet</td><td><a href="https://linea.socialscan.io/">https://linea.socialscan.io/</a></td></tr><tr><td>mantle</td><td>Mainnet</td><td><a href="https://mantle.socialscan.io/">https://mantle.socialscan.io/</a></td></tr><tr><td>cyber</td><td>Mainnet</td><td><a href="https://cyber.socialscan.io/">https://cyber.socialscan.io/</a></td></tr><tr><td>cyber</td><td>Testnet</td><td><a href="https://cyber-testnet.socialscan.io/">https://cyber-testnet.socialscan.io/</a></td></tr><tr><td>story</td><td>Testnet</td><td><a href="https://story-testnet.socialscan.io/">https://story-testnet.socialscan.io/</a></td></tr><tr><td>mint</td><td>Testnet</td><td><a href="https://mint-testnet.socialscan.io/">https://mint-testnet.socialscan.io/</a></td></tr><tr><td>GMNetwork</td><td>Testnet</td><td><a href="https://gm-testnet.socialscan.io/">https://gm-testnet.socialscan.io/</a></td></tr><tr><td>mint</td><td>Mainnet</td><td><a href="https://mint.socialscan.io/">https://mint.socialscan.io/</a></td></tr><tr><td>morphl2</td><td>Testnet</td><td><a href="https://morphl2-testnet.socialscan.io/">https://morphl2-testnet.socialscan.io/</a></td></tr><tr><td>merlin</td><td>Mainnet</td><td><a href="https://merlin.socialscan.io">https://merlin.socialscan.io</a></td></tr><tr><td>manta</td><td>Testnet</td><td><a href="https://manta-sepolia-testnet.socialscan.io/">https://manta-sepolia-testnet.socialscan.io/</a></td></tr><tr><td>taiko</td><td>Testnet</td><td><a href="https://taiko-katla.socialscan.io">https://taiko-katla.socialscan.io</a></td></tr><tr><td>taiko</td><td>Testnet</td><td><a href="https://taiko-hekla.socialscan.io">https://taiko-hekla.socialscan.io</a></td></tr><tr><td>taiko</td><td>Mainnet</td><td><a href="https://taiko.socialscan.io">https://taiko.socialscan.io</a></td></tr><tr><td>manta</td><td>Mainnet</td><td><a href="https://manta.socialscan.io/">https://manta.socialscan.io/</a></td></tr><tr><td>dodo</td><td>Testnet</td><td><a href="https://dodo-testnet.socialscan.io/">https://dodo-testnet.socialscan.io/</a></td></tr><tr><td>gm</td><td>Mainnet</td><td><a href="https://gm.socialscan.io/">https://gm.socialscan.io/</a></td></tr><tr><td>story</td><td>Testnet</td><td><a href="https://story-odyssey-testnet.socialscan.io/">https://story-odyssey-testnet.socialscan.io/</a></td></tr><tr><td>morphl2</td><td>Mainnet</td><td><a href="https://morphl2.socialscan.io/">https://morphl2.socialscan.io/</a></td></tr><tr><td>somnia</td><td>Devnet</td><td><a href="https://somnia-devnet.socialscan.io/">https://somnia-devnet.socialscan.io/</a></td></tr><tr><td>zetachain</td><td>Mainnet</td><td><a href="https://zetachain.socialscan.io/">https://zetachain.socialscan.io/</a></td></tr></tbody></table>


# Active Developer Hackathons

Participate in our latest hackathons.

* Taiko Grant Factory Hackathon: <https://dorahacks.io/hackathon/grant-factory/>
* Linea Oct-Nov Developer Cook Off: <https://www.hackquest.io/en/hackathon/explore/Linea-Oct-Nov-Dev-Cook-off>


# Developer Contribution

We welcome contributions from the developer community to enhance and expand the functionality of UDFs on the Hemera platform. Here’s how you can contribute:

1. **Submit Pull Requests**
   * Clone the [Hemera Indexer GitHub repository](https://github.com/HemeraProtocol/hemera-indexer).
   * Create a new branch for your changes, including documentation, code improvements, or bug fixes.
   * Ensure all contributions follow Hemera’s contribution guidelines, with well-documented code and passing tests.
2. **Report Issues**
   * If you encounter a bug or have a feature suggestion, open an issue on the GitHub repository.
   * Describe the issue or feature in detail, providing examples and steps to reproduce if applicable.
3. **Join Developer Discussions**
   * Connect with Hemera developers on [discord](https://discord.gg/socialscan) to discuss improvements, get feedback, and collaborate on projects.

{% hint style="success" %}

## Introducting UDF Builder's Program

Ready to push boundaries with blockchain data? The **UDF Builder Program** is here for developers who want to dive deep, create, and connect. This is more than just building functions – it’s about setting new standards for data innovation in the Hemera ecosystem.\
\
Here’s what’s in it for you:

* **Top-Tier Resources**: Collaborate with Hemera partners, get insights from industry pros, and access a curated library of tools.
* **Token Rewards**: Get recognized for your work with token grants as you develop groundbreaking UDFs.
* **Network Opportunities**: Connect with other innovators, projects, and potential collaborators.

Bring your skills, your ideas, and your drive – let’s build something that matters.<br>

👉[ **Explore the UDF Builder Program**](https://hemera-devrel.notion.site/UDF-Builder-Program-1283bbc7e8fc806b961bf0a2ea5f1e94) and start making an impact.

{% endhint %}