cypher_substreams Substreams modules

This package was initialized via substreams init, using the evm-minimal template.

Overview

This substream monitors blockchain transactions in real-time, specifically designed to track incoming transactions to specified addresses. It supports both native ETH transfers and ERC20 token transfers.

Features

• Live Monitoring: Continuously monitors blockchain data for incoming transactions
• Address Filtering: Filter transactions by from and to addresses
• ERC20 Support: Detects ERC20 token transfers and extracts relevant data
• Comprehensive Logging: Detailed logging for debugging and monitoring
• Real-time Alerts: Logs matched transactions as they occur
• Performance Optimized: Block filtering reduces data consumption by 90-95%

Performance Optimization

Block Filtering (Recommended)

This substream includes advanced block filtering that dramatically reduces data consumption and processing time. Instead of processing all blocks, it only processes blocks that contain:

1. ERC20 Transfer events (for token transfers)
2. Transactions to monitored addresses (for incoming transfers)
3. Transactions from monitored addresses (for outgoing transfers)

Expected Performance Improvements:

• 90-95% reduction in data consumption
• 80-90% reduction in processing time
• Significantly lower memory usage

Setup Block Filtering:

# Monitor incoming transactions to a specific address
python generate_block_filter.py --to 0x742d35Cc6634C0532925a3b8D4C9db96C4b4d8b6

# Monitor outgoing transactions from a specific address
python generate_block_filter.py --from 0x742d35Cc6634C0532925a3b8D4C9db96C4b4d8b6

# Monitor both incoming and outgoing for an address
python generate_block_filter.py --to 0x742d35Cc6634C0532925a3b8D4C9db96C4b4d8b6 --from 0x742d35Cc6634C0532925a3b8D4C9db96C4b4d8b6

# Dry run to see the generated query without updating files
python generate_block_filter.py --to 0x742d35Cc6634C0532925a3b8D4C9db96C4b4d8b6 --dry-run

After running the block filter generator:

# Rebuild the substream with optimizations
make build

# Run with optimized filtering
./run_live.sh --to 0x742d35Cc6634C0532925a3b8D4C9db96C4b4d8b6

How Block Filtering Works:

The block filter uses the eth_common:index_events module to pre-filter blocks based on:

• ERC20 Transfer events: evt_sig:0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef
• Direct transactions: tx_to:{address} and tx_from:{address}

This ensures your substream only processes blocks that contain relevant transactions, dramatically reducing the data stream from potentially hundreds of MB per block to just the relevant transactions.

Usage

Quick Start - Live Monitoring

For live monitoring of incoming transactions:

# Make the script executable (first time only)
chmod +x run_live.sh

# Start live monitoring
./run_live.sh

Note: When no start block is specified, the substream automatically starts from the current head of the blockchain and continues monitoring new blocks as they arrive. This ensures you're always monitoring the latest transactions without processing historical data.

Manual Setup

# Build the project
cargo build --target wasm32-unknown-unknown --release

# Authenticate with Substreams
substreams auth

# Run in live mode (starts from current head and continues monitoring new blocks)
substreams run substreams.yaml filter_transactions --log-level=info

Historical Data

To process historical blocks:

# Process from a specific block number
substreams run substreams.yaml filter_transactions --start-block 1000000

# Process a range of blocks
substreams run substreams.yaml filter_transactions --start-block 1000000 --stop-block 1000100

Configuration

Address Filtering

You can configure the addresses to monitor in two ways:

1. Configuration File (substreams.yaml)

Configure the addresses to monitor in substreams.yaml:

params:
  filter_transactions: 'to=0xcccd218a58b53c67fc17d8c87cb90d83614e35fd'

You can also filter by multiple addresses:

params:
  filter_transactions: 'to=0xcccd218a58b53c67fc17d8c87cb90d83614e35fd&from=0x1234567890123456789012345678901234567890'

2. Command Line Parameters (Recommended)

The run_live.sh script supports command line parameters for flexible configuration:

# Monitor transactions to a specific address
./run_live.sh --to 0x1234567890123456789012345678901234567890

# Monitor transactions from a specific address
./run_live.sh --from 0xabcdefabcdefabcdefabcdefabcdefabcdefabcd

# Monitor transactions between two addresses
./run_live.sh --to 0x1234567890123456789012345678901234567890 --from 0xabcdefabcdefabcdefabcdefabcdefabcdefabcd

# Use custom parameters string
./run_live.sh --params 'to=0x1234567890123456789012345678901234567890&from=0xabcdefabcdefabcdefabcdefabcdefabcdefabcd'

# Monitor on different networks
./run_live.sh --network mainnet.streamingfast.io:443 --to 0x1234567890123456789012345678901234567890
./run_live.sh --network polygon-mainnet.streamingfast.io:443 --to 0x1234567890123456789012345678901234567890

# Use default parameters from substreams.yaml
./run_live.sh

3. Direct Substreams Command

You can also pass parameters directly to the substreams command:

# Build the project first
cargo build --target wasm32-unknown-unknown --release

# Run with custom parameters
substreams run -e base-mainnet.streamingfast.io:443 substreams.yaml filter_transactions \
    --params filter_transactions='to=0x1234567890123456789012345678901234567890' \
    --start-block 1000000 \
    --limit-processed-blocks 0

# Run on different networks
substreams run -e mainnet.streamingfast.io:443 substreams.yaml filter_transactions \
    --params filter_transactions='to=0x1234567890123456789012345678901234567890' \
    --start-block 1000000 \
    --limit-processed-blocks 0

Supported Parameters

• to: Monitor incoming transactions to this address (supports both direct transfers and ERC20 transfers)
• from: Monitor outgoing transactions from this address

Supported Networks

• base-mainnet.streamingfast.io:443 (default) - Base network
• mainnet.streamingfast.io:443 - Ethereum mainnet
• polygon-mainnet.streamingfast.io:443 - Polygon network
• Any other StreamingFast endpoint

Parameter Format

Parameters use URL query string format:

• Single parameter: to=0x1234567890123456789012345678901234567890
• Multiple parameters: to=0x1234567890123456789012345678901234567890&from=0xabcdefabcdefabcdefabcdefabcdefabcdefabcd
• Multiple addresses: to=0x1234567890123456789012345678901234567890,0xabcdefabcdefabcdefabcdefabcdefabcdefabcd

Examples

See ./examples.sh for a comprehensive list of usage examples.

Output Format

The substream outputs transaction data in the following format:

{
  "transactions": [
    {
      "from": "0x...",
      "to": "0x...",
      "hash": "0x...",
      "value": "1000000000000000000",
      "token_id": "0xeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee"
    }
  ]
}

Token Types

• Native ETH: token_id will be 0xeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee
• ERC20 Tokens: token_id will be the contract address of the token

Publishing

Optionally, you can publish your Substreams to the Substreams Registry.

substreams registry login         # Login to substreams.dev
substreams registry publish       # Publish your Substreams to substreams.dev

Modules

block_filter (New)

This module pre-filters blocks to only include those with relevant transactions:

• ERC20 Transfer events
• Transactions to/from monitored addresses

This dramatically reduces data consumption and processing time.

filter_transactions

This module filters and processes blockchain transactions based on specified address filters. It:

• Processes transactions from pre-filtered blocks
• Applies address-based filtering (from/to)
• Detects ERC20 token transfers
• Extracts transaction values and token information
• Provides comprehensive logging for monitoring

Dependencies

Key dependencies:

• substreams: Core Substreams framework
• substreams-ethereum: Ethereum-specific functionality
• num-bigint: Big integer arithmetic for token amounts
• serde: Serialization/deserialization
• anyhow: Error handling