Data Architecture

Redis-Based Data Architecture

The arbitrage system uses Redis as the central data hub for real-time market data processing. This architecture supports high-frequency data ingestion from multiple cryptocurrency exchanges while maintaining low-latency access for arbitrage calculations and dashboard visualization.

Core Components

The data architecture consists of four main components:

• Data Ingestion: WebSocket streams and REST APIs collect market data

• Data Storage: Redis provides fast key-value storage and pub/sub messaging

• Data Processing: Buffered message processing for high-throughput operations

• Data Distribution: Real-time data flow to arbitrage engine and dashboard

Data Ingestion Layer

WebSocket Stream Management

The system maintains persistent WebSocket connections to collect real-time data:

Exchange Connections:

• Aster Exchange: wss://fstream.asterdex.com for futures data

• Backpack Exchange: wss://ws.backpack.exchange for spot and futures data

• zkLighter Exchange: wss://mainnet.zklighter.elliot.ai/stream for perpetual data

Data Types Collected:

• Orderbook Updates: Bid/ask prices with depth information

• Funding Rates: Real-time funding rate changes for perpetuals

• Market Statistics: Volume, price changes, and market status

Redis Data Structure

Key Organization

The system uses hierarchical key structures for efficient data organization:

Market Data Keys:

exchange:token:market_type
├── aster:BTC:spot      → {"symbol": "BTCUSDT", "best_ask": 45000, "best_bid": 44950, ...}
├── backpack:ETH:futures → {"symbol": "ETH_PERP", "best_bid": 2800, "funding_rate": 0.0001, ...}
└── lighter:SOL:futures  → {"symbol": "SOL/USD", "funding_rate": 0.0002, ...}

Index Keys:

arbitrage:index → {
    "total_tokens": 45,
    "aster_tokens": 15,
    "backpack_tokens": 20,
    "lighter_tokens": 10,
    "available_tokens": "BTC,ETH,SOL,...",
    "updated_at": "2024-01-01T12:00:00Z"
}

Opportunity Logging Keys

Arbitrage opportunities are logged with structured data:

arb:timestamp:type:exchange:asset → {
    "strategy": "CROSS_EXCHANGE",
    "buy_exchange": "aster",
    "sell_exchange": "backpack",
    "buy_price": 45000,
    "sell_price": 45150,
    "profit_pct": 0.333,
    "net_profit_pct": 0.183,
    "total_fees_pct": 0.15,
    "is_profitable": true,
    "priority": "MEDIUM"
}

Historical Data Keys

Funding rate history is maintained for analysis:

funding_history:exchange:token → [
    {"timestamp": "2024-01-01T12:00:00Z", "rate": 0.0001, "mark_price": 45000},
    {"timestamp": "2024-01-01T12:08:00Z", "rate": 0.00015, "mark_price": 45100},
    ...
]