> For the complete documentation index, see [llms.txt](https://docs.nuralex.tech/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.nuralex.tech/technology-stack.md).
# Technology Stack
Architecture, product design, and operating controls for AI-assisted automated trading.
## :chart-line: AI automated trading system
> A comprehensive architecture, product, and operations guide for building a multi-broker, AI-driven automated trading platform on **Google Cloud Platform**, ingesting market data, news, fundamentals, and analyst ratings to drive position sizing and trade timing decisions.
{% hint style="warning" %}
This system trades real capital. AI produces recommendations only. Deterministic risk controls approve every order.
{% endhint %}
{% columns %}
{% column %}
#### :google: Cloud-native platform
Google Cloud provides managed messaging, analytics, security, and observability.
{% endcolumn %}
{% column %}
#### :shield-halved: Risk-first execution
The OMS and risk boundary remain independent of model behavior.
{% endcolumn %}
{% endcolumns %}
### Architecture at a glance
```
Market data ─┐
News ────────┼──▶ Google Cloud Pub/Sub ──▶ AI signal generation ──▶ Risk gates ──▶ Broker execution
Filings ─────┘
```
#### Navigate this guide
1. **Technology** covers services, data stores, and ingestion pipelines.
2. **Product** defines modules, accounts, and broker integration.
3. **Operations** defines rollout gates, monitoring, and hard stops.
### Purpose and scope
This guide is the single source of truth for engineering, product, and operations decisions on the automated trading platform. It is split into three strictly segregated top-level sections:
1. :microchip: **Technology** — stack, pipelines, and data architecture
2. :cubes: **Product** — features, modules, and account management
3. :gear: **Operations** — rollout, monitoring, and failsafes
**Primary cloud**: :google: **Google Cloud Platform (GCP)**. Managed GCP equivalents are preferred where available. This reduces operational overhead.
***
## :microchip: 1. Technology
### 1.1 :layer-group: Technology stack
A trading system has three very different workload profiles that must coexist: **low-latency execution**, **high-throughput batch analytics**, and **ML training/inference**. No single language or database serves all three well, so the stack is deliberately polyglot.
#### 1.1.1 Programming Languages
| Layer | Language | Justification |
| ----------------------------------- | ----------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **Execution Engine / Order Router** | **Rust** | This is the single most latency- and correctness-critical path in the system — it holds the connection to broker APIs and is the last line of code before money moves. Rust gives zero-cost abstractions, no GC pauses (which matter when a pause of even a few milliseconds can mean a stale price at order time), and a strict ownership model that eliminates entire classes of concurrency bugs (data races, use-after-free) at compile time. This is worth the steeper learning curve specifically because this component controls capital. |
| **Data Ingestion Services** | **Go** | See detailed comparison below — Go is the better fit for the ingestion tier specifically, despite Rust being used for execution. |
| **Infrastructure / DevOps Tooling** | **Go**, using **Pulumi** | Pulumi's Go SDK allows infrastructure-as-code to be written in the same language as the ingestion/platform services, with real type-checking, testability (unit-testable IaC), and code reuse (shared structs/config between app and infra code) — a meaningful advantage over HCL-based tools when the team is already Go-fluent from the ingestion layer |
| **AI/ML Research & Model Training** | **Python** | De facto standard; unmatched ecosystem (pandas, PyTorch, scikit-learn, statsmodels); fastest iteration speed for quants/data scientists |
| **Model Serving** | **Python (FastAPI)** wrapping ONNX/TorchScript exported models, or **Vertex AI Endpoints** (GCP-native managed serving) | ONNX export gives language-agnostic, low-latency inference callable from the Rust execution path; Vertex AI Endpoints removes the operational burden of running/scaling your own serving containers |
| **Backtesting Engine** | **Python** (vectorbt, backtrader, or custom) with performance-critical inner loops in **Rust** (via PyO3 bindings) | Python for strategy expressiveness; Rust extensions (rather than Cython) for the inner simulation loop keeps one systems language across the whole platform instead of three |
| **Frontend / Dashboard** | **TypeScript + React (Next.js)** | Strong typing for a UI that displays money and risk state; large ecosystem for charting (Recharts, D3, TradingView widgets) |
**Go vs. Rust for Data Ingestion — Decision**
Both languages were evaluated for the ingestion tier (market data, news, fundamentals ingestion workers). **Go is the recommended choice** for these reasons:
| Factor | Go | Rust | Verdict |
| -------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------ |
| Concurrency model for many parallel WebSocket/REST connections | Goroutines + channels are purpose-built for exactly this I/O-bound fan-out pattern, with minimal boilerplate | `tokio` async runtime is capable but requires more careful lifetime/ownership management for long-lived connection pools | **Go** — simpler code, faster to write and review |
| Latency sensitivity of this tier | Ingestion is I/O-bound (waiting on network), not compute-bound; GC pauses (sub-millisecond to low-millisecond with Go's modern GC) are immaterial next to network round-trip times | Rust's lack of GC is a non-factor here since the bottleneck is never CPU/memory management | **Tie** — Rust's latency edge doesn't apply to this workload |
| Ecosystem/SDK availability (broker & vendor SDKs, Protobuf/Avro, GCP client libraries) | Excellent, mature GCP and vendor client libraries | Good and improving, but noticeably smaller for third-party financial data vendor SDKs | **Go** |
| Developer velocity / hiring pool | Larger pool of engineers productive quickly; simpler mental model | Smaller pool, steeper onboarding | **Go** |
| Where Rust *is* still justified | N/A | Order execution path (above), where compute-bound correctness and zero-jitter matter | **Rust reserved for execution only** |
**Conclusion**: Use **Rust only where it earns its complexity** — the order execution engine. Use **Go everywhere else in the systems tier** (ingestion, stream processing, OMS adapters that are not on the hot path, DevOps tooling) to keep the team fast and the codebase approachable, while still getting a compiled, statically-typed, high-throughput language.
#### 1.1.2 Databases (by Data Type)
| Data Type | Recommended Database | GCP Hosting | Why |
| --------------------------------------------------------------------------- | ----------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Time-series market data** (tick, OHLCV, order book snapshots) | **QuestDB** | Self-managed on **GKE** (StatefulSet + regional persistent SSD) or **Compute Engine** with local SSD for maximum ingest throughput | QuestDB is purpose-built for extremely high-throughput tick ingestion with a SQL interface, native `SAMPLE BY` time-bucketing, and out-of-order ingestion handling — it consistently benchmarks ahead of general-purpose time-series extensions for raw tick-rate write throughput, which matters when ingesting full order-book depth across hundreds of symbols |
| **Unstructured news / text (intermediate storage before embedding)** | **MongoDB** (Atlas on GCP, or self-hosted on GKE) | **MongoDB Atlas on GCP** (recommended — fully managed, same region as compute) | News payloads are semi-structured and vary by source; MongoDB stores the raw article, extracted entities, and sentiment score as the system of record, and acts as the intermediate landing zone *before* vectorization — articles are written here first, then asynchronously embedded and pushed to the vector store |
| **Vector embeddings** (news + filings, for semantic/RAG search) | **Qdrant** | **Qdrant Cloud (GCP-hosted region)**, or self-hosted on GKE | Qdrant offers strong ANN performance (HNSW), payload filtering (filter by ticker/date alongside vector similarity — essential for "similar articles about AAPL in the last 30 days" queries), and a straightforward gRPC/REST API. MongoDB remains the durable system of record; Qdrant is purpose-built purely for similarity search and should be treated as a rebuildable index, not the source of truth |
| **Structured fundamentals** (10-Q/10-K line items, ratios, analyst ratings) | **PostgreSQL** | **Cloud SQL for PostgreSQL** (or **AlloyDB for PostgreSQL** for higher-throughput analytical joins) | Strong relational integrity, mature ecosystem, window functions for ratio calculations; AlloyDB specifically offers a columnar engine for fast analytical queries over fundamentals joined with price history while remaining wire-compatible with Postgres |
| **Order/execution/account state (OLTP, transactional)** | **PostgreSQL** | **Cloud SQL for PostgreSQL** (regional HA configuration) | Financial transactions require strict ACID guarantees; Cloud SQL's synchronous regional replication protects order/position state against zonal failure |
| **Feature store (ML features)** | **Feast** | Online store on **Memorystore for Redis**; offline store on **BigQuery** or **GCS + Parquet** | Standard open-source feature store pattern; keeps training/serving feature parity |
| **Model artifact & experiment tracking** | **MLflow** (or **Vertex AI Experiments** for a fully GCP-native option) | MLflow backed by **GCS** artifact store + **Cloud SQL** backend store | Versioning models, metrics, and reproducibility — critical for tracing which model version produced which trade decision |
| **Cold/bulk historical storage** | **Google Cloud Storage (GCS) + Parquet/Arrow** | Native GCS, with lifecycle rules moving older data to **Nearline/Coldline** storage classes | Cheapest durable storage for 30 years of tick/bar data; columnar format is ideal for analytical scans and integrates directly with BigQuery external tables and Dataflow/Spark |
| **Ad-hoc analytics / warehouse** | **BigQuery** | Native GCP | Serverless SQL analytics over the GCS Parquet archive and structured fundamentals for research queries, without needing to spin up a cluster |
| **Secrets** | **Google Secret Manager** | Native GCP | Centralized, IAM-controlled, automatically audited secret storage for broker API keys and OAuth tokens, replacing a self-hosted Vault |
#### 1.1.3 Caching Engines
| Use Case | Tool | GCP Hosting | Justification |
| ------------------------------------------------ | -------------------------------------------- | ------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Real-time quote cache / last-price lookups | **Redis** | **Memorystore for Redis** | Sub-millisecond reads, fully managed (patching, failover, backups handled by GCP), native pub/sub for fanning out live quotes to strategy engines and UI |
| Session/account state cache | **Redis** | **Memorystore for Redis** | Fast key-value access for position/exposure snapshots used by the risk engine on every order decision |
| Rate-limit token buckets (per broker/vendor API) | **Redis** (Lua scripts for atomic decrement) | **Memorystore for Redis** | Distributed rate limiting across multiple ingestion workers must be atomic and centrally visible |
| Computed feature cache (short TTL) | **Redis** | **Memorystore for Redis** | Avoids recomputing expensive rolling indicators (RSI, VWAP, volatility) on every tick |
#### 1.1.4 Messaging / Event Streaming
**Kafka is intentionally not used.** Running and tuning a Kafka cluster (Zookeeper/KRaft, partition rebalancing, broker capacity planning) is significant ongoing operational overhead for a small team, and GCP offers a directly equivalent, fully managed, enterprise-grade alternative.
| Use Case | Tool | Why |
| --------------------------------------------------------------------------- | ----------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Core event backbone (market data, news, order lifecycle events) | **Google Cloud Pub/Sub** | Fully managed, serverless, no cluster to size or maintain, scales automatically to very high throughput, and carries a formal enterprise SLA. Supports **ordered delivery** (per ordering key — e.g., per symbol) and **at-least-once/exactly-once-ish delivery** (via subscriber dedup), which covers the guarantees needed for market data and order events. IAM-native access control integrates directly with the rest of the GCP stack |
| Long-term event replay (beyond Pub/Sub's retention window) | **Pub/Sub → GCS/BigQuery sink (via Dataflow)** | Pub/Sub natively supports message retention up to **7 days** and **seek**-based replay within that window, which covers operational replay/debugging needs. For full historical reprocessing (e.g., "replay all of 2019's tick events to retrain a model exactly as it happened"), a standing **Pub/Sub → Dataflow → GCS (Parquet)** sink continuously archives every topic, and that GCS archive — not Pub/Sub itself — is the long-term replay source. This is a cleaner separation of concerns than relying on a single system for both real-time transport and long-term log storage |
| Task orchestration (e.g., "fetch 10-K for TICKER") | **Temporal.io** (self-hosted on GKE) | Temporal gives durable execution state, automatic retries/backoff, and visibility into long-running multi-step workflows (fetch → parse → embed → store → notify) that a simple queue does not provide |
| Scheduled/batch jobs (EOD batch, nightly retraining, corporate action sync) | **Cloud Composer** (GCP-managed Apache Airflow) | Removes the operational burden of running Airflow's scheduler/webserver/workers directly; DAG-based dependency management remains best-in-class for batch/ETL scheduling with strong observability, and it integrates natively with GCS, BigQuery, and Pub/Sub operators |
> **Trade-off acknowledged**: Pub/Sub is not a literal drop-in replacement for Kafka's infinite, replayable commit log — it is retention-bounded. The Pub/Sub-plus-GCS-archive pattern above is the standard way GCP-native architectures compensate for this, and it is the right trade for a small team: you get a zero-ops managed message bus for real-time transport, and cheap, durable, infinitely-retained Parquet for replay/backtesting, which is what the historical pipeline needs anyway (Section 1.2.3).
#### 1.1.5 AI/ML Frameworks
| Purpose | Framework | Justification |
| -------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Classical ML (feature-based signal models) | **scikit-learn**, **XGBoost / LightGBM** | Gradient-boosted trees consistently outperform deep nets on tabular financial features and are far more interpretable and faster to retrain |
| Deep learning (sequence models on price/volume) | **PyTorch** | Preferred for research flexibility; strong support for transformers/LSTMs/temporal convolutional networks used in time-series forecasting; trainable at scale on **Vertex AI Training** (managed GPU jobs) |
| NLP / sentiment on news & filings | **Hugging Face Transformers** (FinBERT, or a fine-tuned domain model) | Pretrained finance-domain sentiment models outperform generic sentiment models on financial text nuance ("beat estimates" vs. generic negativity around "beat") |
| LLM-based reasoning over filings/news (RAG) | **LangChain/LlamaIndex** orchestration, querying **Qdrant** for retrieval, + **Claude/GPT-class model via API** or **Vertex AI (Gemini) via API** | Use an LLM to summarize 10-Ks, extract risk factors, and generate qualitative signal explanations that feed (not solely drive) the quantitative model; Vertex AI's Gemini models are a natural GCP-native option alongside third-party APIs |
| Reinforcement learning (position sizing/timing research) | **Ray RLlib** on **GKE** or **Vertex AI (custom containers)** | If you pursue RL for position sizing, Ray RLlib scales training across a cluster; GKE or Vertex AI custom training jobs both support distributed Ray |
| Backtesting/quant research | **vectorbt**, **Zipline-reloaded**, or **backtrader** | Vectorized backtesting (vectorbt) is fastest for parameter sweeps across 30 years of data |
| Experiment tracking | **MLflow** or **Vertex AI Experiments** | Reproducibility and auditability of which model version produced which trade decision — critical for post-hoc compliance review |
> **Design Principle:** Keep the "signal generation" (ML/AI) layer strictly separated from the "decision/execution" (risk-checked order placement) layer. The AI never talks to a broker API directly — it emits a *recommendation* that passes through deterministic, rule-based risk gates before becoming an order. This is elaborated in [Section 2.1](https://claude.ai/chat/ad94ebc0-d162-4a38-bec0-c9ecdb9a67d2#21-core-features) and [Section 3.2](https://claude.ai/chat/ad94ebc0-d162-4a38-bec0-c9ecdb9a67d2#32-system-monitoring--failsafes).
***
### 1.2 :chart-line: Market data pipeline
#### 1.2.1 Architecture Overview
```
┌────────────────────┐
Live Feeds (WS/REST) │ Ingestion Workers │──▶ Cloud Pub/Sub (raw.market.ticks)
(IBKR, Polygon, etc.)│ (Go, per-vendor) │
└────────────────────┘
│
▼
┌───────────────────────┐
│ Stream Processor │──▶ Pub/Sub (normalized.bars.1m/5m/1d)
│ (Go, normalize, │
│ bar-build) │
└───────────────────────┘
│
┌───────────────┼─────────────────────────┐
▼ ▼ ▼
QuestDB Memorystore (Redis, Pub/Sub → Dataflow → GCS/Parquet
(queryable last price cache) (cold archive, 30yr backfill,
history, replay source)
GKE/Compute Engine)
```
#### 1.2.2 Live Data Ingestion
* **Connection model**: Maintain persistent WebSocket connections per vendor per exchange segment; use a supervised worker-per-connection pattern (Go goroutines) with automatic reconnect/backoff and sequence-gap detection.
* **Normalization**: All vendor-specific tick/quote formats are normalized into a canonical internal schema (symbol, exchange, timestamp (exchange + received), bid/ask/last, size, conditions) before being published to Pub/Sub, using a symbol-based ordering key so downstream consumers receive per-symbol events in order.
* **Bar aggregation**: A Go stream processor consuming from Pub/Sub rolls raw ticks into 1s/1m/5m/1d OHLCV bars in real time, publishing to their own Pub/Sub topics and writing directly into QuestDB via its high-throughput ILP (InfluxDB Line Protocol) ingestion endpoint.
* **Deduplication & gap-filling**: Use exchange sequence numbers where available; reconcile against a secondary vendor feed periodically to detect and backfill gaps.
#### 1.2.3 Historical Data (30 Years)
* **Bulk backfill strategy**: Historical bar/tick data is retrieved once via **batch REST/flat-file downloads** (not the streaming API) and loaded into **GCS** as Parquet, partitioned by `symbol/year/month`. This is far cheaper and faster than replaying 30 years through a live feed, and is queryable directly via **BigQuery external tables** for research without duplicating the data.
* **Load into QuestDB**: A batch job (Cloud Run job or GKE CronJob) reads the GCS Parquet archive and bulk-loads it into QuestDB for fast operational queries (e.g., "last 2 years of 1-minute bars for this symbol") — QuestDB holds the queryable working set, GCS holds the full 30-year archive of record.
* **Incremental EOD updates**: A nightly Cloud Composer DAG pulls the prior day's finalized bars from the vendor and appends to both the GCS Parquet archive and QuestDB.
* **Corporate actions**: Splits/dividends must be ingested separately and applied to build both raw and adjusted price series — errors here silently corrupt backtests.
#### 1.2.4 Recommended Data Providers
| Provider | Best For | Notes |
| ------------------------------------------ | ---------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Polygon.io** | US equities tick-level real-time + deep historical (aggregates back decades) | Strong WebSocket API, reasonable pricing tiers, good for both live and historical in one vendor |
| **Alpha Vantage** | Supplementary/backup data, fundamentals overlap, cheap entry tier | Rate limits are restrictive on free/low tiers; best as a secondary/fallback source |
| **FinancialData.net** | Cost-effective historical bulk downloads | Useful for filling long-tail historical gaps cheaply |
| **Interactive Brokers (IBKR) Market Data** | Live data tightly coupled to execution account | Since you're already integrating IBKR for execution, its market data can serve as a validation/cross-check feed |
| **Norgate Data / Tiingo** | Deep, survivorship-bias-free historical equities data | Norgate in particular is well-regarded in the quant community for clean 30+ year daily data including delisted symbols (critical to avoid survivorship bias in backtests) |
| **Databento** | Institutional-grade normalized historical + live tick data | Newer entrant with clean normalized schemas across exchanges, pay-per-use pricing |
> **Critical note on backtests:** Always source historical data that includes **delisted/bankrupt symbols** (survivorship-bias-free). Vendors like Norgate or Databento explicitly support this; free sources like Yahoo Finance typically do not, and backtests built on survivor-only universes systematically overstate returns.
#### 1.2.5 Multi-Vendor Redundancy
Run at least two independent live data vendors in production. Cross-validate prices in real time (e.g., flag if IBKR and Polygon last-trade diverge by more than X bps) and failover ingestion to the secondary vendor if the primary disconnects or lags beyond a latency threshold.
***
### 1.3 :newspaper: News ingestion
#### 1.3.1 Subscribing to News Sources
| Source Type | Recommended Provider | Notes |
| ---------------------------- | ------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------- |
| Real-time market-moving news | **Benzinga News API**, **Polygon.io News API**, or **Bloomberg/Reuters (enterprise tier)** | Benzinga is popular in retail-algo circles for low-latency, ticker-tagged news; Bloomberg/Reuters for institutional-grade coverage at higher cost |
| Social/alternative sentiment | **StockTwits API**, **Reddit API (via PRAW)**, **X/Twitter API** | Useful as a secondary sentiment signal, noisier than professional newswires — weight accordingly |
| Global/macro news | **NewsAPI.org**, **GDELT Project** (free, global event database) | GDELT is particularly strong for macro/geopolitical event detection at no cost |
| Press releases/regulatory | **PR Newswire / Business Wire feeds**, **SEC EDGAR full-text search** | Direct-from-source, lower latency than aggregators |
#### 1.3.2 Ingestion & NLP Pipeline
```
News API/Webhook ──▶ Cloud Pub/Sub (raw.news) ──▶ NLP Worker Pool (Go orchestrator,
Python/HF model workers)
│
┌────────────────────────┼─────────────────────────┐
▼ ▼ ▼
Entity/Ticker Sentiment Scoring Embedding
Extraction (NER) (FinBERT) Generation
│ │ │
└────────────────────────┴─────────────────────────┘
▼
MongoDB (Atlas on GCP)
— system of record, intermediate landing zone —
│
▼ (async embedding push)
Qdrant (vector index)
```
* **Entity extraction**: Use spaCy or a fine-tuned NER model to map free-text mentions to canonical tickers/CIKs, resolving aliases (e.g., "Apple" → AAPL).
* **Sentiment scoring**: Run FinBERT (or a fine-tuned variant) to produce a sentiment score (-1 to +1) and confidence per article per associated ticker, since a single article can be bullish for one company and bearish for a competitor mentioned in it.
* **Deduplication**: Many wire stories are syndicated/republished; use MinHash/SimHash near-duplicate detection before double-counting sentiment signal.
* **Landing in MongoDB first**: Every article is written to MongoDB immediately after entity/sentiment enrichment — this is the durable, queryable system of record. Embedding generation and the push to Qdrant happen as an asynchronous downstream step, so a Qdrant outage or reindex never risks losing source data, and the vector index can be fully rebuilt from MongoDB at any time.
* **Embedding generation**: Generate vector embeddings per article and upsert into **Qdrant**, tagged with payload metadata (ticker, timestamp, source, sentiment) so the RAG/LLM layer can combine semantic similarity with structured filters (e.g., "articles about AAPL, last 30 days, sentiment < -0.3").
#### 1.3.3 Best-Suited Databases
**MongoDB (Atlas on GCP)** is the primary store for raw/processed news documents because:
* News payloads are semi-structured and vary by source (schema flexibility without migrations)
* Native support for nested arrays (multiple tickers, entities, categories per article)
* Horizontal sharding scales well with news volume (potentially millions of articles/year)
* It acts as the **intermediate, durable landing zone** before the (comparatively more fragile, rebuildable) vector index is populated
**Qdrant** sits downstream of MongoDB as the dedicated vector search layer, providing fast approximate nearest-neighbor search with payload filtering for the RAG/LLM layer. MongoDB is the system of record; Qdrant is a purpose-built, rebuildable index — this separation means you never depend on a vector database for data durability, only for search performance.
***
### 1.4 :file-invoice-dollar: Financial reports pipeline
#### 1.4.1 Data Sources
| Data Need | Source | Notes |
| ----------------------------------------- | ----------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Primary 10-Q/10-K filings (raw) | **SEC EDGAR** (full-text search API + EDGAR Online) — free | Authoritative, free, but raw filings are unstructured HTML/XBRL requiring parsing |
| Structured/parsed financials from filings | **sec-api.io** | Purpose-built API that returns pre-parsed XBRL financial statements, section extraction (Item 1A Risk Factors, MD\&A), and full-text search over EDGAR — saves significant parsing engineering |
| Fundamentals + analyst ratings/estimates | **Finnhub** | Provides company fundamentals, earnings surprises, and analyst recommendation trends (buy/hold/sell counts) via a single API |
| Alternative fundamentals/estimates | **Financial Modeling Prep (FMP)**, **Zacks (via partner API)**, **Refinitiv/LSEG (enterprise)** | Use as secondary source for cross-validation of consensus estimates and ratings |
#### 1.4.2 Ingestion Workflow
1. **Filing detection**: Poll SEC EDGAR's real-time filing feed (or subscribe to sec-api.io's webhook) for new 10-Q/10-K filings for the tracked universe of tickers; a Cloud Run function receives the webhook and publishes a `new.filing` event to Pub/Sub.
2. **Retrieval**: A Go worker (Temporal workflow) pulls the raw filing (via EDGAR) or pre-parsed structured JSON (via sec-api.io).
3. **Extraction**: Parse XBRL tags into standardized line items (Revenue, Net Income, EPS, Total Assets, Free Cash Flow, etc.) using a canonical internal schema — GAAP tag names vary company to company, so a mapping/normalization layer (e.g., using the SEC's own XBRL taxonomy) is required.
4. **Ratio computation**: Derive standard ratios (P/E, ROE, Debt/Equity, margins) at ingestion time and store both raw line items and computed ratios in Cloud SQL/AlloyDB.
5. **Analyst ratings sync**: A nightly Cloud Composer DAG pulls updated analyst rating changes (upgrades/downgrades/price target changes) from Finnhub, tagged with the issuing firm and timestamp.
6. **Text-heavy sections (MD\&A, Risk Factors)**: Routed through the same MongoDB → embedding → Qdrant pipeline as news (Section 1.3.2), since these sections are qualitative and don't fit relational tables well.
#### 1.4.3 Database Mapping
| Data Element | Target Store | Schema Notes |
| --------------------------------------------------------- | --------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- |
| Structured line items (revenue, EPS, balance sheet items) | **Cloud SQL for PostgreSQL** — `fundamentals` schema | Normalized tables: `filings`, `income_statement`, `balance_sheet`, `cash_flow`, keyed by `(cik, fiscal_period, ticker)` |
| Computed ratios/derived metrics | **Cloud SQL / AlloyDB** — `fundamentals.ratios` (materialized view or table refreshed on ingestion) | Enables fast joins against price data (via a QuestDB→BigQuery export or federated query) for feature engineering |
| Analyst ratings/price targets | **Cloud SQL for PostgreSQL** — `analyst_ratings` table | Columns: `ticker, firm, action (upgrade/downgrade/initiate), prior_rating, new_rating, price_target, timestamp` |
| Raw filing documents & extracted text sections | **MongoDB (Atlas on GCP)** | Same store as news; linked to Postgres via `cik`/`accession_number` foreign key |
| Filing/report embeddings | **Qdrant** | Enables "find filings semantically similar to X" and RAG-based summarization |
This split — **structured numbers in Cloud SQL/AlloyDB, unstructured text in MongoDB, semantic search in Qdrant** — mirrors the news architecture and keeps the schema consistent across the two "textual" data sources (news and filings).
***
## :cubes: 2. Product
### 2.1 :puzzle-piece: Core features
The system is decomposed into independently deployable modules connected via the **Cloud Pub/Sub** event backbone. This modularity allows each component to be tested, scaled, and even disabled independently — critical for a system moving real money.
#### 2.1.1 Data Aggregation Engine
* Unifies market data (QuestDB), news (MongoDB/Qdrant), fundamentals, and analyst ratings (Cloud SQL) into a single point-in-time-correct feature store.
* Responsible for time-alignment: ensuring that at the moment a decision is made, only data that would have actually been *available* at that timestamp is used (prevents lookahead bias).
#### 2.1.2 AI Signal Generator
* Consumes the unified feature set and produces a **recommendation object**: `{ticker, direction (long/short/flat), conviction score, suggested horizon, model_version, explanation}`.
* Composed of an ensemble: quantitative model (XGBoost/LSTM on price+fundamentals) + qualitative overlay (LLM-summarized sentiment/news signal retrieved from Qdrant).
* **Never places an order directly** — output is purely advisory and flows into the Risk Management layer via Pub/Sub.
#### 2.1.3 Risk Management / Position Sizer
* Translates a signal + conviction score into an actual position size using rules such as: max % of portfolio per position, volatility-adjusted (Kelly-fraction-capped) sizing, sector/correlation exposure limits, and per-account risk tolerance.
* Enforces hard portfolio-level constraints (max drawdown, max leverage, max daily loss) — this module has **veto power** over the AI signal.
#### 2.1.4 Multi-Broker Order Management System (OMS)
* Core order execution path implemented in **Rust** for latency and correctness guarantees; abstracts broker-specific APIs (IBKR TWS/Gateway API, Zerodha Kite Connect, Moomoo OpenAPI, etc.) behind a canonical internal order interface (`place_order`, `cancel_order`, `get_position`, `get_account_balance`).
* Handles order routing logic: which broker/account an order should go to based on account configuration, asset availability, and cost.
* Owns the full order lifecycle state machine (submitted → acknowledged → partially filled → filled/cancelled/rejected) and reconciles broker-reported fills against internal expectations, publishing every state transition to a Pub/Sub `order_events` topic for audit and downstream consumption.
#### 2.1.5 Backtesting & Simulation Engine
* Replays historical market data (from the GCS Parquet archive), news, and fundamentals through the same Signal Generator and Risk Manager code paths used in production (shared code, not a reimplementation) to ensure backtest/live parity.
* Supports walk-forward validation and Monte Carlo stress testing of strategies before paper or live deployment.
#### 2.1.6 Portfolio & Performance Analytics
* Real-time P\&L, exposure, and risk metrics (Sharpe, Sortino, max drawdown, beta) per account and aggregated across accounts, queryable via **BigQuery** over the archived order/position event history.
* Attribution reporting: which signals/models contributed to which P\&L outcomes.
#### 2.1.7 Notification & Audit Layer
* Immutable audit log (Pub/Sub → GCS/BigQuery sink) of every signal generated, every risk decision (approved/rejected/resized), and every order sent — required for post-hoc compliance review and debugging.
* Alerting (email/SMS/Slack/PagerDuty, triggered via **Cloud Monitoring** alert policies) for circuit-breaker trips, execution failures, and anomalous model behavior.
#### 2.1.8 Feature Summary Table
| Module | Primary Responsibility | Key Interfaces |
| ----------------------------- | -------------------------------------------- | ------------------------------------------------ |
| Data Aggregation Engine | Point-in-time feature store | Pub/Sub consumer/producer, Feast |
| AI Signal Generator | Produce trade recommendations | Reads feature store, writes `signals` topic |
| Risk Manager / Position Sizer | Convert signal → sized order, enforce limits | Reads `signals`, writes `approved_orders` |
| Multi-Broker OMS (Rust) | Execute and track orders across brokers | Broker APIs, writes `order_events` |
| Backtesting Engine | Offline strategy validation | Shared code with Signal Generator/Risk Manager |
| Portfolio Analytics | Performance/risk reporting | Reads `order_events`, `positions` (via BigQuery) |
| Notification & Audit | Alerting and immutable logging | Consumes all topics, Cloud Monitoring |
***
### 2.2 :users-gear: User and account management
#### 2.2.1 Logical Account Model
The system uses a **hierarchical account abstraction** so that "trade this strategy" logic never needs to know broker-specific details:
```
User (Owner)
└── Portfolio Group (e.g., "Retirement", "Aggressive Growth")
└── Broker Account Link (e.g., IBKR Account #1, Zerodha Account #1)
└── Broker-native Account ID (opaque, mapped internally)
```
* **User**: A human owner of the system (or, for a fund-style deployment, a client entity).
* **Portfolio Group**: A logical grouping of one or more broker accounts that share a common strategy allocation and risk budget (e.g., a user may run a conservative strategy on one account and an aggressive one on another).
* **Broker Account Link**: A secure, credentialed connection to a specific account at a specific broker, storing OAuth tokens/API keys (in **Google Secret Manager**, never in application DB), account currency, and margin type.
#### 2.2.2 Multi-Broker, Multi-Account Handling
| Concern | Approach |
| ------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Credential storage** | All broker API keys/OAuth tokens stored in **Google Secret Manager**, never in application databases; short-lived tokens refreshed automatically; access controlled via IAM and audited via Cloud Audit Logs |
| **Unified position view** | OMS normalizes positions across brokers into a common schema (symbol, quantity, average cost, currency) and converts to a base reporting currency for aggregated views |
| **Per-account risk budgets** | Risk Manager enforces limits *per account* (regulatory/margin constraints differ by broker/jurisdiction) as well as *per portfolio group* (aggregate exposure across accounts to the same symbol/sector) |
| **Broker capability differences** | An adapter layer exposes a capability matrix per broker (e.g., supports fractional shares? supports options? supports after-hours?) so the Signal Generator/Risk Manager can gracefully skip or adjust orders that a given account's broker can't support |
| **Currency & jurisdiction handling** | Zerodha (INR, Indian equities) and IBKR/Moomoo (USD/multi-currency, global) require FX-aware P\&L calculation and jurisdiction-specific compliance rules (e.g., PDT rules in the US, intraday margin rules in India) — this logic lives in a `jurisdiction_rules` config layer, not hardcoded |
| **Independent kill-switches** | Each account and each portfolio group can be individually paused/halted without affecting others — critical when one broker connection misbehaves |
| **Role-based access control (RBAC)** | If multiple humans (e.g., you + a partner, or future clients) use the system, enforce RBAC via **Google Cloud IAM** custom roles: viewer, strategy-manager (can adjust risk params), admin (can add/remove broker links) |
***
## :gear: 3. Operations
### 3.1 :road: Execution roadmap
A phased rollout is essential — this system controls real capital and must earn trust incrementally.
#### Phase 0 — Foundations (Weeks 1–4)
* Set up GCP project structure (dev/staging/prod projects), IAM policies, and networking (VPC, private service access for Cloud SQL/Memorystore).
* Write infrastructure as code in **Go using Pulumi**, provisioning GKE, Cloud SQL, Memorystore, Pub/Sub topics, and GCS buckets via CI/CD (Cloud Build or GitHub Actions).
* Provision core datastores: Cloud SQL (PostgreSQL), QuestDB (on GKE), MongoDB Atlas, Qdrant, Memorystore for Redis, Pub/Sub topics.
* Stand up **Google Secret Manager** and basic observability (**Cloud Monitoring**, **Cloud Logging**, **Cloud Trace**) before any business logic is written.
#### Phase 1 — Data Pipelines (Weeks 4–10)
* Build and validate the Market Data Pipeline (live + historical backfill) for a small initial universe (e.g., S\&P 500), in **Go**.
* Build the News Ingestion pipeline (MongoDB landing → Qdrant embeddings) and validate sentiment scoring against a manually labeled sample set.
* Build the Financial Reports Pipeline (EDGAR/sec-api.io) and validate parsed fundamentals against a known company's actual filings.
* **Exit criteria**: A point-in-time-correct feature store can answer "what did we know about AAPL at 9:30 am on date X" accurately.
#### Phase 2 — Signal Generation & Backtesting (Weeks 10–18)
* Build the AI Signal Generator (start with a simple, explainable baseline model — e.g., a gradient-boosted tree on fundamentals + momentum — before adding deep learning/LLM components).
* Build the Backtesting Engine and run the baseline strategy across the full 30-year historical dataset (GCS/QuestDB) with walk-forward validation.
* Iterate on model features, incorporating news sentiment and fundamentals incrementally, measuring out-of-sample performance at each step.
* **Exit criteria**: A strategy shows statistically robust, non-overfit out-of-sample performance across multiple market regimes (bull, bear, high-volatility periods).
#### Phase 3 — Risk Manager & OMS (Weeks 16–22, parallel to Phase 2)
* Build the Risk Management/Position Sizer with hard-coded conservative limits, in Go.
* Build the **Rust** OMS core and broker adapters one at a time, starting with the broker offering the best paper-trading/sandbox environment (IBKR paper trading is a strong first choice due to its mature sandbox).
* Integration-test the full pipeline: signal → risk check → simulated order, without touching real broker connectivity yet.
#### Phase 4 — Paper Trading (Weeks 22–30+)
* Connect the full pipeline to live market data and broker paper-trading accounts (IBKR, then Zerodha/Moomoo sandbox if available).
* Run continuously for a minimum of **2–3 full months**, covering varied market conditions.
* Track live paper P\&L against backtest expectations; large divergence indicates overfitting, data leakage, or execution modeling errors (slippage/latency assumptions).
* Build out full monitoring, alerting, and circuit breakers (Section 3.2) during this phase — they must be battle-tested before real capital is at risk.
#### Phase 5 — Limited Live Capital (Weeks 30–38)
* Deploy with a small, deliberately limited real-money allocation (e.g., 1–5% of intended final capital) on a single account/broker.
* Keep human-in-the-loop approval on every order initially ("propose-then-confirm" mode) before moving to full autonomy.
* Gradually increase position size limits as confidence builds, and no circuit breakers are tripped unexpectedly.
#### Phase 6 — Scaled Multi-Broker, Multi-Account Live Trading
* Onboard additional brokers/accounts incrementally, one at a time, repeating a shortened paper-trading validation per new broker integration.
* Move to fully autonomous execution only after a sustained track record (recommend minimum 6 months) of stable, monitored, limited-capital live performance.
* Establish a regular (weekly/monthly) model review cadence: retraining schedule, performance review, and risk parameter reassessment.
#### Roadmap Summary Table
| Phase | Focus | Capital at Risk | Exit Gate |
| ----- | ------------------------------------ | ---------------------- | ----------------------------------------------------------------- |
| 0 | Infrastructure (GCP + Pulumi/Go) | None | Environments provisioned, CI/CD live |
| 1 | Data pipelines | None | Point-in-time-correct feature store validated |
| 2 | Signal + Backtesting | None (simulated) | Robust out-of-sample backtest performance |
| 3 | Risk/OMS build (Rust execution core) | None (simulated) | End-to-end simulated pipeline passes integration tests |
| 4 | Paper trading | None (broker sandbox) | 2–3 months stable paper performance matching backtest |
| 5 | Limited live | Small (1–5% of target) | No unexpected circuit-breaker trips, human-approved orders stable |
| 6 | Scaled live | Full, multi-broker | 6+ months track record before full autonomy |
***
### 3.2 :tower-broadcast: System monitoring and failsafes
#### 3.2.1 Latency Monitoring
* Instrument every hop of the pipeline (data ingestion → feature computation → signal generation → risk check → order placement → broker ack) with **Cloud Trace** (OpenTelemetry-compatible) to measure end-to-end and per-stage latency.
* Set SLOs per stage (e.g., "tick-to-signal < 500ms", "signal-to-order < 100ms") using **Cloud Monitoring SLO objects**, alerting via Cloud Monitoring alert policies → PagerDuty/Slack when breached.
* Dashboard via **Cloud Monitoring dashboards** (or **Google Cloud Managed Service for Prometheus** + Grafana if the team prefers PromQL), showing real-time latency percentiles (p50/p95/p99) per broker connection and per data vendor.
#### 3.2.2 API Rate Limit Management
* Centralized, **Memorystore for Redis**-backed token-bucket rate limiter per (vendor, API key) pair, shared across all ingestion workers to prevent distributed over-limit calls.
* Proactive throttling: back off automatically as usage approaches a configurable percentage (e.g., 80%) of the vendor's published limit, rather than waiting for 429 responses.
* Per-vendor circuit breaker: if a vendor returns sustained errors or rate-limit rejections, automatically fail over to the secondary data vendor (Section 1.2.5) and alert engineering via Cloud Monitoring.
#### 3.2.3 Hard Circuit Breakers (Preventing Rogue AI Trades)
Circuit breakers must be **deterministic, rule-based, and entirely independent of the AI model** — they cannot be a "smarter model watching the model," or a bug in one compromises both. In this architecture, the circuit breakers live inside the **Rust** OMS/Risk boundary specifically because that code path is the one held to the highest correctness bar.
| Circuit Breaker | Trigger Condition | Action |
| --------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Max daily loss** | Portfolio (or per-account) drawdown exceeds a configured % in a single day | Halt all new order submission for the affected account/portfolio; existing positions may be auto-flattened per policy |
| **Max order size** | Any single proposed order exceeds a hard-coded max notional or % of portfolio | Order auto-rejected before reaching the broker; requires manual override to proceed |
| **Max order frequency** | Orders submitted per symbol/account exceed a rate threshold in a rolling window | Throttle/reject further orders (protects against a runaway loop or feedback bug) |
| **Position concentration limit** | Any symbol or sector exceeds a max % of portfolio | New orders that would breach the limit are rejected; existing breaches trigger an alert (not necessarily forced liquidation) |
| **Model confidence floor** | Signal conviction score below a minimum threshold | Signal is discarded before reaching the Risk Manager |
| **Model/data staleness check** | Feature data used for a signal is older than an allowed freshness window | Signal generation paused; alert raised; prevents trading on stale/broken data pipelines |
| **Anomalous price/volume sanity check** | Proposed trade references a price wildly divergent (e.g., >X%) from last known good market data, or in an illiquid/halted symbol | Order rejected; flagged for manual review (guards against bad data corrupting decisions) |
| **Global kill switch** | Manual, always-available override | Immediately halts all order submission across all accounts/brokers; human-operated, independent of the main application (e.g., a separate lightweight Cloud Run service or even a broker-side "disable API trading" toggle) |
#### 3.2.4 Reconciliation & Drift Detection
* Continuous reconciliation job (Go, scheduled via Cloud Composer) comparing internal position/balance records against each broker's actual reported state; any mismatch beyond a tiny tolerance triggers an immediate Cloud Monitoring alert and halts further trading on that account until resolved.
* Model drift monitoring: track live signal distribution and prediction accuracy over time versus backtest-era baselines (queryable via BigQuery); significant drift triggers a review before continued autonomous operation.
#### 3.2.5 Incident Response
* Defined runbooks for each circuit breaker type (who is paged, what manual steps are taken, how trading is safely resumed).
* Post-incident review process: every circuit-breaker trip or reconciliation mismatch is logged (Cloud Logging) and reviewed, with findings feeding back into risk parameter tuning.
***
### :diagram-project: Appendix: reference architecture
```
┌─────────────────────────────────────────────────────┐
│ Google Cloud Pub/Sub │
└─────────────────────────────────────────────────────┘
▲ ▲ ▲ │ │ │
│ │ │ ▼ ▼ ▼
┌──┴───┐ ┌────┴────┐ ┌────┴─────┐ ┌───────────────┐ ┌──────────┐ ┌──────────┐
│Market │ │ News │ │Fundamentals│ │ AI Signal Gen │ │ Risk │ │ Multi- │
│ Data │ │Ingestion│ │ Pipeline │───▶│ (Python/Vertex │─▶│ Manager │─▶│ Broker │
│Ingest │ │(Go, NLP/│ │ (Go, EDGAR/│ │ AI) │ │ (Go) │ │ OMS │
│(Go) │ │FinBERT) │ │sec-api.io) │ └───────────────┘ └──────────┘ │ (Rust) │
└───┬───┘ └────┬────┘ └─────┬──────┘ └────┬─────┘
│ │ │ │
▼ ▼ ▼ ▼
QuestDB MongoDB Atlas Cloud SQL / IBKR / Zerodha /
(GKE/Compute) + Qdrant AlloyDB Moomoo / others
│
▼
GCS (Parquet, cold archive) ──▶ BigQuery (analytics) / Dataflow (Pub/Sub archival sink)
```
***
---
# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.
## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.
Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:
```
GET https://docs.nuralex.tech/technology-stack.md?ask=&goal=
```
`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.