> For the complete documentation index, see [llms.txt](https://statalpha.gitbook.io/statalpha-docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://statalpha.gitbook.io/statalpha-docs/stats-cold-path.md). # Operations STAT-42 protects expensive, materializing statistics requests with Redis singleflight leases and emits one structured `stats_cold_path` event with phase timings. API workers enqueue these events locally and write them to a Redis Stream; the independent `log-forwarder` service sends batches to Elasticsearch. ## Deployment order 1. Configure `APP_ENV`, `CACHE_PREFIX`, `REDIS_URL`, the Elasticsearch log settings and the `REDIS_LOG_*` values from `.env.example`. 2. Start the environment-specific `log-forwarder` service. 3. Deploy the API with `LOG_TO_ELASTICSEARCH=true`. 4. Confirm that the namespaced `logs:elasticsearch` Redis Stream is consumed and `stats_cold_path` events arrive in the environment's log index. 5. Deploy with `DERIVED_TABLES_ENABLED=true` and `DERIVED_READ_THROUGH_ENABLED=false`. This creates and fills V2 tables but keeps request reads on their established path. 6. Rebuild and validate V2 coverage in bounded symbol/date batches. 7. Enable `DERIVED_READ_THROUGH_ENABLED=true` only after QuestDB WAL rows are queryable and coverage has been compared with the source bars. The local API queue is bounded and never performs Redis network I/O in a request thread. A transient Redis failure is retried by the background shipper. The Elasticsearch consumer uses Redis consumer groups, acknowledges only successful bulk requests and sends repeatedly failing records to the configured dead-letter stream. Redis Stream IDs are used as Elasticsearch document IDs so delivery retries are idempotent. ## Consistent QuestDB mutations All derived-table DDL, inserts and administrative deletes use QuestDB's PGWire interface. HTTP `/exec` remains read-only. Writes share one bounded batch policy: `QUESTDB_WRITE_BATCH_SIZE` limits rows and `QUESTDB_WRITE_MAX_BATCH_BYTES` limits the approximate payload size. There is deliberately no automatic mutation retry; the WAL-backed V2 tables instead deduplicate exact result dimensions when a caller explicitly reruns a rebuild. The active V2 tables are: | Data | Default table | Designated timestamp | | ----------------------- | ------------------------------------ | -------------------- | | UTC daily bars | `daily_bars_v2` | `session_start_utc` | | US regular-session bars | `session_bars_v2` | `session_start_utc` | | Session aggregates | `session_aggregated_bars_v2` | `timestamp` | | Aggregate coverage | `session_aggregated_bar_coverage_v2` | `trade_date_ts` | | ORB results | `orb_daily_results_v2` | `trade_date_ts` | | GAP results | `gap_daily_results_v2` | `trade_date_ts` | The table names, PGWire connection and batch limits are configured with the variables documented in `.env.example`. ## V2 rebuild V2 tables are WAL-backed, partitioned by their designated timestamps and deduplicated on all dimensions that affect the stored result. Rebuild canonical rows before enabling read-through: ```bash APP_ENV=prod \ CACHE_PREFIX=prod \ DERIVED_TABLES_ENABLED=true \ DERIVED_READ_THROUGH_ENABLED=false \ ORB_DERIVED_RESULT_TABLE=orb_daily_results_v2 \ GAP_DERIVED_RESULT_TABLE=gap_daily_results_v2 \ DAILY_BARS_TABLE=daily_bars_v2 \ SESSION_BARS_TABLE=session_bars_v2 \ SESSION_AGGREGATED_BARS_TABLE=session_aggregated_bars_v2 \ SESSION_AGGREGATED_COVERAGE_TABLE=session_aggregated_bar_coverage_v2 \ python scripts/rebuild_derived_stats.py \ --symbols AAPL,MSFT \ --start-date 2024-01-01 \ --end-date 2026-07-13 \ --table-type all ``` `--table-type` can target `daily_bars`, `session_bars`, `session_aggregated_bars`, `orb_daily_results` or `gap_daily_results`. The aggregate run also writes its matching coverage rows. Run rebuilds in bounded symbol/date batches. After each batch, wait until QuestDB has applied its WAL, then compare symbol/date coverage and representative values with the raw bar tables before switching reads. GAP and ORB V2 rows are independent from their legacy tables. A repeated exact rebuild is safe because the request dimensions are part of the deduplication keys. ## Rollback First set `DERIVED_READ_THROUGH_ENABLED=false` and restart the API. This returns requests to raw-bar calculation without disabling V2 writes or the Redis singleflight protection used by response materialization. Keep the V2 table names configured so writes remain consistent while a read issue is investigated. Only point an individual table variable back to a legacy table when a table-level rollback is explicitly required; legacy tables do not provide the same WAL and deduplication guarantees. Prewarming is intentionally not part of STAT-42 and is tracked separately in STAT-48; it is not required for live deployment. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://statalpha.gitbook.io/statalpha-docs/stats-cold-path.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.