Telemetry & data lake

Engines live

Every request Heimdall sees — clean or flagged — produces exactly one event. That stream is what powers the Attack Explorer, and it is the raw material for the offline ML data lake.

One event per request

The event is a fixed-size, allocation-free record. It carries:

• a pseudonymized client hash (never the IP), a 2-byte country code
• the site the request was routed to, so multi-site installs can tell which site a rule fired on (operator configuration, not personal data)
• method, version, risk totals (risk, finding_risk, anomaly_risk)
• the effective action and the would-be action (action, would)
• the feature vector (feat_entropy, feat_depth, feat_tokens, feat_nonascii, param_count)
• up to eight findings (engine, category, rule, risk, confidence)
• a bounded path and, for flagged requests only, a bounded evidence span (raw + normalized + offsets)

Privacy by design

The telemetry stream is pseudonymized so it is safe to store and analyze:

• The client IP is HMAC-hashed with a configured key into a uint64 and never stored in the clear; the raw address never enters the event.
• Location is reduced to an ISO-2 country code — no finer.
• Evidence capture is gated (-capture-evidence) and only fires for flagged requests, capturing a bounded snippet of the attacker's payload, mapped back to the raw bytes via the normalization offset map. Clean traffic never has content captured.

The async sink

Emitting an event must never slow down a request. The sink is a non-blocking channel: the request path enqueues and moves on; a background writer JSON-encodes events into a buffered writer with a hand-rolled JSONL encoder (no encoding/json on the hot path). If the channel is full, events are dropped and counted rather than blocking the proxy — availability over completeness.

Output is one JSON object per line (events.jsonl), which is both what the control plane tails for the live Attack Explorer and what the ingest pipeline ships to the data lake.

Into the data lake

A checkpointed ingest process tails the JSONL file and batches rows into ClickHouse over HTTP (FORMAT JSONEachRow), resuming from its last acked byte offset so it is at-least-once and restart-safe. The schema is a single source of truth shared by the engine, the ingester and the DDL, and is conformance-tested so the three can never drift.

Status

The event schema, HMAC pseudonymization, the async JSONL sink and the checkpointed ClickHouse ingest are built and conformance-tested. The data-lake consumers — endpoint learning, anomaly detection, clustering, the LLM analyst — are the offline-ML frontier: some exist as Enterprise scaffolding, most are still ahead. See Intelligence plugins.