Feature extraction
Engines partial
Alongside the detection engines, Heimdall computes a small set of statistical and structural features for every request. These are the numeric signal that feeds telemetry, the data lake and (later) the offline ML models — a compact, privacy-safe description of a request's shape rather than its content.
What is measured
Per request, the extractor computes the maxima across all parameters:
| Feature | What it captures |
|---|---|
| Entropy | Shannon entropy of a value — high entropy flags encoded blobs, tokens, obfuscation. Computed in fixed-point (no floats), for values of at least 16 bytes, scaled and capped. |
| Max depth | Deepest parameter-name nesting (. / [ count) — deep structures signal nested-object abuse. |
| Max tokens | Longest token run in a value — a rough complexity measure used by the SQL/XSS tokenizers. |
| Non-ASCII ratio | Share of non-ASCII / non-printable bytes — a smuggling and obfuscation tell. |
| Param count | Number of parameters — cheap DoS / mass-assignment signal, emitted as a single GDPR-safe byte. |
Design
Feature extraction is zero-allocation and does no floating-point math — the
Shannon entropy is a fixed-point log2 over a byte-frequency table. The tokenizer
it shares with the SQL and XSS engines emits token runs through a callback so no
intermediate slices are allocated.
The features are statistical descriptors, not identifiers: they describe the distribution and structure of a request, never its literal content, which is why they are safe to store in the anonymized telemetry stream and the data lake.
Where features go
- Telemetry — the feature vector is written into every event (
feat_entropy,feat_depth,feat_tokens,feat_nonascii,param_count), so the Attack Explorer and the data lake can slice on request shape. - Endpoint Intelligence — per-endpoint baselines of these features are what "normal" is measured against.
- Offline ML — the same vector is the input to anomaly detection and clustering.
Status
The tokenizer, fixed-point Shannon entropy, non-ASCII/non-printable ratio, nesting depth and parameter count are all implemented, emitted to telemetry, zero-alloc and fuzzed. Two things are still evolving: the character-class histogram is still coarse, and using the feature vector as a real-time detection input (as opposed to a telemetry/ML signal) is deferred — today features inform, they do not gate.