Machine learning in network traffic analysis: What it adds and where it fits

Machine learning in network traffic analysis uses statistical and ML models to learn what normal traffic looks like for a given network, flag deviations from it, classify traffic where payload inspection is unavailable, and rank anomalous activity so analysts see the most interesting cases first.

Machine learning has been on network security datasheets long enough that the word now carries almost no information. Some of what it describes is genuine and useful. Some of it is a rolling average with a confident label.

This page separates the two. It covers which detection problems ML measurably improves, which model families suit which task, why the mathematics of rare events makes precision harder than accuracy figures suggest, and the specific questions that reveal whether a product's ML is doing real work.

Most of this page describes what machine learning can do in network traffic analysis as a discipline. That is broader than what any single product implements, including ours. Where the page describes NetFlow Analyzer specifically, it says so, and that section is confined to the end. Treat everything before it as the general state of the field.

What problem ML actually solves

The problem is that a network has many normals and a static threshold encodes only one.

A link at 400 Mbps is unremarkable during Monday morning login and interesting at 3 AM Sunday. Set the threshold high enough to avoid the Monday alert and you miss the Sunday one. Set it low enough to catch Sunday and Monday generates noise until someone mutes the rule.

Teams respond by writing conditional rules: different thresholds by hour, by day, by device group. That works up to a point, and the point arrives quickly. A network with 500 device groups, weekly and monthly cycles, and per-application patterns has more conditions than anyone will maintain by hand.

Machine learning's core contribution is learning those conditions from data and updating them as the network changes. That is a real contribution, and it is narrower than the marketing suggests.

Where ML clearly helps

1. Multivariate anomaly detection: A host whose byte volume is normal, whose peer count is normal, and whose port distribution is normal may still be anomalous in the combination of the three. Static rules examine dimensions one at a time. Models examine the joint distribution, which is where subtle compromise shows up.

Concretely, a file server sits at the 62nd percentile of its own upload volume, the 71st percentile of distinct peer count, and the 68th percentile of port entropy. No single-dimension threshold fires, because none of those is remarkable alone. The joint position of those three values sits in the top 0.2% of that host's own history, because this server has never previously combined elevated upload with elevated peer diversity. That combination is what a multivariate model surfaces and what three separate thresholds structurally cannot.

2. Adaptive baselining: Networks change continuously through onboarding, application rollouts, seasonal cycles, and architecture shifts. A learned baseline that updates on a rolling window tracks that drift. A hand-set threshold decays from the day it is written.

3. Alert prioritization and clustering: Grouping related alerts into a single incident, ranking by learned severity, and suppressing repeated benign patterns. This is frequently the highest practical return, because analyst attention is the scarcest resource in the pipeline.

4. Traffic classification under encryption: When payload inspection is unavailable, models trained on flow shape (packet size distributions, timing patterns, directional ratios) can attribute traffic to application categories with useful accuracy. The techniques available to any given product depend on its telemetry and implementation. See network traffic classification for how this fits alongside port mapping, DPI, and device-assisted identification.

Where rules still win

Known-bad indicators: If a destination IP appears on a threat intelligence feed, match it. A model is unnecessary overhead for a lookup.

Policy violations: Traffic crossing a segment boundary that policy forbids is a violation by definition. There is no probability involved and no benefit to introducing one.

Compliance controls: Auditors want deterministic, explainable evidence. "The model assigned a low anomaly score" is a weak answer where "no traffic matched this forbidden pattern" is a strong one. Network traffic analysis for compliance covers this distinction.

Well-characterized attack signatures Ransomware lateral movement on SMB and RDP between workstations has a shape you can write down. A rule catches it, explains itself, and tunes predictably.

Mature detection programs run rules and models together, with rules covering the known and models covering the unfamiliar.

ML or rules: A quick reference

Model types and what they fit

Supervised learning is rarely available for attack detection in enterprise networks, because almost nobody has a labeled corpus of their own attacks. That pushes real deployments toward unsupervised and semi-supervised methods, which detect unusual and not malicious. And the most useful production systems tend to be ensembles of simple models with good features. Single sophisticated models underperform them in practice.

That gap between unusual and malicious has a name in the research literature. Sommer and Paxson's 2010 IEEE Symposium on Security and Privacy paper, "Outside the Closed World: On Using Machine Learning for Network Intrusion Detection," calls it the semantic gap, and argues it is why anomaly detection succeeds in domains where the goal is finding similarity and struggles in intrusion detection, where the goal is finding difference that also happens to be hostile. The paper is 16 years old and its argument has aged well.

Why accuracy misleads: the base rate problem

The base rate problem is the effect of extreme class imbalance on precision: when the thing you are detecting is very rare, even a very low false positive rate produces far more false alerts than true ones. Axelsson's 1999 ACM CCS paper, "The Base-Rate Fallacy and its Implications for the Difficulty of Intrusion Detection," established the point for this field, and it is routinely omitted from vendor material.

The arithmetic:

• 10,000,000 flows per day
• 100 genuinely malicious flows, a base rate of 0.001%
• 99% true positive rate
• 1% false positive rate
• → 99 true positives
• → 99,999 false positives
• → precision of roughly 0.1%

An analyst reviewing that queue encounters about one real detection per thousand alerts. Note that the 1% false positive rate is a deliberately generous assumption. Real deployments frequently do worse, and the result is still what you see above.

None of that argues against ML. It argues for three things:

1. Aggregation before alerting: Alert on hosts or sessions, never on individual flows, which reduces the denominator by orders of magnitude.
2. Correlation: Require multiple independent signals before escalation. A host with an unusual peer count, an unusual upload volume, and a first-seen destination is far more interesting than a host with any one.
3. Ranking, not binary alerting: A prioritized queue of the 50 most anomalous hosts today is workable. A binary alert stream at this base rate is not.

Whenever a vendor quotes accuracy without stating base rate and precision, ask for precision at your flow volume.

What features does ML use in network traffic?

Features are the derived measurements a model reads, and their quality determines results more than model choice does.

Useful features in flow-based detection are derived from the raw fields rather than read straight off them:

Note how many of these are relative to the host's own history. Global features produce models that flag every server for behaving unlike a workstation. Role-relative features produce models that flag a server behaving unlike itself, which is the actually interesting event. That dependency is why baselining is a prerequisite instead of an alternative to ML, as covered in how to baseline network traffic.

Operational challenges of ML-based detection

These are the realities nobody puts in the datasheet.

Cold start: Models need history. Expect two to six weeks before output is broadly trustworthy, and note that the figure is rarely uniform across a product. Detection logic tends to be layered: some rules are deterministic and fire from day one, some need roughly a week of observation, and some need considerably longer. Ask a vendor for the breakdown by rule rather than a single number for the system. Where the long tail lands depends less on model convergence than on the longest cycle you need represented, so a network with month-end batch processing needs a full month minimum whatever the model does.

Drift: Networks change and models degrade, a phenomenon known as concept drift. Retraining cadence and drift monitoring are ongoing operational commitments rather than one-time setup.

Poisoned baselines: If an attacker is already present during the learning window, their activity becomes part of normal. This is a real and underdiscussed risk, and it argues for combining learned baselines with rules that do not depend on history.

Adversarial evasion: An attacker who knows a behavioral model is running can shape traffic to stay inside it: pacing transfers below learned volume deviation, matching the timing profile of a legitimate application, and routing through cloud destinations the environment already talks to. Behavioral detection raises the cost of evasion and does not eliminate it.

Explainability: An analyst who cannot see why a host was flagged cannot triage it. Score-only output produces alerts that get closed unread. Ask to see the contributing features.

Feedback loops: Where analysts cannot mark a detection as expected and have future alerting account for it, alert volume only grows, and the usual end state is a muted channel within a quarter.

What it costs to run: Three line items that rarely appear in a quote: compute for training and inference, storage for the feature history the model reads, and analyst time for tuning and feedback in the first quarter. The third is the one teams underestimate, and it needs someone who knows the network well enough to judge what is normal.

Eight questions to ask a vendor claiming AI

1. Which specific detections use a model, and which use thresholds? Expect a named list. A philosophical answer is a non-answer.
2. Supervised or unsupervised? If supervised, what was the training data and how does it relate to my network?
3. How long is the learning period, and what happens during it?
4. What precision should I expect at my flow volume, given my base rate?
5. When a host is flagged, which features contributed and by how much?
6. How do analysts provide feedback, and how does the model incorporate it?
7. What is the retraining cadence, and how is drift detected?
8. Where does inference run, and does my flow telemetry leave my environment?

A vendor who answers all eight concretely is describing a system built by people who have operated one. Vague answers to questions 4, 5, and 8 are the most reliable warning signs. Question 8 in particular tends to surface late in procurement, when a privacy or regulatory review discovers that traffic metadata is being processed outside the estate.

Behavioral analytics in NetFlow Analyzer

Answering the eight questions above for our own product, since a page that sets a standard should be judged against it.

FAQs