Encrypted traffic analysis: Detecting threats without decrypting packets

Deep packet inspection built a generation of network security products on an assumption that stopped holding: that a sensor on the wire could read what was inside the packets. Google's HTTPS Transparency Report tracked encrypted page loads in Chrome rising from roughly 30 to 45 percent in 2015 to the 95 to 99 percent range by 2020, where adoption has since plateaued. Chrome 154, scheduled for October 2026, turns on "Always Use Secure Connections" by default, which will press on the remainder. Malware families adopted encrypted channels years ago, and TLS 1.3 removed several of the fields that inspection tools had been reading in the clear.

Most of this page describes encrypted traffic analysis as a discipline, which is broader than what any single product implements. Handshake fingerprinting in particular requires packet visibility. Where the page describes NetFlow Analyzer specifically, it says so, and that section is confined to the end.

The useful response is a shift in what you look at. Encryption protects the contents of a conversation. It leaves the shape, timing, endpoints, volume, and handshake characteristics of that conversation fully observable, and a surprising amount of detection lives in those properties.

The visibility problem encryption created

Encryption is a security win that happens to be an inspection loss. Three developments compounded it.

Ubiquitous TLS: Given the adoption figures above, unencrypted web traffic is now the exception, and an exception worth investigating on its own.

Malware adoption: Command and control channels moved to TLS for the same reason legitimate applications did. Signature matching on payload no longer sees implant traffic.

Protocol changes: TLS 1.3 encrypts the certificate exchange, which previously exposed subject names, issuers, and validity dates in the clear. Encrypted Client Hello, where deployed, conceals the Server Name Indication field that inspection tools relied on to identify the destination service. QUIC moves transport handshake details into an encrypted envelope over UDP.

Why decryption is a poor default answer

TLS interception, sometimes called break and inspect, is technically available and carries costs that many teams underestimate.

• Compute and latency: Terminating and re-establishing TLS at scale requires meaningful hardware, and it adds latency to every intercepted session.
• Certificate pinning breakage: Pinned applications, including many mobile apps, banking clients, and update mechanisms, fail outright when intercepted. The exception lists grow indefinitely.
• Privacy and legal exposure: Intercepting employee traffic to healthcare, financial, and legal services creates data handling obligations and, in some jurisdictions, direct legal risk.
• A new concentration of risk: The interception point holds plaintext for the whole organization, which makes it an extremely attractive target.
• Diminishing coverage: ECH and pinning steadily reduce what interception can actually read, so the cost stays constant while the return declines.

Selective interception at a small number of high-risk egress points, with documented exclusions, is defensible. Universal interception is a large, growing cost for shrinking coverage.

What stays visible when payloads do not

An encrypted flow still exposes a substantial amount of structure.

Flow records carry most of the left column natively, which is why encryption changed the value of flow analysis relatively little compared with what it did to payload inspection.

TLS handshake fingerprinting

Before encryption begins, the client sends a ClientHello in the clear. Its contents reflect the TLS library that built it: supported versions, cipher suites offered, extensions present, elliptic curves, signature algorithms, and ALPN values. Different software stacks produce distinguishable combinations.

JA3, proposed by researchers at Salesforce in 2017, hashed a fixed ordering of those ClientHello fields into a single fingerprint string. It saw wide adoption and then wide evasion. GREASE values and deliberate extension reordering by browsers caused a single client to produce many different JA3 values, and adversaries could randomize their own fingerprints cheaply.

JA4 and the JA4+ suite, released by FoxIO, addressed those limitations. JA4 sorts cipher and extension lists before hashing, so ordering alone no longer changes the value. The fingerprint is structured in three underscore-separated sections (a_b_c), with the first section human readable: transport type (TLS over TCP, QUIC, or DTLS), the highest advertised version ignoring GREASE, SNI presence, counts of ciphers and extensions, and the first ALPN value. That structure allows partial matching, so an analyst can hunt on the a and c sections while ignoring b, which is how tracked actors who rotate a single cipher are still identifiable. JA4+ extends the family across TLS server responses (JA4S), HTTP (JA4H), certificates (JA4X), SSH (JA4SSH), TCP (JA4T), and timing (JA4L).

Licensing differs across the family. JA4 itself is released under a BSD 3-Clause license, while other JA4+ methods use the FoxIO license, which has separate terms for monetized use. Check the license covering the specific method before building it into a product.

A fingerprint groups implementations rather than establishing identity. Many clients share a TLS library, an automated client can imitate a browser's fingerprint deliberately, and the value changes as software updates. Treat fingerprints as a pivot and a grouping mechanism rather than as attribution.

Behavioral signals that survive encryption

These detections care nothing about payload.

Beacon periodicity: Plot connection start times between an internal host and one external destination. Automated check-ins cluster at near-uniform intervals. Jitter widens the distribution and rarely erases the pattern, because implants still need bounded check-in windows.

Directional asymmetry: Client browsing downloads far more than it uploads. Sustained inversion of that ratio is the exfiltration shape, encrypted or otherwise.

Duration and volume mismatch: A session lasting hours while moving very little data suggests a maintained tunnel rather than a transfer. Interactive shells look like this.

Destination novelty: A destination absent from your entire retention history, now receiving sustained traffic from one internal host, deserves attention regardless of what the traffic contains.

Protocol on the wrong port: TLS handshakes on ports where TLS has no business, or non-TLS traffic on 443, both indicate tunneling.

DNS behavior: High volumes of NXDOMAIN responses, very long query names, or queries to newly registered domains precede many encrypted C2 sessions. DNS remains largely observable in most enterprise environments.

The ransomware detection page applies these same signals to a specific attack chain.

What TLS 1.3, ECH, and QUIC removed

Being precise about the losses matters more than lamenting them.

• Certificate visibility: TLS 1.3 encrypts the server certificate. Detections based on issuer, subject, or validity period stop working on 1.3 sessions.
• SNI: Encrypted Client Hello conceals the requested hostname where both client and server support it. Deployment is uneven, so SNI remains available on a meaningful share of traffic today and should be treated as a declining asset instead of a permanent one.
• Handshake distinctiveness: TLS 1.3 handshakes are more uniform across clients, which reduces fingerprint entropy. JA4's normalization helps and does not fully restore what 1.2 offered.
• Transport metadata: QUIC encrypts most of the transport handshake and runs over UDP 443, so techniques built on TCP flag analysis do not apply.

What none of these removed: endpoints, volume, direction, timing, and frequency. Behavioral detection is the layer that keeps working as protocol-level visibility declines, which is a reasonable argument for weighting investment there.

Building a detection stack that assumes encryption

1. Treat flow metadata as the foundation. It survives every protocol change above.
2. Collect DNS. Resolver logs pair with flow to identify destinations even when SNI is unavailable.
3. Add handshake fingerprinting where you have packet visibility. Zeek supports JA4 through a maintained package, with fingerprints logged alongside connection and SSL records.
4. Enrich destinations. ASN, hosting provider, registration age, and threat intelligence turn an opaque IP into context.
5. Invest in baselines. Behavioral detection is only as good as the definition of normal behind it. See how to baseline network traffic.
6. Reserve decryption for narrow, justified cases, with documented scope and exclusions.

Limitations

Encrypted traffic analysis narrows the gap encryption created. It does not close it.

• You can measure the amount of data leaving the network, but not its contents.
• Low and slow exfiltration hidden inside legitimate high-volume services is genuinely difficult, because the traffic resembles normal SaaS usage in every observable dimension.
• Fingerprints are spoofable by capable adversaries who choose to blend in.
• Behavioral methods produce false positives that require analyst time, and tuning is continuous instead of one-time.
• Sampled flow export undercounts small flows, which is exactly where beaconing lives.

Layering remains the answer. Endpoint telemetry sees plaintext before encryption occurs. Identity and access telemetry sees who authenticated. Network telemetry sees every movement between systems. None of the three is sufficient alone.

Encrypted traffic analysis with NetFlow Analyzer

ManageEngine NetFlow Analyzer works on flow metadata, so its detection capability is unaffected by whether session contents are encrypted.

FAQs