Finding the duplicate that shares no words.

A register full of near-misses

A duplicate isn't just clutter. It splits the history of one issue across two rows, so remediation work on one is invisible from the other, and your open-findings count is wrong in a way nobody can explain. The fix people reach for, telling everyone to search before filing, fails on exactly the cases that matter: you can't search for a phrasing you didn't think of.

The mirror image of a duplicate is a relation. Take “backups are written to a NAS in the server room, attached to the corporate LAN” and “no policy requirement for offline or immutable copies of recovery media”. One is a technical finding, one is a governance gap, and they share no words at all. Together they are your ransomware recovery plan failing twice over: the copy is reachable from the network that gets encrypted, and nothing in your policy set requires one that isn't. Nobody links them, because the two live in different parts of the register and were probably filed by different people months apart.

So the target is one pass, at creation, that answers two questions. Is this the same finding as something already here? And does it connect to anything already here?

Keyword matching, over hashes

A finding's title, affected systems and body are encrypted at rest, each organisation under its own key, which rules out the obvious implementation. We can't decrypt the register on every draft, and keeping a plaintext copy on the side to compare against would undo the encryption for anyone who gets hold of the database. So the first pass compares hashes instead: every finding's vocabulary is stored as per-org token hashes, never as words. Equal words produce equal hashes, so matching is set intersection with nothing decrypted anywhere. That layer has its own write-up in searching data you can't read.

Scoring is mostly token overlap, over the size of the smaller of the two sets. Asymmetric on purpose: a long, detailed draft shouldn't be penalised for saying more than the terse finding it duplicates. How far the ISO controls agree contributes a little on top, weak evidence alone but a useful tiebreak.

Everything above a score gate goes forward as a candidate, and that gate sits far lower than instinct says it should. Measure a real pair and you see why: a draft carries a lot of tokens, and a genuinely related finding shares only a handful of them. Put the gate where duplicates comfortably score, which is where you'd naturally put it, and you never see relations at all. The gate is not the verdict. It only decides who gets looked at, and being stingy there costs you the interesting half of the feature.

What this layer cannot do is find either example from the top of this article. Nothing weighted on shared words can. That's not a tuning problem, it's the structural blind spot of lexical matching, and it needs a different kind of index.

A second index, for meaning

When semantic matching is enabled and an embedding key is available, eligible findings also get an embedding: their text turned into a 1024-dimension vector by Mistral's mistral-embed, where similarity is the angle between two vectors rather than any shared characters. “Unlimited login attempts” and “no authentication throttling” land close together because the model was trained on what they mean. Nothing is replaced. The keyword layer keeps the job it is unbeatable at (a CVE ID, a port, a hostname, a ticket reference), and the vector layer widens what the funnel gets to see.

Two deliberate choices sit behind that. Embeddings always go through our own Mistral account, never an org's bring-your-own-key credential, because a vector minted by a different model isn't comparable to the ones already stored and mixing them would silently poison the pool. That account carries a DPA and a zero-data-retention agreement: text in, numbers out, nothing kept. And it's available on every plan, including Free, when semantic matching is enabled and its dedicated key is available, because duplicate detection that reads meaning is the thing a new register notices first, and that's a strange thing to withhold.

Now the choice that made this layer actually work, which isn't the model or the dimension count. Similarity compares texts, and it is only as meaningful as those texts are comparable. What people type into a register is the least comparable material you will ever handle: a one-line note tapped out on a phone, a pasted scanner block, an OSINT tool dump, three pages lifted from a pentest report. Two accounts of the same issue can differ far more in length and register than they do in meaning, and a vector picks up that difference just as happily as the real one. Compare those directly and you're comparing apples to bananas.

The two indexes deliberately read different text. The blind keyword index stays on the raw title, affected systems, and body so exact identifiers survive; the semantic vector uses the AI-enriched summary, falling back to the raw observation when no summary exists. That gives the lexical layer exact recall and the meaning layer comparable prose.

Two agents make the call

Retrieval only produces suspects. The top candidates get decrypted once, then two AI agents run concurrently over that same set, asking different questions:

• Is this the same finding? Sameness is about scope and identity, not similarity of wording. Two unpatched hosts in the same subnet read almost identically and are two findings.
• Is it connected to any of them? Shared root cause, same asset, one issue amplifying another, same trust chain, a remediation dependency. The harder judgement of the two, and it runs on a stronger model tier for that reason.

Two jobs, two agents, on purpose. Each is tuned for its own question, and each fails alone: a relations failure still leaves you a duplicate verdict, and a duplicate failure still leaves you the candidates to judge by hand.

Directional links, with the reasoning kept

“Related” on its own is nearly useless, so the vocabulary is small and directional. Each link is stored once and read from both ends, so the reverse phrasing is always derived rather than kept as a second row that can drift.

Suggestions arrive on the review step as pre-ticked checkboxes, each with the model's one-line reason for the link. Accepting is the default and opting out is one click, which is the right way round: an unwanted link costs you a tidy-up, a missed one costs you a connection nobody ever makes again. When you accept, the reasoning is saved with the link, so six months later it explains itself instead of sitting there as an unexplained arrow. Bulk imports get the same treatment per row.

For the record, the pair from the top of this article lands as “worsens”: the LAN-attached backup target worsens the missing offline-media policy, and reads as “worsened by” from the policy gap's side.

The pattern, portable

• ✓Run lexical and semantic retrieval side by side. One is unbeatable on identifiers, the other is the only thing that finds a rephrasing, and neither substitutes for the other.
• ✓Never compare scores across the two layers. An overlap ratio and a similarity are different units, and two unrelated findings sit surprisingly high on similarity, so a threshold tuned on one will misfire badly on the other.
• ✓Feed the comparison uniform text. Vectors reward comparability, so compare the normalised write-ups and keep exact identifiers in the keyword layer.
• ✓Retrieve wide, decide narrow. Set the candidate gate low enough to catch the interesting cases, and let a model with the decrypted text make the call.
• ✓Calibrate a classifier from both sides. Telling a model to be conservative gets you silence, not precision.

Two indexes, a shortlist, two agents, and a strict rule about not mixing their numbers. That's the whole thing. The result is that filing a finding tells you what it duplicates and what it connects to before you save it, which is when it's cheap to act on.