The same mug entered twice: once by the buyer with a brand and a capacity, once by the warehouse with a description and a weight. No shared id, no shared spelling, and — the part that defeats most tooling — barely any shared fields. You need the pairs that are the same thing, and a reason to believe each one.
Two records of one product rarely carry the same fields. That is the whole difficulty, and it is where single-number similarity goes wrong.
Two rows that share only a colour look identical on the evidence they share. A single score cannot tell that apart from two rows that agree on nine properties.
Pad the gaps with zeros or empty strings and every sparse record drifts towards every other sparse record — duplicates and unrelated rows alike.
Compare only rows sharing a brand and you miss the duplicate whose brand was never filled in. Compare everything and it is quadratic.
One Space over the catalogue, one batched query per object, and two numbers per pair that separate "alike" from "alike on almost nothing".
What counts as suspiciously alike depends on the catalogue. Hilliance learns the distribution of every number, the vocabulary of every text property, which properties are informative and which ones restate each other — from the catalogue you are deduplicating.
That matters here: two products sharing a common colour is nothing; sharing an unusual capacity is something, and only the population can tell you which is which.
curl -X POST localhost:8080/spaces \ -H 'content-type: application/json' -d '{ "name": "catalogue", "id_field": "sku", "objects": [ … every row, duplicates included … ] }'
Search by id and the object itself is excluded from its own results. Batch those queries — up to 10,000 per request — and the engine encodes them in parallel while each keeps its own pruning.
No blocking keys to choose. Pruning reads only the blocks that could hold a better match than the ones already found, which is what keeps a full sweep affordable.
curl -X POST localhost:8080/spaces/catalogue/search/batch \ -H 'content-type: application/json' -d '{ "queries": [ {"id": "SKU-1001", "limit": 5, "min_relatedness": 0.5}, {"id": "SKU-1002", "limit": 5, "min_relatedness": 0.5}, … ] }' # results identical to running each query on its own
This is the pair that makes deduplication tractable. similarity 0.97 with coverage 0.18 is "looks identical, on almost nothing" — the trap. 0.93 at coverage 0.91 is a well-supported duplicate.
Relatedness folds both into one chance-centred number, which is what you threshold on; keep coverage beside it to decide whether a human needs to look.
// a trap: thin evidence, perfect agreement { "id": "SKU-8841", "similarity": 0.97, "coverage": 0.18 } // a duplicate: most of both rows agrees, and ranks higher { "id": "SKU-1002", "similarity": 0.93, "coverage": 0.91 }
Keep the pairs above your threshold, join them transitively, and you have candidate groups. Send a group's ids to /explain and it tells you what those objects have in common against the population — most distinctive first, with the values behind it.
On the demo population, explaining 789 objects took about 1.3 ms, so explaining every cluster is not a batch job, it is a loop.
curl -X POST localhost:8080/spaces/catalogue/explain \ -H 'content-type: application/json' \ -d '{"ids": ["SKU-1002", "SKU-4471", "SKU-9930"]}' // shape of the answer; values depend on your catalogue "properties": [ {"path": "capacity_ml", "distinctiveness": 0.48, "selection": {"median": 350}, "population": {"median": 420}}, … ]
One request opens the built-in viewer on the population: every object a point, laid out so alike things sit together. Draw around a cluster and it tells you what those objects have in common; click a point for its exact neighbours.
The layout is approximate and says so — on 125,000 products the ten nearest on screen averaged 0.31 relatedness against 0.67 for the true nearest. Everything except the positions is exact, so use the picture to find the question and the numbers to answer it.
open http://localhost:8080/spaces/catalogue/explore # colour by any property, draw a region, # and ask what the region has in common
Decide a winner, upsert it, delete the losers. The next version holds the merged catalogue; the previous one keeps answering exactly as before, which is what lets you show an auditor what the catalogue looked like before the merge — and undo it if the merge was wrong.
Retention decides how long old versions live, and a deleted object can be erased for good when the versions holding it are gone.
curl -X POST localhost:8080/spaces/catalogue/objects \ -H 'content-type: application/json' -d '{ "objects": [{"sku": "SKU-1002", "capacity_ml": 350, "weight_g": 421, "brand": "Hägerlund"}], "delete": ["SKU-8841"] }'
Relatedness puts chance at 0 and scales by the evidence the Space could offer. Measured on 125,000 synthetic products with ten properties:
| Relatedness | What it was | Reading |
|---|---|---|
| 0.74 | the median product against its true best match in the population | The band duplicates live in. Start your threshold here and move it with your own data. |
| 0.53 – 0.89 | that same best-match figure, 10th to 90th percentile | Below 0.53 you are into pairs that are merely similar products. |
| 0.02 | the median random pair of products | Chance. Anything near it is noise. |
| −0.02 | {"type": "mug"} against {"type": "diesel engine"} | Both are products; that is all they share. |
A duplicate is not the same question as a near-identical product. Two units of one item and two listings of one item score alike, because nothing in the data distinguishes them — that distinction lives in your process, not in the comparison. Pair a high threshold with a review step for anything you cannot afford to merge wrongly.
Benchmarks from the engine's own suite on an Apple M3 Pro; the conditions are in the performance section. A sweep is one batched query per object, so the cost scales with the catalogue and with how selective each object is — a row with ten filled properties prunes far harder than one with two.
filters job, not a similarity one.One container on your own machines, the catalogue you have, and an afternoon. Nothing leaves the deployment, and nothing needs labelling first.