Hilliance#

A universal object similarity engine. Let the data define distance.

Hilliance takes arbitrary structured data (JSON objects with any mix of text, numbers, booleans, arrays and nested objects) and learns, from a population of such objects, how to compare them. It answers two questions, fast and exactly:

• How similar are these two objects? compare(A, B) returns a similarity, how much evidence it was based on, and why.
• Which objects are most similar to this one? search(Q) returns the best matches from millions of objects in microseconds.

You do not define schemas, embeddings, feature vectors, normalisation rules, numeric scales, categorical encodings, missing-value strategies or property weights. Hilliance infers them from the data.

JSON objects  ──learn──▶  Space (versioned, immutable)  ──▶  encode · compare · search

Version 1.0.0 — what changed, what it does and what it does not, in CHANGELOG.md.

Why Hilliance#

Comparing real-world records is deceptively hard:

• The data is heterogeneous. One product has type, color and radius. Another has category, colour and a nested manufacturer. A third is a diesel engine with none of those.
• The data is incomplete. Knowing an object is a "mug" tells you something. Knowing it is yellow tells you more. No record is a complete description of the thing it describes.
• Scales are relative. Is a radius of 10 vs 11 a big difference? For cups, yes. For wheels, no. The answer depends on the population.
• Meaning is not spelling. "mug", "coffee mug" and "cup" are close. "mug" and "diesel engine" are not. Exact string matching gets this wrong.

The usual answer is to hand-engineer features for every field, or to serialise the whole object into one text embedding. The first doesn't scale across datasets. The second loses quantities ("radius = 10" as text isn't a number), treats missing fields as different, and can't explain itself.

Hilliance treats every value as an observation about an underlying object, learns how each property behaves across the population, and compares objects only on the evidence they share, while reporting how much evidence that was.

It is not a JSON embedding API and not primarily a vector database. The product is comparison: embeddings, distributions and indexes are internal mechanisms.

Core ideas#

Space#

A Space is a learned comparison environment for one population of objects (products, vehicles, suppliers). Learning a Space discovers:

• which properties exist and what kind each is (numeric, date, text, boolean, identifier)
• the distribution of every numeric property
• the vocabulary and semantic relationships of every text property
• how informative each property is, and which properties are redundant with each other
• which differently named properties are actually the same (color/colour)

The same raw value can mean different things in different Spaces: radius = 10 may be huge among teacups and tiny among bicycle wheels. Spaces are versioned and immutable (products-v1, products-v2, …), so results are reproducible.

Observations, not a single vector#

An object is never collapsed into one embedding. It is kept as a set of per-property observations, each represented in the way that suits it:

mug-17
├── type      "coffee mug"   → semantic (calibrated text similarity)
├── color     "yellow"       → semantic
├── radius    10             → numeric (position in the population's distribution: 0.51)
├── tags      [coffee, gift] → set of semantic values
└── manufacturer.country "Sweden" → semantic

Missing is not different#

If object A has a colour and B does not, that is no evidence, not evidence of difference. Comparisons use only properties observed in both objects, and report how much of the available evidence that covered.

Four numbers per comparison#

similarity: 0.97, coverage: 0.18 means "looks very similar, but on thin evidence". similarity: 0.93, coverage: 0.91 is a much better-supported match, and ranks higher by score and relatedness.

Reading the similarity scale. A similarity of 0 is not "infinitely different": every property is measured relative to the population, so 0 means as far apart as this population allows on the shared properties (the smallest vs the largest radius, a boolean mismatch, text less alike than practically any two values in the data). And 0 is not "unrelated" either. Two random objects score about 0.5 on a property with differing values: that expected value is the property's prior, and it is what chance produces.

Reading relatedness. Relatedness puts chance at 0. It sums, over the shared properties, how far each similarity lies above or below its prior, and scales that by the room the whole Space leaves above or below chance:

Because the scale covers all of the Space's properties, relatedness reflects how much evidence there is, not only which way it points. Measured on 125,000 synthetic products with 10 properties:

One matching property is little evidence about a product described by ten, so single-property comparisons stay close to 0. mug vs diesel engine is only slightly negative: most random pairs of products also have unrelated types. For the same reason, two different colours are about chance level (a random pair almost always differs in colour), while a matching colour is real evidence.

Let the data define distance#

Every per-property similarity is calibrated against the population. A text similarity of 0.8 means "more similar than about 80% of random pairs of different values of this property". Numeric similarity is based on how much of the population lies between two values. This puts text, numbers and booleans on one scale, so they can be combined without hand-tuned weights.

What "similar" means here#

Every discussion of similarity arrives at "but similarity is subjective". It is worth being exact about which part is, and what this engine does about it.

Identity is the objective anchor. Two objects described by the same values are the same on the evidence available, and the engine returns exactly 1.0 — not 0.98, and with no setting that could make it anything else.

Below identity, some bias is unavoidable. This is Watanabe's ugly duckling theorem: count all predicates definable over a finite set of objects and every pair of distinct objects shares exactly the same number of them. Treat all evidence as equal and a swan is as similar to a duck as to another swan. A usable scale between 0 and 1 cannot be had without privileging some evidence over the rest. The question is never whether to interpret, only where the interpretation comes from.

Meaning is not in the bits. Comparing raw values settles nothing, as two comparisons of the same property show:

Identical digits, no similarity; different digits, identity. What separates them is a model of what the numbers refer to — here the unit table, elsewhere the vocabulary of a text property or the distribution of a numeric one. Note also what the engine is answering: are these alike, not are these the same thing. Two identical rows may be two units of one product, and no comparison of their descriptions can tell you which question you meant.

So the claim is not that nothing is interpreted. It is that nothing is imported. Each stance in the system is estimated from the population rather than decided by a person:

There are no per-customer weights to tune, nothing a support engineer can nudge, and every number is reproducible from the version and traceable through /explain. The prior is the population, which is the least arbitrary one available: it is the only thing in the room that is not somebody's opinion.

What genuinely stays subjective, and is better said than hidden:

• Which properties exist at all. The schema is the largest single lever on every number here, and it is yours, decided before any comparison happens.
• Where the population ends. The same two objects score differently in a catalogue of mugs and a catalogue of everything, because the baseline moved with the population. Numbers from different Spaces are not comparable.
• What the embedder read. Semantic similarity inherits the text the model was trained on. That fast and firm are far apart in one language's corpus is a fact about that corpus, not about beds.
• Similar for what. Substitutable on a shelf, priced alike, shipped alike — these are different questions, and no population picks between them for you.

The first three are visible and can be changed: edit the schema, choose the population, choose the model. That is the difference between a measurement and an opinion.

A walkthrough#

The numbers below come from a Space learned from 125,000 synthetic products with the bundled semantic model.

Semantic, not string, matching:

Missing is not different:

The missing colour costs coverage, not similarity. A different colour is real evidence of difference.

Different key names, explained per property (colour resolves onto color automatically):

POST /spaces/products/compare
{ "a": {"type": "mug", "color": "yellow", "material": "ceramic"},
  "b": {"type": "coffee mug", "colour": "gold", "material": "porcelain"},
  "explain": true }

{ "similarity": 0.942, "coverage": 1.0, "score": 0.661, "relatedness": 0.262, "compared": 3,
  "contributions": {
    "type":     {"kind": "semantic", "similarity": 0.986, "weight": 0.541, "status": "compared"},
    "color":    {"kind": "semantic", "similarity": 0.893, "weight": 0.900, "status": "compared"},
    "material": {"kind": "semantic", "similarity": 0.977, "weight": 0.553, "status": "compared"} } }

type and material carry less weight than color here because they are redundant with each other (and with radius and price) in this population: what they say in common is counted once, not three times, while what each says alone keeps its full weight. score (0.66) is lower than similarity (0.94) because only 3 of the Space's 10 properties were observed. The others are filled with what two random products would score. relatedness (0.26) says the same on a chance-centred scale: clearly related, on about a third of the evidence a product can offer.

Search over the 125,000 products for {"type": "mug", "color": "yellow", "radius": 4.5} returns exact yellow mugs of that size in 24 µs, scanning 5 of 123 blocks.

Constraints that are not similarity. "Like this cup" is a question for the model; "in porcelain", "under 25" and "in stock" are not. Those go in filters, the one literal part of a query, and they compose with the ranking rather than replacing it:

POST /spaces/products/search
{ "object": {"type": "mug", "color": "yellow", "radius": 4.5}, "limit": 3,
  "filters": [ {"property": "material", "any": ["glass", "porcelain"]},
               {"property": "price", "min": 10, "max": 25},
               {"property": "in_stock", "any": ["true"]} ] }

Two things worth reading off that table. Swapping ceramic for porcelain costs no relatedness (0.297 either way): the model learned that those materials are alike, so a porcelain mug answers the question as well as a ceramic one. Demanding a price over 40 does cost it (0.191), because no yellow mug in this population is that expensive — the best remaining answers are plates, and the number says so instead of pretending.

Filters never enlarge the scan: a block whose summary rules it out is skipped before anything is scored. Whether that shows up as time depends on the query. A selective one already reads two blocks and gains nothing (13 µs → 18 µs, the per-object check); an expensive one gains: the eight-property query drops from 155 µs over 23 blocks to 132 µs over 14 when restricted to one type. The wider point is that a constraint never costs you a linear scan.

Quick start#

Docker#

docker run -p 8080:8080 -v hilliance-data:/data hilliance/hilliance:1.0.0

hilliance-data is a named volume, and it should stay one: the server runs as an unprivileged user (uid 10001), so a bind-mounted host directory — -v "$PWD/data:/data" — is not writable by it on Linux, and the first write fails with Permission denied. Docker Desktop on macOS hides that. If you do want the artifacts in a directory you can see, chown 10001:999 it first, or run with --user $(id -u):$(id -g).

Published for linux/amd64 and linux/arm64, so a pull fetches whichever the machine needs. latest follows the newest release; name a version when you would rather decide yourself. To build it instead:

docker build -t hilliance .
docker run -p 8080:8080 -v hilliance-data:/data hilliance

The image includes the embedding model (Model2Vec potion-multilingual-128M, 102 languages, 530 MB, CPU-only), which makes the image about 640 MB. It needs no network access at runtime and data never leaves your environment. The model is pinned to a commit and checked against its SHA-256 digests at build time.

For English-only data, a smaller image with potion-base-8M (30 MB):

docker build -t hilliance:en --build-arg MODEL_REPO=minishlab/potion-base-8M \
  --build-arg MODEL_REVISION=main --build-arg MODEL_SHA256= .

Spaces are served only by the model they were learned with, so after switching an image's model, re-learn each existing Space with the new one: hilliance rebuild <space> --embedder=model (see Upgrading). Until then /health lists them under incompatible_spaces.

From source#

cargo build --release
HILLIANCE_MODEL=/path/to/potion-multilingual-128M ./target/release/hilliance   # semantic embeddings
./target/release/hilliance                                            # lexical fallback, no model needed

First requests#

# Learn a Space
curl -X POST localhost:8080/spaces -H 'content-type: application/json' -d '{
  "name": "products",
  "objects": [
    {"id": "mug-1",  "type": "mug",           "color": "yellow", "radius": 4.5, "material": "ceramic"},
    {"id": "cup-1",  "type": "cup",           "color": "white",  "radius": 4.0, "material": "paper"},
    {"id": "eng-1",  "type": "diesel engine", "color": "black",  "radius": 45,  "material": "cast iron"}
  ]}'

# Compare two objects
curl -X POST localhost:8080/spaces/products/compare -H 'content-type: application/json' \
  -d '{"a": {"type": "coffee mug", "color": "gold"}, "b_id": "mug-1", "explain": true}'

# Find the most similar objects
curl -X POST localhost:8080/spaces/products/search -H 'content-type: application/json' \
  -d '{"object": {"type": "teacup", "color": "yellow"}, "limit": 5}'

(Real Spaces should have enough objects for distributions to mean something: hundreds at least, ideally thousands or more.)

If a call comes back asking for a key#

{"error": "this call needs a key (send `Authorization: Bearer <secret>`)"}

Then the server you reached wants one, and /health says which it is without a key at all:

curl -s localhost:8080/health
# {"status":"ok","auth":"open"}      — the calls above work exactly as written
# {"status":"ok","auth":"required"}  — everything but /health needs a key

A fresh docker run is open, and says so at startup: no key or token is set: every caller may read, learn and delete Spaces. So a server asking for a key when you expected an open one is usually a different server — an earlier container still holding the port. docker ps shows what is published where, and -p 8090:8080 puts the new one somewhere else. Then every example needs that port too: localhost:8090.

Running it with a key instead. One file, mounted read-only:

SECRET=$(openssl rand -hex 24)
cat > keys.json <<JSON
[ {"id": "ops", "secret": "$SECRET", "role": "admin"} ]
JSON

docker run -p 8080:8080 -v hilliance-data:/data \
  -v "$PWD/keys.json:/keys.json:ro" -e HILLIANCE_KEYS=/keys.json \
  hilliance/hilliance:1.0.0

It confirms what it loaded, and never logs a secret:

INFO hilliance: keys loaded from /keys.json admin=1 write=0 read=0

Every call then carries it:

curl -H "Authorization: Bearer $SECRET" localhost:8080/spaces

A key the server doesn't know is 401; one it knows that may not do this is 403. Roles, per-Space scoping and end-user tokens are under Who may call it.

Configuration#

Who may call it#

Every endpoint except /health needs Authorization: Bearer <secret>:

curl -H "Authorization: Bearer $SECRET" localhost:8080/spaces

Keys and roles. HILLIANCE_KEYS names a JSON file of keys. Each has an id, which names it in logs and errors but is never the secret, a role, and optionally the Spaces it may act on:

[ {"id": "ops",     "secret": "…", "role": "admin"},
  {"id": "ingest",  "secret": "…", "role": "write", "spaces": ["catalogue"]},
  {"id": "backend", "secret": "…", "role": "read"} ]

Which role a call needs is decided by the route — its method and pattern — so no Space or object name can make a write look like a read. A key scoped to named Spaces is refused on any other, by name or by version id (catalogue-v3), and may not learn one either. A key the server does not know gets 401; a key it knows that may not do this gets 403, because answering 401 to a valid key would be a lie about whether it is known.

The two tokens still work. HILLIANCE_TOKEN is an admin key and HILLIANCE_READ_TOKEN a read key, so an existing deployment upgrades unchanged. Setting them together with HILLIANCE_KEYS is refused rather than resolved, so there is never a question which decides. A read token without a write token is still refused at startup: it would look like protection and leave every call open. Secrets are compared in time that reveals neither their content nor their length, and every key is compared on every call, so the time taken says nothing about which one matched.

/health answers anyone, for container probes, but without a key only says {"status": "ok", "auth": "required"}; what the server serves, its role and its nodes are for key holders.

Without a key the server is open: anyone who can reach the port can read every Space and delete them all. It logs a warning at startup saying exactly that. Run it behind something that authenticates, or set HILLIANCE_KEYS.

Scoped tokens: the end user's credential. A customer's application already knows who its users are. Rather than keeping a second copy of that here, it mints a short-lived token for each user, signed with a key both sides hold, and this server verifies it:

{"alg": "HS256", "kid": "k2", "typ": "HSC"}
{"sub": "user-42", "spc": "catalogue", "lbl": ["u:42", "team:nordics"],
 "iat": 1759900000, "exp": 1759900900}

It is a JWS compact token — header.payload.signature, base64url, HMAC-SHA256 — so every language has a library for it. HILLIANCE_SCOPE_KEYS names a JSON file of key id to secret (32 bytes or more); several keys may be live at once, which is how one is rotated. Without that file the server takes no scoped tokens at all.

A scoped token may only read, and only the Space its spc names — by name or by version id. Endpoints that describe the server rather than a Space (GET /spaces, /metrics) are refused, as is learning a Space. The signing algorithm is this server's choice and not the token's: only HS256 with typ: HSC is accepted, so neither none nor an asymmetric name can turn a token into one that verifies itself. A token must carry sub, spc, iat and exp, may live no longer than HILLIANCE_SCOPE_MAX_TTL (default 15 minutes), and is allowed a minute of clock skew either way.

Minting one is a dozen lines in any language — here is the whole of it, and it is what the tests verify against:

import base64, hashlib, hmac, json, time

def mint(secret, kid, sub, space, labels, ttl=300):
    b64 = lambda b: base64.urlsafe_b64encode(b).rstrip(b"=").decode()
    now = int(time.time())
    head = b64(json.dumps({"alg": "HS256", "kid": kid, "typ": "HSC"}).encode())
    body = b64(json.dumps({"sub": sub, "spc": space, "lbl": labels,
                           "iat": now, "exp": now + ttl}).encode())
    sig = hmac.new(secret, f"{head}.{body}".encode(), hashlib.sha256).digest()
    return f"{head}.{body}.{b64(sig)}"

Nothing revokes a token once minted, which is why the lifetime is minutes: the lifetime is the revocation window.

What the labels mean. An object is visible to a caller when the labels it carries and the labels the caller holds have one in common. Grants are made per Space, by object id:

curl -X POST localhost:8080/spaces/catalogue/acl -H "Authorization: Bearer $WRITE_KEY" \
  -d '{"grant":  [{"label": "u:42", "ids": ["doc-7", "doc-9"]}],
       "revoke": [{"label": "team:nordics", "ids": ["doc-3"]}]}'

They need the write role — a grant changes who sees, not what exists — and the answer carries the seq they were written at, which GET /spaces/{space}/acl/status reports back. An id that is not in the Space is recorded anyway: refusing it would answer whether an object exists, and a permission may legitimately arrive before the object it is about.

Labels live beside the versions, in <data>/.acl/<space>/, as an append-only log folded into a snapshot: permissions change far more often than populations, and a version is immutable. Every change is flushed before the call returns, because a lost grant is an inconvenience and a lost revoke is a disclosure.

What a scoped caller gets. Search, batch, rank, compare, reading an object, projections and explanations all honour the mask, and honour it where filters are honoured — before anything is scored, so a hidden object neither appears in an answer nor moves the threshold that decides how much of the population is read. A caller who may see 64 of 20,000 objects reads two blocks, not the population.

The answers also lose what would describe the rest of it: how many blocks a search read, which filter values nothing has, each match's place in the population's order, the size of the population, the ids a request named that are not visible, a property's own smallest and largest value, whether a value occurs in the data at all, and the population figures under an explanation. Version history and the state of the grant store are an operator's business and are refused outright.

An object a caller may not see answers exactly as one that is not there — 404, never 403 — and that holds whether it is asked for, compared against, ranked among candidates, or named as the query itself: a query by id is a question about that object, so it is checked like any other. rank drops such ids silently rather than naming them back.

What a scoped caller keeps is the model: what the engine made of the values it sent, each property's weight, where a number falls in the learned distribution. That is the pooled part, and it is shared by design (below).

Revocation. A revoke takes effect on the writer immediately — the next query sees it. A follower or a cluster node picks up the change within HILLIANCE_ACL_POLL (200 ms) of the file reaching it, plus whatever a query already in flight takes. There is no revocation list for tokens themselves: a token already minted keeps its labels until it expires, which is why the default lifetime is minutes.

In a cluster, the coordinator forwards the caller's own token and each node masks the objects it holds; it cannot do this itself, since it holds no population. Grants are therefore fanned out to every node, and a partial result is an error naming the nodes that missed it — a revoke that lands on two nodes of three is a disclosure on the third. Grants and revokes repeat harmlessly, so the fix is to send it again.

What a restricted caller can still learn. The model is pooled: one Space, learned from every object in it, which is the reason to share a Space at all. So a caller restricted to 200 of 50,000 objects can still read, from /spaces/{space} and /encode, the distribution of every numeric property across all of them — minimum, quartiles, maximum — the five most frequent values of every text property with their counts, whether an arbitrary value appears anywhere in the population and how often, and how informative each property is. It cannot learn which objects have those values, or that any particular object exists.

That is the right trade for a catalogue and the wrong one where the population itself is the secret. For those, give each tenant its own Space: the statistics are then learned from that tenant's objects alone.

More guards are on by default:

• A query is cut off at HILLIANCE_READ_TIMEOUT (408) and a learn at the longer HILLIANCE_WRITE_TIMEOUT. A write that times out still finishes — the caller stops waiting, the write keeps the lock until it is done — so a slow write is never interleaved with the next.
• Requests beyond HILLIANCE_MAX_CONCURRENCY are shed with 503 and Retry-After rather than queued.
• A panic in a handler answers 500 and the server keeps serving (release builds unwind for this).
• Reads take bodies of at most HILLIANCE_MAX_READ_BODY_MB (16); a batch may ask for 100,000 matches in all (1,000 when explained), a projection lays out at most 200,000 objects, and a retention period is at most a century.
• Error messages show the data directory as <data>, not where it is.
• The viewer runs under a strict Content-Security-Policy: only its own script, with a fresh nonce per page, and every value from the data is escaped before it becomes HTML.

TLS is not built in; terminate it at a proxy.

Several embedders in one server#

A Space is served by the embedder it was learned with, so one process can hold more than one — an English model and a multilingual one, or an old model kept for Spaces not yet re-learned:

HILLIANCE_MODELS="en=/opt/models/potion-base-8M,multi=/opt/models/potion-multilingual" hilliance serve

• New Spaces get the first model unless they ask: {"name": "produkter", "objects": [...], "embedder": "multi"}, by configured name or by fingerprint.
• Queries, comparisons and encodings use the embedder of the Space they address, so every Space answers at once.
• Changing or re-learning a Space keeps its embedder: a new version has to stay comparable with the ones before it.
• /health lists what the server has, with dimensions and fingerprints. A Space whose embedder is missing is still listed under incompatible_spaces and not served, and the log names what the server does have.
• hilliance build --embedder=NAME picks the model on the command line.

Fingerprints are hashes of a model's actual output rather than its path, so two entries pointing at the same weights are refused as duplicates instead of quietly splitting your Spaces in two.

Learning a population larger than memory#

build reads everything at once, which is the fastest thing to do until the population stops fitting. --stream reads a JSON Lines file three times instead and holds one shard at a time:

1. Sample and learn. Count the objects, keep a uniform reservoir sample (--sample=N, 100,000 by default), and learn the model from it. The sample's cluster keys give the boundaries that divide the population into shards.
2. Route and spill. Read again, encode each object with the frozen model, and append its line to its shard's file. Nothing is kept but counters: one per vocabulary value, not per observation.
3. Build each shard. One at a time: read its file, lay out its columns, write it to the pool, and let it go before the next. The spilled file is the shard's population file, so it is moved into place rather than copied.

A repeated id means the later line replaces the earlier, as in a request body. The first pass notes every id's key and line (24 bytes per object, briefly); the lines that are replaced — usually none — are all that is kept, and they are skipped from then on.

One million synthetic products (175 MB of JSON Lines), under memory pressure from other work on the machine:

Memory follows the shard size and the learning sample, not the population: with 250,000-object shards, the build's own peak was about 230 MB at 500,000 objects and 270–330 MB at two million. It costs extra time for the passes and one encode per object — about 1.5× here, and up to 2.3× on an idle machine, where the in-memory build is faster.

What it trades. The model is learned from the sample, so numeric distributions, calibration and the weights derived from them are estimates. Everything counted stays exact: presence, observations and vocabulary counts come from every object, values the sample never saw are added to the vocabularies before any shard is laid out, and weights are rescaled to the true counts while keeping the redundancy discount the sample measured. Objects are placed by the same rule as an in-memory build, so search behaves the same — tests assert that a streamed Space holds the identical population and returns overlapping neighbourhoods for the same queries.

The same build runs behind POST /spaces when the body is JSON Lines (see POST /spaces). Over HTTP, 500,000 products (91 MB) peaked at 264–274 MB in the server, against 1.81 GB for the same population as one JSON body; two million (350 MB) at 417–461 MB.

Offline learning#

Spaces can be learned outside the server and shipped as artifacts:

hilliance build products objects.jsonl [id-field]        # JSON array or JSON Lines
hilliance build --stream products objects.jsonl          # larger than memory (JSON Lines)
hilliance build --embedder=multi produkter objects.jsonl # with a particular model
hilliance check [space] [--deep]                    # verify artifacts (see Keeping the data)

API reference#

All endpoints take and return JSON. Anywhere a Space is named you can use an exact id (products-v3) or a bare name (products), which resolves to the latest version. Responses always name the exact Space that answered. Errors look like {"error": "..."}, with status 400 (bad request), 404 (unknown Space or object) or 422 (unprocessable, e.g. embedder mismatch).

GET /health#

{ "status": "ok", "embedder": "model2vec/256#643f71b3dbf60ba8", "spaces": 3, "incompatible_spaces": [] }

POST /spaces: learn a Space#

{ "name": "products",
  "objects": [ {...}, {...} ],
  "id_field": "id",
  "options": { "aliases": true, "redundancy": true, "filter_only": ["tenant"] } }

{ "id": "products-v1", "name": "products", "version": 1, "parent": null,
  "object_count": 125000, "property_count": 10, "content_hash": "c007804062c3d99a", "status": "ready" }

• name: [a-z0-9][a-z0-9_-]*, at most 64 characters, not ending in -v<number>. Re-using a name creates the next version.
• id_field (optional): top-level key holding each object's external id. It defaults to id when objects have one; otherwise objects are addressed as #<index>. The id is metadata, never evidence.
• options (optional): aliases merges differently named properties that hold the same thing; redundancy shares weight among properties carrying the same information. Both default to on. filter_only lists paths to store and allow filters on while keeping them out of the model — a tenant or owner column, which decides who may see a row and says nothing about how alike two rows are. New versions and rebuilds inherit all three.

Or stream it. A population too large for one JSON body goes in as JSON Lines — one object per line — with the settings in the query string:

curl -X POST 'http://localhost:8080/spaces?name=products&id_field=sku' \
  -H 'Content-Type: application/x-ndjson' --data-binary @products.jsonl

The body is written to disk as it arrives (<data>/.uploads/), never held in memory, and learned with the streaming build once it is complete. The write lock is only taken then, so a slow upload does not hold up other changes. The query accepts name (required), id_field, embedder, sample (objects to learn the model from, 100 to 1,000,000, default 100,000), aliases, redundancy and filter_only (comma-separated paths). The answer is the one above plus "learning": "streamed" and a stream summary (bytes, lines, sampled, shards, late_values, upload_ms, learn_ms). Repeated ids collapse as they do in a JSON body — the last line wins — and are counted in changes.duplicates_in_request.

application/x-ndjson, application/jsonl and their common spellings are recognised. The upload is capped by HILLIANCE_MAX_STREAM_GB (413 beyond it) and, with learning, by HILLIANCE_STREAM_TIMEOUT. All uploads in progress share HILLIANCE_MAX_UPLOADS_GB between them, so several at once cannot fill the disk: one that would take the total past it is refused with 507 Insufficient Storage and its spool removed, and each upload's share returns when it finishes; a broken line is a 422 naming it, and nothing is published. The upload, the spill files and the finished Space are on disk at the same time for a moment, so allow about three times the upload's size. Compressed bodies are not accepted yet.

POST /spaces/{space}/objects: upsert and delete objects#

{ "objects": [ {"id": "mug-1", "type": "mug", "color": "yellow"} ],
  "deletes": [ "bowl-7" ],
  "learning": "auto" }

{ "id": "shop-v2", "parent": "shop-v1", "version": 2, "object_count": 3, "status": "ready",
  "learning": "incremental", "learned_from": "shop-v1", "changes_since_learn": 2,
  "changes": { "inserted": 0, "replaced": 1, "unchanged": 0, "deleted": 1, "not_found": [],
               "duplicates_in_request": 0, "deduplicated": 0 } }

• objects (alias upserts) inserts new ids and replaces existing ones. deletes removes ids. The change set is applied to the parent's population and learned as a new version (201). The parent stays loaded and unchanged.
• An upsert identical to the stored object is a no-op. If nothing changes, no version is created: the response is 200 with "status": "unchanged".
• Changes must target the latest version. A bare name always does; an older exact id returns 409, so concurrent writers can't silently fork history.
• learning decides how the change set becomes the next version (see incremental learning):
• auto (default): incremental when possible, otherwise a full re-learn. The response then carries fallback_reason.
• incremental: incremental or 422 with the reason.
• full: always re-learn.
• Returns 400 when an id is both upserted and deleted in one request, or when the change set would remove every object.

GET /spaces/{space}/objects/{id}#

{ "id": "mug-1", "space": "shop-v2", "object": {"id": "mug-1", "type": "mug", "color": "yellow"} }

Returns an object's stored JSON as of that version, or 404. shop-v1 still returns the blue mug. Every Space records, per object, which version's objects.jsonl holds it and at which byte offset. A read is one index lookup and one seek, independent of population size (about 0.4 ms over HTTP).

DELETE /spaces/{space}/objects/{id}#

Deletes one object, creating the next version (201), or returns 404 if the id isn't in the Space.

DELETE /spaces/{version} · DELETE /spaces/{name}?all=true#

Deletes one version, or (with ?all=true on a bare name) every version of a Space:

{ "name": "shop", "deleted": ["shop-v2"], "rebased": ["shop-v3", "shop-v4"], "remaining": ["shop-v1", "shop-v3", "shop-v4"] }

Shards that remaining versions still read are compacted first, so every remaining version stays complete, and shards nothing uses any more are then removed. rebased lists the remaining versions that had shards compacted (see Versions, retention and deletion). Pinned versions return 409, and so does the last remaining version: use ?all=true for that. Version numbers are never reused, so re-creating a deleted Space continues after its highest version, and a deleted id answers 404 "was deleted".

POST /spaces/{space}/acl · GET /spaces/{space}/acl/status#

{ "grant":  [ {"label": "u:42", "ids": ["doc-7", "doc-9"]} ],
  "revoke": [ {"label": "team:nordics", "ids": ["doc-3"]} ] }

{ "seq": 10294, "granted": 2, "revoked": 1, "labels": 8123 }

Who may see which objects, by label (see Who may call it). Needs the write role. Grants and revokes are idempotent, applied together, and on disk before the call returns. At most 10,000 ids per grant; send more in batches. On a coordinator the change is fanned out to every node and the answer lists each node's seq; if any node missed it the call fails naming them, and the fix is to send it again. acl/status reports how far a server has applied, per node on a coordinator.

GET /spaces/{space}/retention · PUT /spaces/{space}/retention[?dry_run=true]#

{ "keep_last": 3, "keep_within": "7d", "keep_daily": 14, "keep_weekly": 8, "keep_monthly": 12,
  "pinned": ["shop-v1"] }

Count- and age-based rules, with the semantics of backup tools like restic and borg: a version is kept if any rule keeps it.

The latest version is always kept. With no rules, every version is kept. PUT replaces the policy and applies it immediately. ?dry_run=true only reports what would happen.

Policies run after every new version and also periodically (HILLIANCE_RETENTION_INTERVAL, default 10m), so age rules expire versions even when nothing is written. GET explains the current state:

{ "policy": {"keep_within": "7d"}, "pinned": ["shop-v1"], "pending_deletion": ["shop-v2"],
  "versions": [ {"id": "shop-v4", "created_at": "...", "kept_by": ["latest", "within"], "shards": 4, "own_shards": 1, "shares_data_with": ["shop-v3"], ...},
                {"id": "shop-v2", "kept_by": [], ...} ] }

kept_by lists every reason a version survives. An empty list means the next enforcement deletes it. own_shards counts the shards the version wrote itself; the rest it shares with the versions in shares_data_with. Responses that create a version include "retention": {"deleted": [...]} when the policy removed older ones.

GET /spaces · GET /spaces/{space}#

Lists manifests, or returns one manifest plus the discovered schema. For each property the schema includes:

• kind, presence, distinct, multi_valued
• learned weight and base_weight, and prior
• aliases and redundant_with
• a distribution summary for numbers, top_values for text
• unit when the population named one, which is the unit every number reported for that property is in
• array for a property inside an array of objects (variants for variants.color): its values are compared element by element with the other properties of that array

POST /spaces/{space}/compare#

{ "a": {...} | "a_id": "obj-1",
  "b": {...} | "b_id": "obj-2",
  "explain": false }

{ "similarity": 0.942, "distance": 0.058, "coverage": 1.0, "score": 0.661, "relatedness": 0.262, "compared": 3,
  "space": "products-v1", "contributions": { ... } }

Either side can be an inline object or a population member by id. contributions (with explain) lists every property involved, with its kind, weight, similarity and status (compared, missing_in_a, missing_in_b).

POST /spaces/{space}/search#

{ "object": {...} | "id": "obj-123",
  "limit": 10,
  "rank_by": "score",
  "min_coverage": 0.0,
  "min_similarity": 0.0,
  "min_relatedness": null,
  "filters": [],
  "explain": false,
  "exhaustive": false,
  "approximate": false }

{ "matches": [ { "id": "obj-15320", "similarity": 1.0, "coverage": 0.29, "score": 0.68, "relatedness": 0.297 } ],
  "exact": true,
  "space": "products-v1", "objects": 125000, "blocks_scanned": 7, "blocks_total": 123,
  "resolved_properties": { "Colour": "color" }, "unknown_properties": [], "took_us": 55 }

• Search by a member with id; the member itself is excluded from the results.
• rank_by: score (default; favours well-supported matches) or similarity (evidence-only). Both are pruned.
• Results are exact: identical to comparing the query with every object. exhaustive: true turns pruning off to verify this.
• approximate: true (or {"probes": n}) trades that for speed on very large free-text vocabularies — see Approximate search. It may miss results; every number it returns is still exact, and the response says "exact": false whenever it actually approximated anything.
• blocks_scanned / blocks_total show how much of the population pruning had to look at.
• min_relatedness (−1…1) keeps only objects at least that related, e.g. 0 for "more alike than chance". It is an exact threshold on the same evidence that ranks by score, so it also prunes: blocks that cannot reach it are skipped. Filters combine; an object must pass all of them.
• filters are hard constraints, the one part of a query that is literal rather than learned:

"filters": [ {"property": "color", "any": ["navy", "crimson"]},
             {"property": "price", "min": 100, "max": 700},
             {"property": "delivered_at", "max": "2026-06-30"},
             {"property": "tags", "exists": true} ]

any needs the value as the population spells it (case and spacing are normalised, nothing else): "navy" means that value, not something navy-ish. On a set-valued property the object passes if it carries any of the listed values. min/max are inclusive, in the data's own units, and accept dates for temporal properties. exists selects objects that have the property, or those that don't.

Filters prune rather than cost. They read the same per-block summaries pruning uses — which value groups a block holds, the range its numbers span — so whole blocks are dropped before anything is scored, and every surviving object is then checked exactly. A filtered search never scans more blocks than the same search unfiltered; on an expensive query that means less work (155 µs over 23 blocks → 132 µs over 14 on the demo population), and on an already-selective one the per-object check costs a few microseconds.

A property the model should ignore. Learning a Space with "options": {"filter_only": ["tenant"]} (or ?filter_only=tenant on a streamed upload) stores that path and allows filters on it, but keeps it out of the model: it carries no weight, never makes two objects alike, and does not appear among a comparison's contributions. Who owns a row decides who may see it, and says nothing about how alike two rows are. It also keeps the column where kind inference would otherwise drop it — a per-row owner id reads as an identifier, and identifiers are stored nowhere, so nothing could be filtered on them. A filter-only path is matched whole, never by resemblance, and new versions and rebuilds inherit it.

A property the Space doesn't have is a 400 with the filterable properties listed. A value nothing has isn't an error: it comes back in unknown_filter_values, and a filter naming only such values matches nothing (rather than silently matching everything).

POST /spaces/{space}/search/batch#

{ "queries": [ {"object": {"type": "mug", "color": "yellow"}},
               {"id": "obj-42", "limit": 5},
               {"object": {"type": "diesel engine"}, "rank_by": "similarity"} ] }

{ "results": [ {"matches": [...], "blocks_scanned": 1, "blocks_total": 123, "resolved_properties": {}, "unknown_properties": []},
               {"matches": [...], ...},
               {"error": "object \"obj-42\" not in space products-v1", "status": 404} ],
  "space": "products-v1", "objects": 125000, "queries": 3, "took_us": 541 }

Up to 10,000 queries per request, each with the same fields as search, asking for at most 100,000 matches in all (1,000 if any query sets explain). Results come back in order and are identical to running each query on its own. A query that fails (e.g. an unknown id) reports its error in its slot; the others still run. Batching pays off for bulk jobs such as deduplication, matching a feed against a catalogue, or "find neighbours of every object":

• queries are encoded in parallel
• prunable queries run in parallel, each keeping its own best-first pruning
• exhaustive queries share one pass over the population, so each block's columns are loaded once for all of them

GET /metrics#

Prometheus text format: a latency histogram and a count per route pattern and status (route="/spaces/{id}/search", never the id, so the label space stays bounded), plus gauges for Spaces, objects, layouts computed, cache hits, requests shed and requests that panicked. It follows the same token rules as any other read.

hilliance_request_duration_seconds_count{route="/spaces/{id}/search",status="200"} 2
hilliance_spaces 1
hilliance_objects 125000
hilliance_projection_cache_hits_total 1

GET /spaces/{space}/explore#

The built-in viewer (see Exploring a Space): open it in a browser and look at the population. No build step, no network access, nothing to install.

POST /spaces/{space}/projection#

{ "sample": 5000, "dims": 3, "landmarks": 128, "neighbors": 10, "epochs": 300, "seed": 1,
  "include_neighbors": false }

{ "space": "products-v1", "objects": 125000, "points": 5000, "dims": 3,
  "ids": ["obj-46512", "..."], "coords": [0.523, -0.301, 0.344, "..."],
  "properties": [ {"index": 9, "path": "type", "kind": "semantic", "weight": 0.541,
                   "distinct": 10, "presence": 1.0, "coherence": 0.632} ],
  "quality": {"on_screen": 0.31, "best": 0.67, "random": 0.03},
  "cached": false, "took_ms": 905 }

Coordinates for every sampled object, dims floats each, centred and scaled into [−1, 1]. The same options always give the same picture, and each layout is computed once per version and then reused (cached).

• coherence is how often an object and its true nearest neighbours carry the same value for that property (numeric properties: how close the values sit). It says which property explains why things are alike, and makes the best default colouring.
• quality is the honest measure of what the flattening costs: the mean relatedness of the points that land next to each other, against exact search (best) and against arbitrary objects (random).

POST /spaces/{space}/explain#

{ "ids": ["obj-15320", "obj-4471", "..."] }

{ "space": "products-v1", "objects": 789, "population": 125000, "took_us": 1300,
  "properties": [
    { "path": "price", "kind": "numeric", "weight": 0.56, "distinctiveness": 0.43,
      "raw": 0.51, "noise": 0.08, "presence": 0.64, "population_presence": 0.65,
      "detail": "numbers", "mean_position": 0.21,
      "selection":  {"p25": 18.04, "median": 23.02, "p75": 27.37},
      "population": {"p25": 20.66, "median": 38.41, "p75": 106.44} },
    { "path": "type", "kind": "semantic", "distinctiveness": 0.38, "detail": "values",
      "values": [ {"value": "teacup", "share": 0.22, "population_share": 0.10, "count": 176} ] } ]
}

What a group of objects has in common: every property, most distinctive first. Works on any set of ids — a region drawn in the viewer, a search result, a list from elsewhere.

• distinctiveness (0–1) is how differently the property behaves in the group than in the population: total-variation distance between the two distributions, or for numbers the largest gap between their cumulative distributions, with "the property is missing" counted as a value of its own.
• A small group differs from the population by chance alone, so the same measure is taken on several random groups of the same size and the strongest of them (noise) is subtracted. Twelve arbitrary objects explain nothing; twelve teacups still do, with a score that honestly stays lower than three hundred teacups would give.
• Numbers are reported in the data's own units (the ECDF is inverted), next to the population's quartiles.
• The population side needs no scan: a Space already knows every value's frequency and every property's presence, and numeric values are stored as positions in the population's distribution. Explaining 789 objects takes about 1.3 ms.

POST /spaces/{space}/projection/select#

{ "sample": 5000, "dims": 3, "filters": [ {"property": "type", "any": ["teacup", "mug"]} ] }

Which of a projection's points pass the filters, in point order (passes, plus a matching count). Lets a viewer show a filtered population without recomputing or moving the layout.

POST /spaces/{space}/projection/values#

{ "property": "type", "sample": 5000, "dims": 3 }

Returns one value per point, in the projection's order: values + categories for text and booleans, or values (position in the distribution, 0–1) + min/max for numbers. −1 means the object doesn't have the property.

POST /spaces/{space}/rank#

{ "object": {"type": "mug", "color": "yellow", "radius": 4.5},
  "ids": ["obj-44288", "obj-98094", "..."],
  "limit": 10, "explain": false, "min_relatedness": null, "filters": [] }

{ "matches": [ {"id": "obj-44288", "relatedness": 0.274, "similarity": 0.96, "coverage": 0.29} ],
  "candidates": 200, "unknown_ids": ["obj-retired-4"], "took_us": 128, "space": "products-v1" }

Ranks a list you supply instead of the whole Space, for candidates that came from somewhere else: an inventory query, a shortlist, another system's constraints.

• The numbers and the order are exactly what search would have produced over those objects — tested by ranking every id in a Space and comparing with a search.
• Every option of search applies: thresholds, filters, explain, and rank_by. limit defaults to the whole candidate list.
• Ids the Space doesn't hold come back in unknown_ids rather than failing the request, since inventory and Space drift apart between versions.
• Candidates that share no property with the query are not results, as in search.
• Cost is one exact comparison per candidate, with no pruning to help: 200 candidates in 128 µs, 5,000 in 808 µs. Use it when the constraints live outside Hilliance; when they are properties of the objects, filters on search is the cheaper route, because it prunes whole blocks instead of comparing every candidate.

POST /spaces/{space}/encode[?vectors=true]#

The body is the object itself. Returns its representation in the Space:

• for each property: kind, weight, value(s); for numbers the distribution_position, for text whether the value was in_population and its population_frequency (plus the vector with ?vectors=true)
• resolved_properties and unknown_properties
• the Space id, its version (content hash) and the embedder id

Exploring a Space#

GET /spaces/{space}/explore opens a viewer on the population: every object a point, laid out so that alike things sit together.

open http://localhost:8080/spaces/products/explore

It is part of the server (one self-contained page, no CDN, no build step), so it works in the Docker image, offline, on your own data.

It carries a credential when the server wants one. A browser cannot put a header on the request that loads a page, so the page itself is served to anyone — it holds no data, and every number on it comes from API calls it makes afterwards. Those calls need a key, which the page reads from the address fragment and keeps in memory only:

http://localhost:8080/spaces/products/explore#token=<key or end-user token>

A fragment is never sent to a server and never reaches its logs, as a query string would, and it is removed from the address bar once read. With an end user's token the viewer shows that user's own objects and nothing else.

• Colour by any property. The list is ordered by how well each property explains which objects are alike, so the most informative colouring comes first, and the percentage next to each name says how often neighbours agree on it.
• Click a point to see the object's JSON and its exact nearest neighbours, with relatedness, similarity and coverage, in microseconds. Lines on the map join it to the nearest plotted objects.
• Search in place. Paste an object, and the matches light up where they are, so you see which part of the data your query lands in, and min_relatedness trims the tail. The ranked matches are listed with their relatedness, similarity and coverage; clicking one selects it, and matches outside the plotted sample are marked rather than hidden.
• Filter on any property — values, a numeric range, or presence. Objects that fail fade out of the map, the search results and the region selection, so you explore one slice of the population at a time. The mask is computed server-side in a fraction of a millisecond.
• Draw around a group ("Select region", or hold Alt and drag; Shift and drag pans) and it tells you what that group has in common: the properties that set it apart, with the values and numbers behind them. The same button explains a search result.
• 2D or 3D, drag to turn, scroll to zoom, and up to 20,000 points at 7 ms a frame. The 3D view is a turntable: sideways drag spins it all the way round, vertical drag tilts and stops just short of straight down, so the cloud never turns over and the drag never reverses on you. Perspective is deliberately mild — the camera sits eight cloud-radii away, which keeps near and far points within 1.25x of each other, enough to read depth without turning the layout into a bulging lens. JSON is syntax-highlighted, both the object you read and the query you write.

What the layout is, and is not#

A Space has no vector per object, only comparisons, so coordinates are solved from distances: every sampled object is compared with a few hundred landmarks, classical MDS places the landmarks and then everything else (global structure), and a neighbour-guided refinement pulls each object towards its true nearest neighbours (local structure).

The layout is approximate, and the viewer says so. A population of ten properties genuinely needs about ten dimensions: the eigenvalues barely decay, so no flat picture can keep every neighbourhood. Measured on 125,000 products, the ten points nearest on screen average 0.31 relatedness, against 0.67 for the true nearest neighbours and 0.03 for arbitrary objects. So the map is far better than chance and never as good as search.

That is why everything except the positions is exact: the neighbours, the numbers, the search results and the explanations all come from the ordinary API. Use the picture to find the question, and the numbers to answer it.

Why is that a group?#

Drawing around a cluster answers it from the data. A region of the demo population, 789 objects, explained in 1.3 ms:

Small, cheap drinkware that is in stock. The same call explains a search result (POST /spaces/{space}/explain), so "what do these 500 matches have in common?" is one request.

Cost at 125,000 objects on an M3 Pro: 2,000 points in 0.9 s, 5,000 in 2.1 s, 20,000 in 6.8 s, then cached per version. Most of it is the exact nearest-neighbour search that the refinement needs.

Objects, identity and updates#

Objects with an id (from the Space's id_field) are entities:

• Ids are unique within a version. Learning rejects duplicate ids. Create requests and CLI builds collapse repeated ids, keeping the last occurrence, and report how many they collapsed (duplicates_in_request).
• Changes are upserts and deletes by id. Registered a mug as blue that was actually yellow? Send it again with the same id:

curl -X POST localhost:8080/spaces/shop/objects -H 'content-type: application/json' \
  -d '{"objects": [{"id": "mug-1", "type": "mug", "color": "yellow"}]}'

The next version contains exactly one mug-1, now yellow, in its original position. Search, a_id lookups and comparisons all see the corrected data.

• Versions stay immutable. Every change produces a new version (shop-v1 → shop-v2), and the old one keeps answering exactly as before. Consumers can pin a version and move on their own schedule, and every version can be rebuilt exactly from its parent plus the change set.
• Optimistic concurrency. Changes apply only to the latest version. Using a bare name (shop) means "latest", resolved under the write lock. Targeting an older version returns 409 instead of silently branching history.
• A correction changes the population statistics. Replacing blue with yellow slightly shifts colour frequencies, weights and calibration for every comparison in the new version. That is intended: a Space describes the population as it is.
• Queries are always live. Comparing or searching with an inline object uses exactly the data you send, so nothing needs updating for that. The lifecycle only concerns which objects are in the population.

Objects without an id are anonymous. They can be added, but not replaced or deleted, and their #<index> handles are only meaningful within one version. Spaces written before ids were enforced may contain duplicate ids; the next change collapses them to the last occurrence (deduplicated in the summary).

Incremental learning#

Small change sets don't re-learn the model. An incremental version keeps the parent's learned model frozen:

• kinds and aliases
• numeric distributions and calibration
• value groups, weights, priors and redundancy

It updates everything that describes membership:

• the column store, with new objects merged into the clustered order so pruning stays effective
• block summaries, which are always exact, so search stays exact
• vocabularies: new text values are embedded, assigned to their nearest value group, and appended. For int8 vocabularies the group bounds grow to cover them.
• counts, presence and the id index

No population file is read. The change set is resolved against the parent's id index, and no-op upserts are detected with a stored per-object content hash. Statistics (counts, presence) are updated from the changed objects alone.

Only the shards the change touches are rewritten; every other shard is shared with the parent, on disk and in memory. A replaced object stays in its shard, and a new one goes to the shard whose cluster-key range holds it. Each rewritten shard stores only its delta: the objects it inserts or replaces (its own objects.jsonl) plus its base shard and deleted ids (changes.json). Unchanged objects are still read from the ancestor file that holds them. The full population is reconstructed by replaying deltas when a full re-learn needs it.

The cost of a change is proportional to the shards it touches, not to the Space. Changes to random objects touch every shard once they are numerous enough, so large change sets gain less.

The frozen model can't represent every change honestly, so learning: "auto" falls back to a full re-learn when any of these holds, and reports why:

• an object has a property the model has never seen ("warranty" is not in the learned model)
• a value doesn't fit its property's learned kind, e.g. text in a numeric property
• more than 10% of the population has changed since the last full learn (drift)
• more than 32 incremental versions have been chained since the last full learn

Every manifest records its lineage: learned_from (the version whose population the model was learned on), changes_since_learn and incremental_depth.

An incremental version is deterministically reproducible from its parent plus the change set. It is not identical to learning the same population from scratch. It is the same population seen through the older model, which is exactly what the drift limit bounds.

How it works: the statistical model#

1. Observations#

Objects are flattened into (path, scalar) observations:

• Nested objects use dotted paths: manufacturer.country.
• Array elements are observations of their parent path, so tags becomes a set-valued property.
• Objects inside an array keep their fields together. variants: [{color, size, price}, …] becomes the properties variants.color, variants.size and variants.price, and every value also records which variant it came from (see Arrays of objects).
• null and blank strings count as absence, not values.

Arrays of objects#

A shirt sold as a red S and a blue L is not the shirt sold as a red L and a blue S, though both have the colours {red, blue} and the sizes {S, L}. So the properties of one array are compared element by element:

1. Each element on one side is matched to the element on the other side that agrees with it most, counting every field both record as evidence (w·(s − prior), so a field only one of them records costs nothing).
2. Each value is then compared with the values of its matched element only. If that element doesn't record the field, or the value sits in no element, it is compared with all of them as before — missing is not different.

Each property still contributes its own similarity, weight and explanation; only which values meet changes. Asking for {"variants": [{"color": "red", "size": "S"}]} now ranks the red-S-and-blue-L shirt clearly above the red-L-and-blue-S one, where flattened they tied. Arrays nested in arrays (items[].parts[]) pair within the innermost array.

Search stays exact. Every value picks its partner from a subset of what flattening allowed, so no property ever scores higher paired than flattened. The search kernels still score flattened, which makes their scores and block bounds upper bounds. A candidate that could enter the top k is scored exactly (with compare, like rank) before it is admitted. That refinement only runs for queries it can affect — two or more values inside one array. Everything else, including every query on data without arrays of objects, runs exactly as before.

2. Kind inference (per path, from the population)#

3. Property names: aliases, confirmed by data#

Different sources name the same property differently (color/colour, manufacturer.country/maker.country). Name similarity only proposes candidate pairs:

• the same key after spelling normalisation (productColor = product_color)
• spelling similarity ≥ 0.8 (colour/color)
• name-embedding cosine ≥ 0.55 (price/cost = 0.72)

The data decides. Two paths merge into one property only if all of these hold:

• they almost never occur in the same object (≤ 5%)
• they have the same kind
• their values are compatible: text value centroids with cosine ≥ 0.75, or number distributions with Kolmogorov–Smirnov ≤ 0.25 — compared in one unit, so radius_mm and radius_cm are recognised as the same measurement while genuinely different quantities are still rejected

Similar-sounding names that always occur together are never merged: min_price/max_price (cosine 0.58), width/height, origin_country/shipping_country.

At query time, unfamiliar keys (Colour, cost, manufacturerCountry) resolve onto known properties when two things hold:

• the name is close enough: same normalised spelling, spelling similarity, or embedding cosine ≥ 0.7
• the value kind fits

A key is never resolved onto a property the object already has. Every resolution is reported.

4. Redundancy: evidence is counted once#

type: "mug" and category: "tableware / mug" are different properties that say the same thing. Their names are not similar (the embedding cosine of type/category is 0.39), so redundancy is measured from the values:

• the normalised mutual information between each pair of properties, with a Miller–Madow small-sample correction
• scaled by how often the other property is present

Each partner's dependence is the fraction of a property's information it claims. Adding those fractions up would count one slice many times: in the demo data radius, material and price all restate much the same part of type, and summing their claims (about 0.55 + 0.54 + 0.33) charged type for 1.4 copies of itself, leaving it a weight of 0.37 out of 0.90 — less than half of color's, which predicts nothing.

So claims are nested rather than added. The largest claim covers the smaller ones, and a slice claimed by t partners is held by t + 1 properties and counts once, split t + 1 ways. A property keeps everything only it knows, plus an equal share of each slice it holds with others — the Shapley value of "what does this set of properties tell you". In the demo data type now weighs 0.54 of its 0.90, and radius, material and price each keep 60–77% of theirs.

Where the old sum was exact, this is too: k copies of the same thing claim all of each other, so each keeps exactly 1/k and together they count as one. Independent properties are untouched, and restating a fact under more names leaves the weight of everything touching it roughly where it was. The schema lists redundant_with for each property.

5. Numbers: population-relative scale#

Each value maps to its mid-rank empirical CDF position u ∈ [0, 1]. Similarity is (1 − |u_a − u_b|)², floored at zero. Whether 10 vs 11 is "big" depends on how much of the population lies between them. The ECDF is invariant to monotone transforms, so skewed or log-normal data needs no special handling.

Beyond the range, positions carry on rather than stopping at the end: a query value one tail above the largest seen sits at about 1.1, ten tails above at 2. A tail is the span of the outermost tenth of the distribution, so how far outside a value counts as depends on the data — 20 past the maximum is nothing in a population spanning hundreds and absurd in one spanning a single unit. Without this, every value above the maximum would be the maximum, and a query for a price of a million would match the dearest thing in the catalogue perfectly.

Far enough out, similarity is simply zero: it is a population percentile, and nothing can be more different than "as different as this population allows". So a query for a radius of 1000 where nothing exceeds 40 returns matches with similarity 0 and a negative relatedness, rather than a confident wrong answer. A filter bound outside the range is answered by the range too — price ≥ 1000000 matches nothing instead of matching the most expensive thing there is.

Units. Sources disagree about units, and a number means nothing without one. A unit is read from the end of a key (radius_mm, weight-kg, lengthCm) or from the value itself ("12 cm"), and everything a property holds is converted into one unit: the one its representative path uses, so a catalogue in centimetres goes on reporting centimetres. The schema names it ("unit": "cm").

Two paths that measure the same quantity in comparable units are candidates to merge whatever their names score, and the value check then confirms them on a common scale — so radius_mm and radius_cm from two suppliers become one property with 600 observations rather than two with 300 each. After that the unit is just notation: {"radius_mm": 95}, {"radius_cm": 9.5} and {"radius_cm": "95 mm"} are the same query, and radius_mm ≥ 100 is the same filter as radius_cm ≥ 10. An explicit unit on a value beats the one its key implies. Objects added later are converted the same way, so a radius_mm arriving at a centimetre population lands where it belongs.

Only ratio scales convert, because only they convert by a factor. Currencies are not units: a rate is a fact about a day rather than about the quantity, so price_usd and price_eur stay two properties. Temperatures are left alone for the same reason — °C to °F needs an offset. A population that never mentions a unit is untouched.

6. Text: calibrated semantics#

• Embedding. Every semantic property's vocabulary is embedded once.
• Calibration. Raw cosines aren't comparable across properties (colours might all sit at 0.7–0.9, product names at 0.1–0.6). So Hilliance samples random pairs of distinct population values and stores the quantiles of their cosines. A raw cosine becomes its percentile among them.
• Unseen values. Values that never appear in the population are embedded on the fly and calibrated the same way.

Identical values score exactly 1. Two different values never tie with an exact match: they map into [0, 1 − 1/256].

7. Weights#

A property's base weight is its Gini impurity, the probability that two random observations differ:

• constant properties get 0
• near-constant flags get very little
• identifiers are forced to 0

Redundancy then shares base weight among properties that carry the same information.

8. Combining the evidence#

Only properties observed in both objects contribute:

• similarity = Σ w·s / Σ_shared w
• coverage = Σ_shared w / Σ_union w
• score = (Σ_shared w·s + Σ_other w·prior) / Σ_all w

• relatedness = E / Σ_all w·(1 − prior) when E ≥ 0, else E / Σ_all w·prior, where E = Σ_shared w·(s − prior)

prior is the expected similarity of two random objects on that property: 1 − gini for booleans (the chance of a match), and a match's chance plus half the rest for text and numbers (differing values average 0.5). Ranking by score is equivalent to ranking by E, i.e. evidence beyond chance. Thin lucky matches don't beat well-supported ones, and sparse objects aren't favoured over rich ones. relatedness rescales E by constants of the Space, so it ranks identically, and a min_relatedness is an exact bound on E that search can prune with.

Sets such as tags use symmetric Chamfer similarity: each element's best match, averaged in both directions.

A list has its own chance level. Offering more values means every value finds a closer partner, so the Chamfer rises with the length of the lists compared, whether or not they have anything to do with each other. Measured on 400 unrelated tags: two values against two score 0.22, ten against ten 0.85, two hundred against two hundred 0.93 — with no value in common. Charging a long list the chance of a single value would therefore report every long list as strong evidence.

So a property that holds lists learns a chance curve instead of one prior: what a value expects to find as its best match among 1, 2, 4 … 1024 others, sampled from the property's own values at learn time. Chance for a comparison is then the average of what each side expects to find in the other, 0.5·(best_of(nb) + best_of(na)), and prior — the flat number used for evidence that is missing entirely — is the curve at the sizes the population actually carries. Single-valued properties learn no curve and are unaffected.

On 5,000 machines with option lists of 2 to 200 entries, this moved two hundred unrelated options against two hundred others from +0.32 relatedness to −0.00, and restored half the discrimination the list had been costing the properties around it (same-family versus different-family gap 0.48 → 0.57, against 0.66 for the same population with no list at all).

Performance#

Search and compare are the product, so the read path is optimised aggressively. Learning pays for it by preparing a layout that is fast to search. Everything below is exact: pruning never changes results, and tests assert that pruned results are identical to exhaustive ones and to pairwise compare.

Headline numbers#

Apple M3 Pro (12 threads), synthetic 10-property product population, top-10 ranked by score, median of 31 runs (cargo run --release --example scale -- <n>):

With a near-unique free-text title on 1M objects (… -- 1000000 titles):

Other operations:

• compare two stored objects: 57–66 ns (the higher end when a property holds lists, which costs a chance lookup per side)
• encode a query with three never-seen text values: 2.6 µs
• learning: about 3 µs per object (2M objects in 5.8 s)
• incremental change of a few objects: 60 ms at 1M in one shard, 18 ms at 4M in 16 shards (see Sharding)
• artifact size: 38 bytes per object for the 10-property population (75 MB for 2M), loaded in 48 ms

Selective queries stay flat as the population grows. How much pruning helps depends on the data: it is large when populations have structure (categories, correlated attributes) and queries are selective. The worst case is the "pruning off" column, which scales linearly.

Under sustained load#

scripts/soak.py runs the load described below against a real server — a writer, a read replica on the same directory, mixed queries, change sets, retention, projections, and grants and revokes with end users reading under them. It checks that nothing fails or is shed, that every answer is the right one, that no masked answer holds an object its caller may not see, and that memory and file descriptors do not trend upward; it exits non-zero if any of that is untrue. A short run is scripts/soak.py --minutes 2 --rate 100 --objects 20000. A 20-minute soak on one machine (a 12-core laptop, with the load generator on the same machine): 500,000 products, a writer and a follower sharing a directory, 400 reads per second split between them (search, filtered search, search by member, fetch, compare, batches of 8, rank over 50 ids), a change set every 3 seconds (20 upserts, 2 deletes) with retention keeping 3 versions, a 5,000-object projection every 30 seconds, and a 100,000-object streamed upload halfway.

480,443 requests, none failed and none shed; no read took longer than 100 ms. Reads while a projection or the upload was running had a p99 of 8.2 ms, against 1.5 ms otherwise: heavy work runs on thread pools of its own, so it slows queries slightly rather than queueing them. Over 402 versions, retention kept exactly three, the follower stayed within 5 seconds of the writer, file descriptors stayed flat, and the writer's memory moved between 0.7 and 1.8 GB without trending upward. hilliance check --deep read every object back afterwards. Change sets are the slow part, and that is structural: an incremental version rewrites each shard it touches (here 250,000 objects each), so smaller shards make changes cheaper.

Earlier runs of the same soak found three problems, all fixed before this one: objects were read by file name, so retention compacting a shard broke reads on versions still being served (14,900 failed fetches); projections shared the queries' thread pool and stalled them for seconds (read p99 4.5 s while one ran); and a follower dropped old versions before loading new ones, so under load it briefly had none.

How search is fast#

Columnar store. Each property is one contiguous column, not a collection of object records:

• vocabulary ids stored as u8/u16/u32, whichever fits
• positions in the numeric distribution stored as u16
• CSR offsets for set-valued properties

Query tables. A text query value is compared once with each distinct population value, which produces a table with a trailing zero slot for "missing". Scoring an object is then one branch-free table lookup per property. Ranking by score only needs Σ w·(s − prior), which is one accumulator. Small vocabularies carry a precomputed u16 similarity matrix.

Exact block-max pruning. 1. At build time, objects are reordered so that similar objects sit together. The sort key is the most central properties: those whose values predict others, such as type → radius, material, price. 2. Objects are stored in blocks of 1,024, grouped into super-blocks of 64 blocks. Each block and super-block keeps a small summary per property: - text values: a 64-bit mask of the value groups present (vocabularies are clustered into ≤ 63 groups by embedding) - numbers: the min/max position 3. For a query, the summaries bound the best score any object in a block can reach: - super-blocks are bounded first, so bound overhead is O(N / 65,536) - blocks are visited best-bound first; the best few run on the calling thread to set a threshold - the rest are pulled from one best-first list by all threads, stopping as soon as no remaining bound can beat the current k-th best 4. The threshold carries the index tie-break, so blocks that could only tie, and would lose, are skipped too.

Pruning similarity ranking. Ranking by score bounds a sum. Ranking by similarity needs a bound on a weighted mean, Σ w·s / Σ w, taken over the query properties each object actually has.

• The naive bound (the largest per-property maximum) is 1.0 whenever a block contains any exact match on any single property, so it would never prune.
• The block summaries also record whether any object in the block lacks a property. Properties that no object lacks are mandatory: every object's mean includes them.
• The bound therefore starts from the mandatory properties' block maxima, then adds optional properties in descending order of their maxima while each raises the mean. This greedy is optimal.
• In structured data, central properties (like type) are nearly always present. So a block without mugs has its bound pulled down by type, even if it contains yellow objects.
• A tiny relative margin covers float rounding. Keys and bounds are capped at 1, so blocks that could only tie a perfect match are still skipped via the index tie-break.

At 2M objects, similarity ranking takes 10–206 µs pruned versus 0.8–3.4 ms exhaustive (9–120×), and tests assert identical results.

Large vocabularies (free text). A property like title can have as many distinct values as there are objects.

• int8 storage. Vocabularies above 8,192 values (quantize_vocab_above) store their vectors as per-vector-scaled int8. The query is quantised once too, so every similarity is an exact integer dot product: 4× smaller, and bit-identical in compare, in search and on every CPU.
• SIMD. The int8 dot has hand-written kernels, chosen at runtime:
• x86-64: AVX-512 VNNI (vpdpbusd, 512 bits), AVX-VNNI (vpdpbusd, 256 bits), AVX2 (vpmaddubsw + vpmaddwd)
• aarch64: sdot (Apple M-series, Graviton 2+)
• portable scalar fallback

All of them compute the exact integer, so results are identical on every CPU. A 256-dimension dot takes 3.8 ns with sdot on an M3 Pro, versus 6.8 ns scalar (cargo run --release --example simd_bench).

• Self-validating dispatch. Before a kernel is used, it runs against the scalar reference on a battery of vectors (awkward lengths, extreme, sparse and random codes). A kernel that disagrees is skipped for the next level down, and the server logs why. The chosen kernel is logged at startup and reported in /health. HILLIANCE_SIMD caps the level (scalar, avx2, avxvnni, avx512vnni, dotprod).
• No table when other properties prune. Similarities are computed only for objects in blocks that survive pruning. Group bounds come from centroids and radii (⟨q,v⟩ ≤ ⟨q,c⟩ + ‖q‖·r).
• Table when needed. When a query still needs a large part of the vocabulary, it switches to one fused parallel table pass. That is bounded by memory bandwidth, about 2.6 ms per 1M values.

Zero-copy loading. Every array that grows with the population or vocabulary lives in columns.bin, 64-byte aligned and little-endian:

• columns and set offsets
• per-object weights, hashes and storage locations
• embeddings (float32/int8), counts, groups and similarity matrices
• packed string tables for ids and vocabularies

At load, the file is memory-mapped and every array is a view into it. space.bin holds only the model and metadata. Vocabularies are packed string tables with a sorted index, so a load never allocates per-value strings or rebuilds hash maps.

Consequences:

• Startup is instant, even with many versions loaded.
• Pages are read on demand and shared through the OS page cache. Replicas on one host, or several versions of one Space, don't duplicate memory, and Spaces can be larger than RAM (the OS pages cold columns out).
• Newly learned or rebased versions are served from their mapped artifact too; the build's heap copy is dropped. Shards shared with a version already loaded are reused, not mapped again.
• Writes are copy-on-write. An incremental change or rebase on a mapped version copies only the arrays it rewrites. The mapped files themselves are never modified.

Batch search. Space::search_batch (and POST …/search/batch) gives results identical to calling search per query:

• plans are built in parallel
• prunable queries run in parallel
• exhaustive scans share one pass over the blocks

At 2M objects:

• 200 member queries: 2.6× faster as a batch (about 6,600 queries/s).
• 32 similarity-ranked member queries: 1.7 ms for the batch, down from 82 ms before similarity pruning.
• Shared full scans (exhaustive) gain about 1.2× at this size. The columns (about 30 MB) mostly stay in the M3's caches, so scans are compute-bound. The shared pass matters when columns exceed cache, where each separate query would stream the population from DRAM.
• Over HTTP, 500 member queries in one batch took 37 ms versus 193 ms as individual requests (5.2×).

Full-mode kernel. Similarity ranking and the similarity/coverage filters accumulate three values per object: the score gain, Σ w·s and Σ w. The scan of each block that pruning doesn't skip is branch-free:

• Id columns gather [gain, w·s, w, 0] from one 16-byte table entry per object, with an all-zero entry for "missing".
• Numeric columns multiply by a 0/1 presence mask. Positions are always finite, so masked objects add exactly 0.
• Candidate selection runs in two passes. First a vectorisable pass computes every object's ranking key, folding evidence, both filters and the coverage division into a mask (the division is skipped when no coverage filter is set). Then only keys that can still beat the current k-th best reach the heap. With a full heap that is one predictable compare per object.

The values are the same ones the branchy version accumulated, so rankings are unchanged bit for bit. At 2M objects, a similarity-ranked scan went from 1,650 µs to 1,060 µs, and a batch of 32 such scans from 122 ms to 82 ms.

Concurrency. Spaces are immutable Arcs, so reads never wait for writes. External ids are packed into one buffer with a sorted index (about 12 bytes per object).

Approximate search#

Exact search scores a free-text query against every distinct value of a property once. For most properties that is nothing: a colour has a dozen values. A near-unique field — titles, descriptions — has as many values as objects, and scoring a query against a million titles reads a quarter of a gigabyte of embeddings, however few blocks the search itself then touches. That sweep is what dominates a title query (about 4 ms at a million titles), not the scan.

So a semantic vocabulary of 100,000 values or more also gets an index (ann_min_vocab in BuildOptions), and a search that asks for "approximate": true uses it instead of the sweep:

• The vocabulary is clustered into about 2·√V clusters, nested inside the value groups search already keeps, and every value is described by where it sits in its cluster: the cluster's mean, plus its coordinates along the cluster's eight main directions of spread (16 bytes per value).
• A query ranks the clusters, not the values. The probes clusters nearest it are scored exactly; every other value is estimated — its dot product with the query within its cluster's main directions. Estimates are per value, not per cluster, which is what lets {"title": …, "type": "mug"} find the best-titled mugs even when the title's nearest neighbours are all chairs.
• Only scanned objects are scored, and blocks are bounded by each cluster's best possible estimate, so vocabulary size stops mattering. Per query the work is ranking the clusters plus the objects pruning leaves.
• What comes back is exact. Search collects twice the candidates asked for (plus 16) and re-scores them with compare, so every number, and the order, is what an exact search would report for those objects. What is approximate is only which objects were found: one whose value was estimated too low can be missed. Probing every cluster is exact search, and the tests check that it returns the same results.

On a million synthetic products with near-unique titles (cargo run --release --example scale -- 1000000 titles), where recall counts a returned object as a match when it is at least as good as the exact tenth:

Over a wider set of 30 queries at a million titles, recall@10 is 0.963 at 4 probes, 0.970 at 32 and 0.997 at 128. The misses concentrate in queries that resemble nothing in the population ("heavy steel industrial pump" among cups and chairs), where the best results are a flat field of weak matches and more probes are what help. These numbers come from the hash embedder on synthetic titles; recall with the bundled semantic model has not been measured yet, so measure it on your own data (the scale example shows how) before relying on a probe count. The index adds about 25 MB per million values to the model and nothing measurable to the learn time.

Sharding#

A Space is one learned model plus N shards. Each shard has its own column store, block summaries, id index and population file. Learning creates ceil(objects / HILLIANCE_SHARD_OBJECTS) shards (default: one per million objects).

Shards partition the cluster order, not id hashes. Objects are sorted by the same cluster key that orders blocks (the codes of the most central properties), and that order is cut into equal, contiguous key ranges. Similar objects therefore share a shard, and a shard's blocks are as homogeneous as those of an unsharded Space. That matters: with hash routing, every block in 16 shards spanned a 16× wider key range, and the rich-object query scanned 393 blocks instead of 67.

Search is one best-first pass over all shards. Super-blocks of every shard compete in one priority order under one shared pruning threshold, and results merge exactly. Pruning is therefore as effective as with one shard, and the unpruned scans run on all cores. At 4M objects, 1 shard vs 16 shards:

Results are identical regardless of the number of shards. Tests compare sharded and unsharded Spaces query by query, and pruned against exhaustive search.

What sharding buys:

• Changes rewrite only the shards they touch (18 ms instead of 280 ms for a small change at 4M), and unchanged shards are shared with the parent version on disk and in memory.
• Learning lays out shards in parallel (8.5 s instead of 14.4 s at 4M).
• The unit of storage and of future distribution: a shard is a self-contained directory that another node could serve.

Lookups by id probe every shard's id index (one binary search each, well under a microsecond per shard), because an object's shard follows its content.

Versions, retention and deletion#

Versions share data. Every version lists its shards in the shard pool (.pool/). An incremental version lists the parent's shards for everything it didn't touch, and each shard it rewrote reads unchanged objects from ancestor shards' files. A shard therefore outlives the version that wrote it for as long as a later version uses it. Deleting versions removes their directories and then settles the pool:

• Compaction. A shard still used by a remaining version, but reading from a shard that nothing references any more, is rebased onto its nearest referenced ancestor. Its delta is recomputed directly against that ancestor. For example, deleting v2 from v1 ← v2 ← v3, where both changed shard 3, makes v3.s3 a delta against v1.s3 that combines both changes. Only objects that differ from the ancestor are written; everything else points into the ancestor's files. When no referenced ancestor exists (e.g. the full-learn root was deleted), the shard is materialised as a full population.
• Collection. Shards of the Space that no remaining version reads are removed.

Compaction runs oldest first, so each shard is rebased onto one that is already settled. It rewrites only storage pointers and deltas: the learned model, columns and ids are untouched, so search and compare results are identical before and after. Tests check this for every remaining version, together with byte-identical object reads, complete populations, and a pool that holds exactly what the remaining versions read.

Crash safety:

• A shard is rewritten in a temporary directory and swapped in atomically (X → .X.old, .X.tmp → X).
• Deleted versions and shards are renamed away (.X.deleting) before removal.
• Victims are only removed after every rebase has completed, and shards only after the versions. Every intermediate state is readable; an interruption at worst leaves unused shards, which the next deletion collects.
• At startup the server repairs whatever an interruption left behind.

Per-Space settings (the retention policy, pinned, and the version high-water mark that prevents reuse of numbers) live in $HILLIANCE_DATA/.meta/<name>.json. Retention decisions use each version's created_at from its manifest (UTC). A version with an unreadable timestamp is never deleted by an age rule.

In-flight requests that already hold an old version keep working in memory. A GET …/objects/{id} that races with the deletion of a file it reads can fail with 500 once; retrying hits the compacted version.

Versioning, artifacts and determinism#

$HILLIANCE_DATA/products-v1/          one directory per version
    manifest.json   id, version, parent, embedder id, content hash, shard count
    schema.json     discovered properties and learned statistics
    space.bin       the learned model and metadata (small)
    columns.bin     the model's large arrays (vocabulary embeddings, …), aligned
    shards.json     the shards this version consists of
$HILLIANCE_DATA/.pool/products-v1.s0/ one directory per shard, shared by versions
    shard.bin       shard metadata (small)
    columns.bin     columns, block summaries, ids, hashes and storage locations,
                    aligned; memory-mapped at load
    objects.jsonl   full learns: the shard's population; rewritten shards: only
                    the objects they insert or replace
    changes.json    rewritten shards only: the base shard and deleted ids

A shard is named after the version that wrote it (products-v3.s2: version 3, shard 2) and never changes afterwards. Each shard records, for every object, which shard's objects.jsonl holds it and its byte offset. GET …/objects/{id} therefore never scans a file, and an incremental version never rewrites the population. A full re-learn reconstructs the population by replaying each shard's chain of deltas back to the last full learn.

• Immutable versions. A version never changes after it's written. New data produces products-v2 with parent: "products-v1", and both stay servable.
• Atomic and durable writes. Versions and shards are written to temporary directories and renamed into place, shards first, so a crash never leaves a half-written Space visible. A rename only orders what the kernel has already written, so every file is flushed to the device before its directory is, the directory before the rename that publishes it, and the parent after — otherwise a power cut can leave a version that looks complete and holds empty files. It costs about 15% of a save (338 ms against 294 ms for 200,000 objects); HILLIANCE_FSYNC=off skips it for data you would rather rebuild than wait for.
• Portable. Learn in one environment, test, then deploy the directory elsewhere.
• Embedder fingerprints. The embedder id is a hash of the model's actual output on probe strings, not a file path. A server only serves Spaces learned with the same embedder, and lists others under incompatible_spaces in /health.
• Deterministic.
• vocabularies are sorted
• calibration sampling uses a seeded PRNG
• int8 dots are exact integers
• search ties break by index

The same population, options and embedder give the same content_hash, the same Space and the same answers, regardless of input order or CPU.

Keeping the data#

Checking an artifact directory#

hilliance check              # every version: model decodes, shards present, count matches
hilliance check products     # one Space
hilliance check --deep       # also read every object's stored JSON back

Exit status is non-zero if anything is unreadable, so it works in cron or CI. It repairs interrupted writes first (the same recovery the server runs at startup) and reports what it did. On a damaged copy it says which shard is missing rather than failing later during a query:

FAIL  products-v1  reading shard products-v1.s0: No such file or directory (os error 2)

A version written by an older binary is not damaged, only unreadable to this one, and check says so separately (it does not fail the run):

OLD   beds-v2  artifact format 8, this build reads 13; `hilliance rebuild beds-v2`

Upgrading: rebuilding older versions#

The binary artifacts change format as the engine improves, and a new binary does not read old ones. It does not need to: every version also keeps its population as plain JSON Lines, and the files that say how those make up the version — manifest.json, shards.json, each shard's changes.json — are JSON too, in a layout that does not change. rebuild replays them (shards, then their change sets in order, deletions included) into a new version learned by the current binary:

hilliance rebuild --all --prune     # every Space whose latest version is outdated
hilliance rebuild beds              # the latest version of one Space
hilliance rebuild beds-v2           # a specific version

• The new version is the next one of the same Space, with parent pointing at the one it was rebuilt from. The source is left alone unless --prune is given, which then removes the outdated versions up to the one rebuilt, along with pool shards nothing else still needs. A newer outdated version is kept: it holds changes the rebuild did not see.
• It learns with the embedder the version was learned with, found by its fingerprint; --embedder=NAME learns with another. --sample=N sets the learning sample, as for a streamed build — which is what a rebuild is, so memory stays bounded whatever the size.
• On a version that is already current it re-learns with this binary's rules, which is how improvements to learning reach an existing Space.
• Objects without an id are renumbered (#<index>); everything with an id keeps it.
• Run it with the writer stopped, or on a copy: like build, it writes versions into the directory itself.

Upgrading a deployment#

A release that changes the artifact format needs the data rebuilt, because the new binary will not read the old versions — it starts anyway, serves nothing from them, and lists them as outdated_spaces. The whole sequence, with the writer stopped:

docker build -t hilliance .                                     # the new binary
docker stop hilliance && docker rm hilliance                    # stop the writer
docker run --rm -v hilliance-data:/data hilliance rebuild --all --prune  # new format
docker run -d --name hilliance -p 8080:8080 -v hilliance-data:/data hilliance

rebuild --all takes every Space whose latest version is outdated, replays its population and change sets into a version the new binary wrote, and --prune removes the versions it replaced. Copy the directory first if you want the old ones back: everything in it is immutable, so a copy is a backup.

It is quick. Nine Spaces holding about 55,000 objects, two of them learned with a 512 MB multilingual model, rebuilt in under two seconds together, after which check reported 0 in an older format.

Read-only replicas need no action beyond restarting them on the new binary: they follow the directory and pick up the rebuilt versions on their own. Do the writer first, since a replica on the old binary cannot read what the new one wrote.

Outdated versions no longer keep a server from starting. It serves everything it can read, logs the command for the rest, and lists them as outdated_spaces in /health. Rebuilding the beds Spaces from format 8 took under 200 ms each and gave back identical populations.

Backup and restore#

The artifact directory is the whole state, and everything in it is immutable once written, so a copy is a backup. A filesystem or volume snapshot is the simplest correct answer, because it is atomic.

Copying a live directory works too, with one ordering rule and one pause:

1. Stop retention for the duration (HILLIANCE_RETENTION_INTERVAL=off, and no deletions), because deletion is the only thing that removes or rewrites a file.
2. Copy the version directories first, then .pool/, then .meta/. Shards are written before the version that lists them, so a shard listed by a copied version is already on disk when the pool is copied.
3. Run hilliance check --deep against the copy.

Restoring is copying the directory back. Artifacts are portable across machines and architectures; a server only serves Spaces whose embedder fingerprint matches its own, so restore it next to the same model.

Erasing something for good#

Deleting an object creates a new version without it — the older versions still hold it, which is the point of immutable versions but not what a deletion request means. To erase the data itself:

# 1. remove the object (creates the next version without it)
curl -X DELETE localhost:8080/spaces/products/objects/person-1
# 2. drop every version that still holds it
curl -X PUT localhost:8080/spaces/products/retention -d '{"keep_last": 1}'

The second step is what erases it: deleting those versions leaves their shards unreferenced, so the surviving version's shards are rewritten from its own population and the old shard files are collected. Tests assert exactly this — that no file under the data directory contains the object's bytes afterwards, and that the Space still answers with every remaining object intact. The same applies to a corrected value: the old one lives in the version that recorded it until that version goes.

Two caveats worth stating plainly: a backup taken before the erasure still contains the data, and the pages may survive on the underlying storage after the file is removed, which is a disk-level concern rather than something an application can promise.

Replicas#

One server owns the artifact directory and writes to it. Any number of others can serve the same data read-only and pick up new versions without restarting:

# the writer
hilliance serve

# a follower: serves what it finds, writes nothing
HILLIANCE_READ_ONLY=true HILLIANCE_REFRESH=30s hilliance serve

A follower rescans the directory on its interval and adjusts what it serves: versions that appeared are loaded, versions that are gone stop being served, and versions whose shards were rewritten underneath it are reloaded. That last case is the one that matters: retention compacts a shard under its own name, so "the version is still there" does not mean "what I hold is still right". Each served version keeps a fingerprint of its shard files, and a change in any of them forces a reload.

What a follower does not do: it never enforces retention, never repairs an interrupted write (recovery belongs to the writer), and refuses every call that would change data with 405 and a message saying where to send it. Reading, searching, ranking, explaining and projecting all work normally, and /health reports "role": "follower" with last_refresh_seconds_ago.

Getting the files there. Followers need the same directory, by any means:

• a shared volume (NFS, EFS) mounted read-only — simplest, one copy of the data;
• a file sync (rsync, aws s3 sync, a sidecar) into each follower's own disk — copy .pool/ before the version directories, so a version never arrives before the shards it lists. A version whose shards are missing simply fails to load and is retried on the next refresh, so a wrong order costs a cycle rather than correctness.

Files that change underneath. A loaded version reads its columns through memory maps and its objects through file handles taken when it loaded, so it keeps reading exactly the files it was built against even after the writer's retention rewrites or removes them: the data stays readable until the last version using it is dropped. A refresh loads what is new before dropping what is gone, and keeps serving a gone version while its successor is on disk but not loadable yet, so a follower that falls behind never has a moment without a version of a Space. Handles cost one file descriptor per population file in use, shared by every version that reads it; allow for that in ulimit -n with large pools.

What this is not. There is no leader election, no failover and no replication log: a follower is a cache of a directory, and how up to date it is depends on the refresh interval and the sync. As a last line of defence, a read that would return a different object than the one asked for answers 503 rather than somebody else's data.

More than one machine#

A Space that outgrows a machine is split by shard. Every node gets the version directory (the learned model) and a subset of .pool/; a coordinator asks them all and merges:

# node A: shards 0 and 1          # node B: shards 2 and 3
HILLIANCE_PARTIAL=true HILLIANCE_READ_ONLY=true hilliance serve

# the coordinator: no data of its own
HILLIANCE_NODES=http://node-a:8080,http://node-b:8080 hilliance serve

The merge is exact, and the nodes never talk to each other. Ranking is a total order, so the best k overall are always among the best k of each node: merging per-node results and keeping the best k is the answer one machine would give. Each node prunes with its own threshold, which costs a little scanning and nothing in correctness.

It is the same answer, not merely an equally good one. Every match carries shard and row — where the object sits in the population's order — and ties break on that, which means the same thing on every node. A test asserts a two-node cluster returns byte-identical results to a single machine holding all four shards, across ranking by score and by similarity, with filters and relatedness floors.

When a node is down the answer would be quietly incomplete, so the coordinator fails with 503 naming the node. HILLIANCE_ALLOW_PARTIAL=true answers from the nodes that replied instead, and every response says nodes_answered of nodes and how many objects were actually searched. /health lists each node and whether it is reachable, and a node that holds a share reports it ("share": [{"space": "cups-v1", "shards": 2, "of": 4}]).

Splitting the pool is a copy: build the Space on the writer, copy the version directory to every node, and give each node the shard directories it should serve. A node refuses to start if it has none of a Space's shards, and ignores the shards it does not have.

What this is not: there is no rebalancing, no replication of a shard to two nodes (a node's loss removes its objects from the answer), and no distributed writes — learning and changes happen on the writer that owns the whole directory, and the split is a copy afterwards.

Deployment#

• One container, CPU only. The model is baked into the image, so it needs no GPU and no network at runtime.
• State lives in the artifact directory (/data), mounted as a volume. Servers load every compatible Space at startup and hold them in memory. Read-only replicas serve the same directory and pick up new versions on their own (see Replicas).
• Memory. Versions are memory-mapped (HILLIANCE_LOAD=mmap), so memory is managed by the OS page cache. Hot columns stay resident, cold ones are paged out, and files are shared between processes. For predictable latency, give the page cache room for the shards' columns.bin of the versions you query most (versions share unchanged shards, so this is less than the sum per version). Large text vocabularies dominate, at about 256 bytes per distinct value (256 dimensions, int8).
• Platform. Linux and macOS (the Docker image is Linux). Mapped files stay valid while a version is deleted or replaced (POSIX inode semantics). Windows would refuse to delete mapped files, so HILLIANCE_LOAD=read is the safe choice there.
• Writes are serialised per server: one Space is learned at a time. Reads are never blocked.
• Health and probes. The server binds only after every Space is loaded, so answering /health also means "ready". The image has no curl, so it probes itself: hilliance healthcheck opens a connection to HILLIANCE_ADDR, asks for /health and exits non-zero if it doesn't get a 200. The Dockerfile wires it to HEALTHCHECK with a 60-second start period, which is the window for loading large Spaces.
• Scraping. GET /metrics serves Prometheus text; HILLIANCE_MAX_PROJECTIONS (default 1) keeps layout computation from competing with itself for cores, since each one already uses them all.

Production readiness#

Short version: ready to run in production as a read-heavy service, behind a proxy you control, with one writer. Not ready if you need automatic failover, per-tenant authorization, or a cluster that rebalances itself.

What has been measured or reviewed here:

• A 20-minute soak at 400 reads a second alongside writes, retention, projections and a streamed upload: 480,443 requests, none failed and none shed, search p99 1.5 ms, no read over 100 ms, flat file descriptors and no memory trend (see Under sustained load). Three earlier defects that run exposed are fixed.
• A security review, with everything it found fixed: authorization decided by route rather than by path text, a token compare that leaks neither content nor length, a minimal /health for unauthenticated callers, caps on batch sizes, read bodies, projection samples and retention periods, data-directory paths scrubbed from errors, and a nonce-based CSP with escaping in the viewer (see Who may call it).
• The artifact path: check --deep reads every object back, rebuild replays a population into the current format, erasure rewrites history, and retention prunes versions — all exercised by tests and by the soak.
• CI runs the suite on x86-64 and arm64, with clippy as an error, the scalar fallback, a build without the embedding model, and a daily cargo audit. It builds the Docker image on every push.

What is yours to decide or still open:

• TLS terminates at your proxy. Service keys carry a role (read, write, admin) and may be scoped to named Spaces; a customer's application mints short-lived tokens for its own end users, and what those users may see is enforced per object, in the cluster as well (see Who may call it). What is not built: an audit log, per-field visibility, and any revocation of a token short of its expiry. The statistics stay pooled by design, which is a disclosure worth reading before a security review finds it.
• One writer, chosen by you. Replicas follow the directory read-only; nothing elects a leader or fails over (see Replicas).
• Backups are your snapshots of the artifact directory (see Backup and restore).
• Writes are seconds, not milliseconds: a change set rewrites the shards it touches — about 2.4 s for 250,000-object shards. Size shards for the change rate you expect.
• Verified at 500,000 objects on one machine, with search benchmarks to a million. Beyond that, sharding across nodes is a copy you perform (see More than one machine).
• The AVX-VNNI and AVX-512 kernels have not run on real hardware yet. They self-check against scalar before being used, so the fallback is safe, but the timings are unproven (see Known limitations).

Architecture and code layout#

crates/hilliance-core     the engine (no I/O except artifact.rs)
  observe.rs              JSON → (path, scalar) observations
  build.rs                discovery, kind inference, aliases, redundancy, learning, layout
  numeric.rs              population distributions (ECDF)
  calib.rs                semantic similarity calibration
  embed.rs                Embedder trait; HashEmbedder, Model2VecEmbedder (feature `model2vec`)
  space.rs                the Space: properties, vocabularies, shards (column store, block summaries)
  encode.rs               object → per-property evidence; key resolution; JSON representation
  compare.rs              pairwise comparison, coverage, score, explanations
  search.rs               exact pruned search
  filter.rs               hard constraints: compiled to value ids and ranges, pruned by block
  explain.rs              what a group of objects has in common, against the population
  population.rs           object identity: upserts and deletes between versions
  project.rs              laying a population out in 2D/3D for the viewer
  incremental.rs          incremental versions: frozen model, updated membership
  stream.rs               learning from a file larger than memory: sample, spill, build per shard
  retention.rs            retention policies; deleting versions: shard compaction and collection
  simd.rs                 int8 dot products with runtime CPU dispatch
  buf.rs                  zero-copy arrays: owned or memory-mapped, copy-on-write
  artifact.rs             save/load versioned Spaces and the shard pool
  synth.rs                deterministic synthetic data (tests, benchmarks, demos)
crates/hilliance-server   axum HTTP API, the built-in viewer (explore.html) and the `hilliance build` CLI
  api.rs                  routes, request shapes, tokens, timeouts, load shedding
  cluster.rs              fanning a query out to nodes and merging their answers exactly
  metrics.rs              request counters and latencies in Prometheus format
  state.rs                loaded Spaces, per-Space settings, projection cache

Representation strategies (Kind plus a column type) and aggregation (compare.rs / search.rs) are separate, so either can evolve independently. To add a new representation, you add a Kind, a Column variant, an element similarity and a scan kernel.

Development (the toolchain is pinned to Rust 1.98.1 in rust-toolchain.toml; cargo picks it up automatically):

cargo test --workspace --release                             # 121 tests: 105 engine, 16 HTTP (routes, errors, auth, limits, metrics,
                                                             #   embedders, followers, clusters)
cargo test --release -p hilliance-core                       # the engine alone: 105 tests: behaviour, exactness, determinism, identity,
                                                             #   incremental, retention, mmap, batch, sharding, relatedness, projection,
                                                             #   explanations, filters, rank, erasure, streaming, out-of-range numbers
cargo run --release --example scale -- 2000000 [titles|plain] [shard_objects]   # latency at scale
cargo bench -p hilliance-core                                # criterion micro-benchmarks
cargo run --release --example simd_bench                     # int8 dot throughput per SIMD level
scripts/test-x86-qemu.sh                                     # core tests as x86-64 code (AVX2 + baseline) via QEMU

Known limitations and roadmap#

Data model

• Stored numbers saturate at the learned range. A query beyond the range is handled (see Numbers), but a stored position is 16 bits inside 0…1, so an object added by an incremental change whose value exceeds what the model learned sits at the edge, indistinguishable from the previous extreme. A full re-learn widens the distribution and separates them.
• Redundancy is measured pairwise. Nesting the claims (see Redundancy) stops a central property being charged once per property it explains, and is exact for copies and for independent properties. In between it assumes partners agree about which part of a property is the predictable one; two partners that explain genuinely different parts of it are treated as overlapping, so it is discounted slightly more than it should be. It can be switched off per Space.
• Units are reconciled from names and values only. radius_cm and radius_mm merge and convert (see Units), but a unit nowhere written down cannot be guessed, and currencies and temperatures are deliberately left alone.
• Set similarity compresses as lists grow. Chance is now measured per set size (above), so a long list is no longer mistaken for evidence — but the Chamfer itself still leaves little room at the top: with two hundred values a side, exact agreement scores 1.00 against a chance level of 0.93, so a perfect match and a coincidence are only 0.07 apart. Long lists are therefore weak evidence in either direction. An overlap term alongside the semantic best-match would widen that range, and rarity weighting inside the set would let an unusual value count for more than a common one; neither is implemented.
• Paired queries prune less well. Block bounds for arrays of objects are computed from flattened values, so they are loose exactly when pairing matters. On 200,000 shirts, {"color": "red", "size": "S"} inside one variant takes 82 µs (2 of 196 blocks), and adding a price to that variant takes 1 ms (141 blocks), against 14 µs for a single value. Results are exact either way. Element-aware block summaries would tighten this.
• Kind inference is heuristic (90% thresholds). Per-Space hints ("zip": "categorical") would be a small addition.

Search and scale 1. Exact free-text queries on huge vocabularies still sweep the vocabulary (about 4 ms at a million titles). Approximate search avoids the sweep at 1–1.6 ms with high recall. Its per-object estimates are random reads into a 16 MB table, so it is memory-latency-bound rather than bandwidth-bound; smaller records would trade recall for speed. 2. The VNNI kernels have still not run anywhere. - Verified on real hardware: the AVX2 kernel, by CI on an x86-64 runner — 12.3 ns per dot against scalar's 45.8, or 20.9 G int8 MAC/s — and the whole core suite as x86-64 code, there and under QEMU with and without AVX2. - Not yet executed: AVX-VNNI and AVX-512 VNNI. QEMU does not emulate them and the CI runner does not have them (dispatch: avx2 []), so they wait for a machine that does. Their assembly has been inspected, and they self-check against scalar before being used, so the fallback is safe either way. - No timings from your own CPUs: run simd_bench where you deploy. 3. No leader election or failover. Replicas follow a directory (see Replicas); which server owns the writes is your decision, not the system's. 4. No rebalancing or shard replication. Splitting a Space across nodes is a copy you perform (see More than one machine); moving shards between nodes, or holding one shard on two nodes so a loss is survivable, is up to you. 5. Streamed uploads are uncompressed, and change sets (POST /spaces/{space}/objects) still arrive as one JSON body. A population larger than memory can be learned over HTTP or from the CLI, but changing it happens in batches that fit.

Authorization

• The statistics are pooled, deliberately. A caller who may see part of a Space can still read the shape of all of it: distributions, the most frequent values with their counts, whether a value exists anywhere (see Who may call it). Where the population itself is the secret, give each tenant its own Space.
• A token cannot be revoked, only expired. The lifetime is the revocation window.
• Grants are per object, not per field. An object is visible or it is not; hiding some properties of a visible object is a different feature.
• No audit log. Nothing records which principal read what.

Learning

• Incremental versions cost O(size of the touched shards). Within a shard, the column store is rewritten. Smaller shards make small changes cheaper, at the cost of more id-index probes per lookup.
• Deletion refuses when versions of the Space exist on disk but aren't loaded, e.g. learned with a different embedder, because it can't verify what depends on them.

License#

Hilliance is source-available under the Business Source License 1.1, not an open-source licence.

• You may read, modify, self-host and run it in production, including behind your own product.
• You may not offer it to third parties on a hosted or managed basis — that is, sell its functionality as a service.
• Four years after a version is published, that version becomes Apache-2.0. The clock runs per version.
• Other terms are available: ask, and a commercial licence can say something different.

Where the source is. It is not published in a public repository, so there is no link to one here. It is available on request under this same licence — write to info@hilliance.com — and what you receive builds the image you can already pull.

Commercial use#

Nothing here is gated. There is no licence key, no trial, no edition that withholds a feature: the build you can download is the whole engine, and whether you may use it is a question about the licence rather than about the software.

Free, with nothing to ask and nobody to tell:

• running it in production, at any size, for your own business;
• putting it behind your own product, where what your customers buy is your product;
• modifying it, and running what you modified;
• evaluating it, for as long as you like.

Needs a commercial licence — write to info@hilliance.com:

• offering Hilliance itself to third parties as a hosted or managed service, which is the one thing the Business Source License holds back;
• terms this licence does not give you: a perpetual grant, a different change date, warranties, indemnities, or anything your procurement needs in writing;
• support with an answer time attached. The licence costs nothing to comply with and promises nothing in return; a support agreement is where response times, upgrades and someone to call live.

Four years after a version is published it becomes Apache-2.0, and that version needs nothing from anybody. The clock runs per version, so the one you are reading has its own date.

If you are unsure which side of the line you are on, the question that usually settles it: does what your customer pays for consist substantially of Hilliance's functionality? Running it inside a furniture retailer's search is the first list. Reselling it as "search as a service" is the second.

Every dependency is permissive (MIT, Apache-2.0, BSD, Zlib, Unlicense), and the Docker image redistributes the Model2Vec model under MIT. NOTICE credits them. deny.toml lists the licences a dependency may carry, and CI runs cargo deny so a copyleft crate cannot arrive unnoticed in an upgrade and make the image undistributable.

Glossary#