Two rows arrive and something has to decide whether they are the same thing. The machinery for deciding is the same whether the thing is a person, a shop, or a tin of tomatoes.
What is not the same is which fields you are allowed to believe.
That is the part I got wrong for a long time, and the part I keep seeing other people get wrong. The maths is commodity. Knowing that a matching postcode means something for a clinic and almost nothing for a person is not.
The rule underneath all of it
From John Talburt, in Entity Resolution and Information Quality. It sounds obvious right up until you count how many systems break it:
Entities do not exist in the information system. They exist in the real world. Information systems store references to entities, never the entities themselves.
A row is not a person. A row is a claim somebody made about a person, using the fields their system happened to have, for their own purposes.
Two rows about one person will disagree, because the two people who wrote them wanted different things.
I find that distinction clarifying rather than pedantic. A system that believes it stores people will eventually tell you it is certain about one.
The vocabulary, and why each word earns its place
Talburt is careful with terms in a way that pays off later. Four of them do most of the work.
An entity is the thing in the world. Your neighbour. The clinic on the corner. It has no fields, because it is not data.
An entity reference is a record that mentions it. A row in a spreadsheet, a name in a PDF, a line in a registry. References are what you actually have.
Identity attributes are the fields of a reference that could distinguish one entity from another. A national ID number. A surname. A registered company number.
Descriptive attributes are everything else. Eye colour, opening hours, staff count. Useful information, no help at all in telling two things apart.
Two references that point at the same entity are said to co-refer. Resolution is the act of deciding whether they do, and that word is doing something specific. You are not measuring similarity. You are deciding co-reference.
Talburt has a longer vocabulary for what happens after that, and it is worth knowing the shape of it even if you never build one. Keeping identity stable over time is entity identity information management, and the store it lives in is an identity knowledge base: a record, per entity, of what has been decided about it, so that next month’s data can be matched against last month’s conclusions rather than starting again.
That is a stateful system with its own governance obligations. Deciding co-reference is not.
The five activities
Talburt decomposes the whole field into five. Naming them is useful mostly because it shows how much of “entity resolution” is not the matching step everybody talks about.
| Activity | What it means | |
|---|---|---|
| 1 | Reference extraction | getting references out of documents and systems |
| 2 | Reference preparation | what references look like when compared |
| 3 | Reference resolution | deciding co-reference |
| 4 | Identity management | holding identity steady over time |
| 5 | Relationship analysis | what connects the entities once resolved |
Most tooling sells you number three.
What counts as an entity
Anything you can point at twice and mean the same thing. That is a low bar and it is meant to be.
People. Places, including clinics, addresses and settlements. Products, at whichever level you decide counts. Organisations. Also vessels, accounts, creative works, events, and any number of things nobody has written up properly.
The test is not whether something feels important. It is whether two references to it can disagree while still meaning the same thing.
| Thing | The conditional group is | The question you answer first |
|---|---|---|
| People | credentials lent by a relationship | who assigned this, and is the relationship still alive? |
| Places | geometry and landmarks | is this proposing a candidate, or deciding one? |
| Products | size, colour, edition, pack | the same at what level: model, item, or batch? |
| Organisations | legal form, and site against operator | is this the party, or the place the party operates from? |
Where each one breaks
People is where the philosophy stops being optional, because a database is a committed bundle theorist with no substance underneath to hold the attributes together. Three tiers of identity data, why most of your record is borrowed rather than yours, and the uncomfortable fact that Customer 360 is the same technical operation as a patient record with a different beneficiary.
Places is where space is the trap. Coordinates for one clinic routinely sit kilometres apart between surveys, so geometry proposes candidates and is never allowed to decide one. Also the address-as-identity argument, and the billions of people who live somewhere named rather than addressed.
Products is where the question is malformed until you fix a granularity. Two tins of tomatoes are the same product and two different things, both correctly, and you cannot answer until somebody says which they meant.
Organisations has no note yet. The hard case is a site and the company operating it sharing a name and a coordinate, so every string signal and every spatial signal points the wrong way at once, and only a declared class refutes it.
Relationships, and the graph everyone wants to build
The fifth activity is the one people jump to, because it is where the interesting questions live. Who is connected to whom. Which accounts share a household. Which suppliers sit behind one parent company.
There is an ordering constraint and it is structural rather than a preference.
Relationships connect entities. Before resolution you do not have entities, you have references. Build the network first and one person appears in it three times, under three spellings, with their connections split between the copies.
You have not modelled a network. You have modelled your data-entry history.
This is the argument for what gets called an entity resolved knowledge graph: resolve first, then let the resolved things be the nodes. The name is newer than the idea, and the idea is just Talburt’s ordering taken seriously.
It matters more than it sounds. A graph over unresolved mentions does not fail loudly. It gives you plausible answers about a world that is subtly wrong, and centrality scores computed over duplicated nodes look exactly like centrality scores computed over real ones.
One thing I would add to the standard telling. A relationship claim is itself personal data. Who lives with whom, who transacts with whom, is often more sensitive than either party’s name, and a graph that resolves people carefully and then treats the edges as neutral infrastructure has protected the wrong half.
What is deliberately missing
These absences are decisions, so they are worth naming.
Flows. Lots, consignments, containers. Two hundred farms’ cocoa goes into one container. The question is not are these the same but did this become that, which is directed, non-symmetric, and not an equivalence relation at all.
Under commingling, identity genuinely does not survive: the referent becomes a mass balance rather than an object. Treat it as co-reference and the system either asserts false precision or collapses into uselessness.
Documents. A document has an issuer, a subject, and a self-declared origin that is trivially forgeable. That is a provenance ladder, not a co-reference problem.
Events, accounts, vessels, works. Each is a real type with its own conditional group, and none has been worked through properly. A financial account expires with the relationship that issued it. A vessel has an IMO number that survives renaming and reflagging, which makes it the rare case where a durable identifier actually exists.
If one of those is your problem, the vocabulary above should transfer. If it does not, I would like to know.
Where I run out
The vocabulary is good. It is the best I have found for keeping the question straight, and most confusion I meet in this work is someone collapsing entity into reference without noticing.
What it does not give you is the sorting. Talburt tells you identity attributes are the ones that distinguish entities. He does not tell you that a postcode distinguishes clinics and barely distinguishes people, because that is not a fact about entity resolution.
It is a fact about postcodes, and clinics, and where the data came from. Somebody has to know it, write it down, and keep it current, and that part has no theory at all.
Related
- What entity resolution actually is, on my thoughts about entity resolutions.
- Talburt, Entity Resolution and Information Quality (2011), for the vocabulary above