monkragAll notes ↗
← Back to the graph

Check the quality

Duplicates

Separate repeated records from real events that happen to look alike.

2 min readStarter note

A duplicate is an extra representation of the same record or event within the scope you are checking. Similar-looking rows are candidates for investigation. They are not enough evidence, on their own, to justify deletion.

Define what should be unique before searching for duplicates. An invoice number might repeat because each row represents an invoice line. Two identical payments might be a duplicate import, or two payments that really happened. The source context decides.

A small-business example

A shop imports a bank statement twice. It also receives two equal payments from the same customer on the same day. Removing every repeated date-and-amount pair could erase one genuine payment while missing duplicated records with slightly different descriptions.

The bookkeeper compares the source statement references and the import batches, keeps a list of suspected duplicates and removes only confirmed extra copies from the working dataset. The original statements remain available for checking. The revised totals are then compared with the source.

Try this

Write down what makes one record distinct in your table. Find repeated values under that rule and inspect a few complete groups, including a group you expect to be legitimate. Record why each confirmed duplicate is an extra copy before removing it from a working version.

One row one thing establishes the row's meaning. Unique identifiers supplies stronger matching evidence, while Reconciliation checks the effect of a correction.