Skip to content
DuoBolt

How duplicate matching works

A duplicate file, in DuoBolt, means one thing: the contents are identical, byte for byte. Not a similar name, not the same size, not a matching date. That definition is what makes the results safe to act on, and everything on this page is in service of reaching it without reading your whole disk.

For photos that merely look alike, that is a different question with a different engine — see how grouping works.

The pipeline

Section titled “The pipeline”

  1. Walk and admit

    DuoBolt walks the folders you chose and decides what is even eligible: regular files only, inside the size range, past the system/hidden rules and the extension filter. Files under folders whose extension you excluded (.app, .photoslibrary) are pruned with their whole subtree.

  2. Partition by size

    Files are bucketed by exact byte length, and any size with only one file is dropped immediately. Two files of different sizes cannot be identical, so the overwhelming majority of a disk is eliminated here — without a single byte being read.

  3. Prehash the head and tail

    For each surviving bucket, DuoBolt hashes the first and last 128 KiB of each file and compares those. Files whose ends differ cannot be duplicates, and are dropped without ever being read in full.

    On network volumes this deliberately weakens to a 4 KiB head-only read — round-trips cost more than reads there, so more candidates are allowed through to the next stage instead.

  4. Hash in full with BLAKE3

    Only what survives is read end to end and hashed with BLAKE3, streamed in chunks sized to the file — 128 KiB for small files up to 16 MiB for anything over half a gigabyte.

  5. Group

    Files sharing a full-content hash become a group. A group needs at least two files to exist at all, paths are sorted, and groups are ordered by how much space they actually waste.

The prehash is a speed optimization only. Turning off Two-stage scan changes how long a scan takes, never what it finds.

What the result means

Section titled “What the result means”

A group is a proof of identity

Every file in a group produced the same full-content BLAKE3 hash. Names, folders, dates, permissions and tags played no part in it.

Reclaimable is not group size

A group of five 4 GiB files does not free 20 GiB — it frees 16, because one copy stays. On macOS it may free less still, if some of those copies are APFS clones.

What DuoBolt does not claim

Section titled “What DuoBolt does not claim”

Being straight about the edges is more useful than a perfect-accuracy badge.

  • It compares hashes, not bytes. DuoBolt declares a duplicate when two files produce the same 256-bit BLAKE3 hash; it does not then re-read both files to compare them byte by byte. BLAKE3 is a cryptographic hash and no collision has ever been produced, but the distinction is real and worth stating.
  • Small files are outside the default scan. The minimum file size defaults to 64 KiB, which means small text files, config files and every zero-byte file are not considered. Lower it if that is what you are after.
  • Clone detection is macOS and APFS only. On Windows, on HFS+, and on any network share, DuoBolt cannot tell whether two copies share storage — so reclaimable space there is an upper estimate.
  • Hardlinks have no separate handling. With Collapse symlinks on — the default — a hardlinked file is admitted once and never appears as its own duplicate. Turn it off and each link counts as a full copy, including in the reclaimable figure.
  • Unreadable files are skipped, not reported. A file DuoBolt cannot open is left out of the scan rather than raised as an error.

Performance principles

Section titled “Performance principles”
  • Multi-core hashing with thread counts chosen from the machine, not guessed.
  • Streaming reads with chunk sizes scaled to the file, so a 40 GiB disk image never lands in memory.
  • A hash cache keyed on modification time, so a repeat scan re-reads only what changed.
  • Deterministic grouping — the same folders with the same settings produce the same result, in the same order.

Best practices

Section titled “Best practices”
  • Review before removing. Everything goes through Review Selection; nothing is removed by scanning.
  • Protect what must not be lost. Protected folders make files unselectable rather than merely warning you.
  • Use Bulk Select rules — Keep Newest, Keep Oldest, Keep Protected — instead of ticking hundreds of boxes by hand.
  • Move to Archive when you want a reversible removal that does not depend on the Trash’s retention.
  • Be careful on network volumes: files deleted from a NAS bypass the Trash and are removed at once. That is an OS limitation, not a DuoBolt choice.
  • On macOS exclude directory extensions like app and photoslibrary, and keep Detect APFS clones on so shared storage is not counted as free space.
  • Narrow with an extension include list when you know what you are hunting — it is the single biggest scan-time saving available.

Next

Setting up a scan — every control in Scan parameters and what its default is.