Skip to content
DuoBolt

BLAKE3: What It Is and How DuoBolt Uses It

A duplicate file finder answers one question: are these two files byte-for-byte identical? At scale the answer is to hash the candidates and compare the hashes. The hash function decides how fast that runs and how far a match can be trusted.

DuoBolt is built on BLAKE3 — a hash function published in 2020 that is several times faster than MD5 and SHA-256 on one core, can use every core, and has SHA-256’s security level. This page explains what BLAKE3 is, why we chose it over the alternatives, and how it flows through DuoBolt’s scanning pipeline.

What Is BLAKE3?

Section titled “What Is BLAKE3?”

BLAKE3 is the fourth in the BLAKE family of cryptographic hash functions. It was published in 2020 by Jack O’Connor, Jean-Philippe Aumasson, Samuel Neves, and Zooko Wilcox-O’Hearn. Unlike its predecessors (BLAKE, BLAKE2), BLAKE3 was designed for multi-core CPUs, SIMD instruction sets (AVX-512, NEON), and streaming workloads.

Three properties matter for a duplicate finder:

Tree-structured

BLAKE3 splits its input into 1 KiB chunks processed in parallel across CPU cores. The chunks combine into a Merkle tree, so hashing a large file scales almost linearly with the number of available cores.

SIMD-accelerated

Hand-tuned kernels for AVX2, AVX-512 (x86), and NEON (ARM) run several hash compressions per clock cycle. Modern laptops and desktops hit multi-GB/s single-thread throughput.

Cryptographically secure

BLAKE3 provides 128-bit collision resistance and 256-bit preimage resistance — the same security level as SHA-256. Collisions are computationally infeasible for any dataset you will ever scan.

BLAKE3 vs MD5 vs SHA-256

Section titled “BLAKE3 vs MD5 vs SHA-256”

MD5 and SHA-256 were designed for CPUs with one core and no SIMD. Rough figures:

Algorithm Throughput (single core) Parallelism Cryptographic status
MD5 ~600 MB/s None Broken — collisions producible in seconds
SHA-1 ~700 MB/s None Broken — SHAttered attack (2017)
SHA-256 ~400 MB/s None Secure, but slow
BLAKE2b ~1 GB/s Limited Secure
BLAKE3 Several GB/s Yes — multi-core and SIMD Secure

Numbers are rough steady-state throughput on a modern x86 or Apple Silicon laptop; SIMD-specific kernels widen the gap further. See the official BLAKE3 benchmarks for detailed measurements.

For a duplicate finder, the consequence is practical: scanning 1 TB of data with SHA-256 takes several times longer than with BLAKE3.

Why Collision-Broken Hashes Matter (Even for Dedup)

Section titled “Why Collision-Broken Hashes Matter (Even for Dedup)”

An adversarial file can collide with an unrelated file under MD5 or SHA-1. If your duplicate finder uses a broken hash and deletes one of the “duplicates”, you may lose the wrong file. BLAKE3 and SHA-256 have no known feasible collision attacks, so a match means byte-identical content with overwhelming probability.

DuoBolt does not rely on a secondary byte-by-byte compare after hashing, because BLAKE3’s 128-bit collision resistance makes a false positive astronomically unlikely.

How BLAKE3 Flows Through DuoBolt

Section titled “How BLAKE3 Flows Through DuoBolt”

The hash is not used as a single monolithic pass. DuoBolt applies BLAKE3 in two stages to minimize I/O on large datasets:

  1. Candidate grouping by size

    Files are first bucketed by byte count. Two files of different size cannot be duplicates — no hashing needed.

  2. Head+tail prehash

    For each size-matched candidate, DuoBolt BLAKE3-hashes only the first and last 128 KiB (on network volumes, the first 4 KiB only; see Core Concepts). This prefilter eliminates files that share a size but differ at the edges (common for media files with different containers or encoders).

  3. Full-content BLAKE3 hash

    Only candidates that pass the prehash are fully hashed. This is where BLAKE3’s multi-core and SIMD advantages matter most — on a large video or disk image, BLAKE3 spreads the one file across cores.

  4. Grouping and output

    Files with matching full-content hashes are grouped as duplicates, and the results screen renders each group with previews, paths and sizes.

The two-stage design means DuoBolt almost never reads a file’s full contents unnecessarily. On a NAS share that matters most: when few files share a size, a scan of 1.09 TiB is mostly a walk over the folder tree, and it finished in 10 seconds over SMB — see the NAS guide and benchmark results.

The prehash is a speed optimization only. Files whose head and tail bytes differ cannot be duplicates, so DuoBolt never reads them in full — but everything it reports as a duplicate has been confirmed by a full-content hash.

What Other Duplicate Finders Use

Section titled “What Other Duplicate Finders Use”

The tools in our benchmark suite:

  • Czkawka also uses BLAKE3 — the closest architectural peer. DuoBolt’s advantage over Czkawka on our benchmark suite — 1.66× across a source tree of 1.69 million files, narrowing to a tie on a NAS — comes from per-root parallelism and streaming I/O, not the hash itself.
  • dupeGuru uses xxHash, with MD5 as a fallback. xxHash is fast and not cryptographic. DuoBolt is 11.5× faster across a source tree of 1.69 million files on our benchmark suite, where per-file cost decides the result.
  • Gemini 2 does not name the algorithm it uses, so there is nothing here to compare it against.

Hash algorithm alone does not make a duplicate finder fast; Czkawka uses the same hash. A named hash also lets you look up its collision resistance yourself.

Learn More

Section titled “Learn More”

Next Steps

Section titled “Next Steps”
Core Concepts How DuoBolt's full detection pipeline works end-to-end
Performance Benchmarks Real-world scan times against Czkawka, dupeGuru, and Gemini 2
Download DuoBolt macOS and Windows — free to scan, free to start cleaning
DuoBolt vs Czkawka The closest architectural peer — same hash, different engine