Dup Detect

Finds duplicate photos by what they contain — not by filename, date or size — and files the extras away into a folder you choose.

A scan moves nothing. It produces a list. The only thing that touches a file is the Move button under that list, and it asks first, in numbers.

Five passes, cheapest first, each one narrowing what the next has to look at. That ordering is what lets a whole-drive scan finish in reasonable time: the expensive tests only ever run on the handful of files that survived the cheap ones.

The passes do not each report their own findings. They contribute evidence to a single grouping that is merged as they go. This matters in a case that comes up constantly:

a.jpg and b.jpg are byte-identical. c.png is the same picture, saved losslessly.

Handled pass by pass, the two JPEGs would pair off in pass 3 and — being "already resolved" — never reach pass 4, so the PNG would never be connected to them and the answer would quietly omit the best copy. Here all three come back as one set, and the PNG is the keeper.

The expensive passes decode one file per set and share what they learn with the rest of it, so decoding cost tracks the number of distinct pictures rather than the number of copies.

If a group is held together partly by a perceptual guess, it is shown as a guess — never as a certainty — even if some of its members are byte-identical to each other.

Ranked in this order. The first difference decides it, and the app shows the reason it picked each keeper, in words, next to the file.

• More pixels — a 4000×3000 beats a 1000×750 of the same shot, and a full-size photo always beats its thumbnail. This is the rule that matters most, so it is first.
• Greater bit depth — 16-bit holds tone an 8-bit file cannot.
• Better format — lossless before lossy: raw/TIFF/PSD → PNG/BMP → JXL/AVIF/HEIC/WebP → JPEG → GIF.
• Larger file — within one format, more bytes means less was thrown away.
• Shallower path — Photos/x.jpg over Photos/import/tmp/x.jpg.
• Older, then shorter name, then alphabetical.

The last three exist so the outcome never depends on the order the drive happened to be walked in. Rules 1–4 tie by definition for byte-identical copies, so those fall through to 5–7 — which is why the tool behaves exactly like a plain de-duplicator when formats are not in play.

• Anything that is not an image, decided by reading the file's magic bytes — a text file named .jpg is left alone, and a photo with no extension is still found.
• Videos, unless you tick the box. They are then compared as whole files only, never frame by frame.
• $Recycle.Bin, System Volume Information, Windows and similar system folders.
• Symlinks and junctions are never followed — that is how a scan ends up in an infinite loop.

• Move, never delete. The word "delete" appears in the source exactly once, in a comment saying so.
• Cross-volume moves are copy → verify → unlink. The source is only released after the copy has been read back and its hash matches. A bad cable cannot eat a photo.
• Every move is recorded before the next one starts, in dup-detect-manifest.jsonl in the duplicates folder — so an interrupted run still has a truthful record, and is still fully undoable.
• Duplicates are filed by how they matched. Exact Copies, Same Picture and Looks Alike sub-folders are created as they are needed, so the matches worth checking by eye are never mixed in with the ones that are certain. Each moved file's note travels with it.
• Names never collide. Within a sub-folder, IMG_0001.jpg from two folders becomes IMG_0001.jpg and IMG_0001 (2).jpg. The manifest keeps the original path of each.
• The duplicates folder may not be inside the scanned folder. The app refuses, because otherwise the next scan would find the files it just moved.
• Unreadable files are reported, never guessed at. Locked, permission-denied and corrupt files are listed at the end and skipped.
• Every moved file is left with a note explaining itself — beside IMG_0001.jpg goes IMG_0001.jpg.json, naming the copy that was kept, how the two matched, and why that one stayed. Any single file can be checked on its own, in Notepad, without this app. Undo takes the notes away with the photos; turn them off next to the Move button if you would rather have a clean folder.

• Start with one folder, not the whole drive. Learn what the results look like on something you know well before pointing it at everything.
• Leave look-alike (pass 5) off for the first run. Passes 1–4 cannot be wrong. Get the certain duplicates out of the way first; they are usually the bulk of it.
• Put the duplicates folder on the same drive if you can. A same-volume move is a rename — near-instant. Across drives every file is copied, verified and only then removed, which is much slower (and much safer).
• Read the list before moving. Sort by the reason if anything looks surprising. The keeper's reason is written next to it in plain words.
• Turn look-alike on for a second pass, once the exact duplicates are gone — and read that list properly. This is the one where a wrong answer is possible.
• Do not delete the duplicates folder for a while. That folder is the undo. Once it is gone, so is the ability to put anything back.

If the image library could not be loaded, passes 4 and 5 are skipped and the scan says so. Whatever passes 1–3 found is still correct and still returned — you get the byte-identical duplicates, just not the cross-format ones.

They were locked by another program, permission-denied, or corrupt. They are skipped, never guessed at. Close whatever holds them and scan again.

Use Undo. It reads the manifest in the duplicates folder and puts every file back where it came from, along with its note. This works as long as that folder and its dup-detect-manifest.jsonl are intact.

Only on the portable Windows build: it unpacks itself to a temporary folder on every launch. That is the extraction, not the app. Use the installer on a machine you use regularly.

Dup Detect v · build · commit