Comments (5)
(side note - how big are the 'tiny blocks' referenced in the algorithm?)
They are also configurable:
--max-prefix-size <BYTES>
Maximum prefix size to check in bytes
Units like KB, KiB, MB, MiB, GB, GiB are supported.
Default: 16KiB for hard drives, 4KiB for SSDs
--max-suffix-size <BYTES>
Maximum suffix size to check in bytes
Units like KB, KiB, MB, MiB, GB, GiB are supported.
Default: 16KiB for hard drives, 4KiB for SSDs
from fclones.
@clunkclunk the algorithm you describe is implemented by https://github.com/kornelski/dupe-krill
from fclones.
Why not have some more 'tiny block hashes' at various points in the file aside from the first and last tiny blocks to determine uniqueness before doing the time consuming entire file hash?
Can you give me an example of a situation where files of the same size would match with the beginning and end, but not the middle? Even matching the ends turns out to filter out very few files. I believe such situations would be extremely rare and that wouldn't justify the added complexity and cost.
from fclones.
For reference on a way to speed up dedupes, I typically use this is like so:
fclones group --cache /my/media/folder --min 500MiB --max-prefix-size 128MiB --max-suffix-size 64MiB --skip-content-hash
I set a large prefix size, suffix size, and skip the content hash alltogether. As well as skip small files where there's a higher false negative likelihood and less benefit to this approach. I've processed multiple thousands of media files and seen zero false negatives with cursory manual investigation.
For what it's worth, I do see value in random / "statistical" checks, though like mentioned it would add a lot of complexity. What I like about it is that it gives you a "reliable" statistical confidence instead of just the beginning and end. You get to set a threshold of how comfortable you are that is mostly independent of the kind of file, rather than relying on the fact that the middle chunks haven't been tampered with.
from fclones.
I can't say what files they are (I can't seem to find a verbose option), but I see this when scanning a very large directory:
[2023-10-27 13:54:46.198] fclones: info: Found 2303 (276.5 GB) candidates after grouping by prefix
[2023-10-27 13:54:53.994] fclones: info: Found 2298 (274.9 GB) candidates after grouping by suffix
[2023-10-27 14:50:36.020] fclones: info: Found 2186 (271.0 GB) redundant files
These should all be zip and rar files.
I should open up a ticket for a verbose option for what's unmatched between steps 4 and 5, but I definitely have files with the same size, prefix, and suffix that don't have the same content - or maybe just update to the latest version.
Another option would be to do random chunk matching of a configurable percentage of the file size using a seed based on the file size to pick which chunks.
from fclones.
Related Issues (20)
- Change to group progress output in 0.33.0 breaks piping to file via stdout HOT 2
- fclones depends on libc6:amd64 (>= 2.36) HOT 4
- On android, running on termux, hard or soft link creation fails after creating the dupes file.
- Hard links are reported as duplicates
- Feedback, showdown against 3 other tools HOT 4
- How to find AND isolate/extract unique files that are in one directory but not another? HOT 2
- Add ability to filter by magic HOT 1
- How to deduplicate and compress?
- Sort file chunks instead of hashing HOT 1
- fclones scans a ton of files that have no chance of matching HOT 4
- fclones `move` ignores standard format log from `group`?
- /var/lib/snapd/void: Permission denied HOT 1
- ARM binary HOT 1
- group output according to order on command line
- "remove" does not work if some files have future modification dates HOT 2
- Why hash whole files? HOT 2
- hard links treated as duplicates
- Pattern matching doesn't work at all on Windows
- the dir specified in --cache is empty after a complete run
- [Feature Request] Support for `dedupe` command on ReFS on Windows HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from fclones.