Dedup APFS clones in native bulk scanner#19
Conversation
Request ATTR_CMNEXT_CLONEID and ATTR_CMNEXT_PRIVATESIZE alongside the existing fileid/allocsize attrs by passing FSOPT_ATTR_CMN_EXTENDED. clone_id groups files sharing APFS extents; private_size is the byte count that would be freed by deleting the file (i.e. blocks unique to it, not shared with clone siblings). For each file: - clone_id == 0: charge full allocsize (file is not part of any clone family; private == allocsize anyway) - first occurrence of clone_id: charge full allocsize (covers the shared extents + this file's private blocks) - subsequent occurrence: charge private_size only (just this file's CoW-modified blocks; shared was already counted) Total for an N-member clone family becomes shared + sum(private_i), which equals actual on-disk usage. Correctly handles both pristine clones (private ~ 0) and heavily-diverged clones (private ~ allocsize). A naive 'first wins, rest zero' approach under-counted diverged clones — common in macOS Library/Containers where apps clone on install but data evolves over months. Per-file cost: two extra Pointer.readLong() calls, one ConcurrentHashMap.add(), and 16 more bytes per packed entry (~440-700 entries per 64 KB buffer, down from ~500-800). Buffer parsing layout: private_size at +56, clone_id at +64, variable region starts at +72. The JVM-fallback ParallelDirectoryScanner used in dev mode (mvn javafx:run) does not dedup clones — Java NIO doesn't expose either attribute — so dev builds still overcount.
|
Hi Scott! Thanks a bunch for the PR! This is a very helpful one. Now, should the ATTR_CMNEXT_CLONEID be 0x00000100 rather than 0x00000200? 0x200 looks like it would be ATTR_CMNEXT_EXT_FLAGS: The reason your 7% improvement looks okay despite this is that EXT_FLAGS is usually 0 on user data, so the cloneId == 0 short-circuit hides the bug, and most files fall through to baseline behaviour, and a few collide accidentally. Once the value is right, every real APFS clone family should be properly detected. A heads-up on perf once the value is corrected: because every APFS inode has a unique non-zero cloneId (it's the data-stream id, not a "set if cloned" flag), seenCloneIds.add(cloneId) will fire for every file rather than every clone. To keep the hash-set lean, the cleanest fix is to also request ATTR_CMNEXT_CLONE_REFCNT (bit 0x00001000, uint32) and short-circuit on refcnt <= 1 before touching the set, that way only files currently sharing extents get tracked. Two more things if you want to keep iterating (lower prio, but very nice to have):
Anyways, thank you so much for helping fix the overcounting on APFS! |
|
Also, the documentation is actually incorrect, and should be updated to reflect these changes. |
2 participants
This PR dedupes APFS clones in the native macOS bulk scanner improving the accuracy of the usage estimates. Some helpful screenshots - this fixes about a 7% over-read on my mac.
new
old
Per entry: two extra
readLongcalls, oneConcurrentHashMap.add, and 16 more bytes in the packed entry (~440 to 700 entries per 64 KB buffer, down from ~500 to 800). Layout:private_sizeat +56,clone_idat +64, variable region at +72.In practice, this is still very fast on my mac, but takes a fair bit longer - from ~40s up to ~70s to do the entire drive. Whether or not the juice is worth the squeeze I leave as an exercise for the maintainer!