Detect duplicates

[brainstorm]

Basic primitive: Detect which data has been unnecessarily been duplicated across the filesystem(s). Can be reused by higher level modules.

Scenario: A user has copied several .fastq files with same content across the filesystem.

Action: Detect them and compile a list with paths to those.

[jhagberg]

iRODS controlled files? Filesystem?

[samuell]

Would be very nice. Though, as written in another issue, I wonder how this can be integrated with our iRODS strategy. As long as files are managed by iRODS, iRODS can manage multiplicity (and actually deviates the need for it at all), and while files are outside iRODS, it is harder to have iRODS keep track of them in a good way, without loosing the sync between the file system and it's metadata catalog (the icat database)... food for thought ...

[samuell]

If we had tons of iRODS slaves, we could of course run iRODS FUSE everywhere :) ( ... but probably never ever get any speed close to panasas ... :( )

[jhagberg]

Nope.

But I think we have a proof of concept how to use the speed of panasas but still have the files managed by iRODS.

panasas should be one or many "resources" in iRODS. As well as SweStore.

I need to investigate if there exist some kind of "dedup" in iRODS.

[brainstorm]

Sorry guys, I was merely referring to dedup within iRODs, not on the (external) filesystem(s).

[jhagberg]

I have asked a short question on iRODS chat about deduplication.
http://groups.google.com/group/irod-chat/browse_thread/thread/2a291b45d71ef701

[brainstorm]

As they comment in the list, detecting is fairly simple, that's what we want to go for, for now, that would be very valuable.