checksums.csv question

[gabrielstedman]

Am I correct in assuming that this proxy is intercepting the hash of the original file and translating to the encoded version to trick clients into thinking it's the same file to avoid duplicates/reupload?

If this is the case, won't a csv become very big overtime and slow to read/write? Wouldn't it be better to store this info in a DB?

PS: I really like the changes this fork introduces.

[joojoooo]

yes, but it's the other way around and it translates only 1 way, from modified hash to original hash. It intercepts the upload file and calculates the original hash from it and modified hash from the converted file. Then replaces any modified hash from Immich with the original.

Of course the CSV gets bigger the more images you upload, a database would take even more space.
The whole CSV is read into a map at startup so lookup can't get any faster and a database is surely slower than a map stored in RAM

Reading a million hashes CSV into a map only takes like a second or less on my machine and only takes a few megabytes of RAM/storage, so size or reading speed are not an issue.

Writing to the CSV is also as fast as possible: new hashes are appended at the file end in a separate goroutine. The whole file never gets fully rewritten and no entry ever gets deleted.

I don't think there is any database that can achieve the speed, size, simplicity of this CSV solution

[gabrielstedman]

This makes perfect sense thanks for the explanation