Curious about the project's status

[jordan-dlh]

Hello,

It is written in the README:

As of January 29, 2022 The Par3 specification is in near-final form. This repository will hold a reference implementation. After we have working code and worked out most of the bugs, the specification will be finalized.

Are we any closer to the final specification? What kind of help would be needed for par3 to become stable? Would it be essentially testing, or code writing?

As there is virtually no alternative (at least that I know of) to Parchive in its niche, I think a new and improved version of the format such as par3 is important. As a user I'm especially interested in the use of Unicode and Par inside another file.

Regards.

[mdnahas]

Hi Jordan, The next hurdle is a reference implementation. There was work towards an implementation on Windows. I'd prefer that the reference implementation run on Unix (Linux and other variations). I got busy with other things and wasn't able to help. Once the reference implementation has identified any problems in the specification and we've patched them, the specification will become final. Mike Nahas

…

On Tue, Mar 11, 2025, at 9:33 AM, Jordan wrote: Hello, It is written in the README: > *As of January 29, 2022* The Par3 specification is in near-final form. This repository will hold a reference implementation. After we have working code and worked out most of the bugs, the specification will be finalized. > Are we any closer to the final specification? What kind of help would be needed for par3 to become stable? Would it be essentially testing, or code writing? As there is virtually no alternative (at least that I know of) to Parchive in its niche, I think a new and improved version of the format such as par3 is important. As a user I'm especially interested in the use of Unicode and Par inside another file. Regards. — Reply to this email directly, view it on GitHub <#8>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAI52P7ZBVIYUGOMFFMYNS32T3X2DAVCNFSM6AAAAABYZCO4GOVHI2DSMVQWIX3LMV43ASLTON2WKOZSHEYTANZYHE3DCNQ>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***> jordan-dlh*jordan-dlh* created an issue (Parchive/par3cmdline#8) <#8> Hello, It is written in the README: > *As of January 29, 2022* The Par3 specification is in near-final form. This repository will hold a reference implementation. After we have working code and worked out most of the bugs, the specification will be finalized. > Are we any closer to the final specification? What kind of help would be needed for par3 to become stable? Would it be essentially testing, or code writing? As there is virtually no alternative (at least that I know of) to Parchive in its niche, I think a new and improved version of the format such as par3 is important. As a user I'm especially interested in the use of Unicode and Par inside another file. Regards. — Reply to this email directly, view it on GitHub <#8>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAI52P7ZBVIYUGOMFFMYNS32T3X2DAVCNFSM6AAAAABYZCO4GOVHI2DSMVQWIX3LMV43ASLTON2WKOZSHEYTANZYHE3DCNQ>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[Yutaka-Sawada]

I implemented sample application of PAR3 for Windows OS. While implementation, I wrote some note of updating points here. Please refer it for future specification.

Also, I fould a problem in current PAR3 format. Some users requested a method to repair PAR files, not input files only. There is no strict method to determine a PAR3 file is complete or not at this time. There is no way to re-construct a PAR3 file of original state. Refer my old post for detail. Though I wrote my idea to solve this problem, it's not efficient so much. I hope someone has a nice idea than me.

[mdnahas]

Yukaka Sawade, I would not describe this as a problem in the current PAR3 specification. The message thread includes the user's complaint <#1 (comment)>: "I am worried there are many other instances of damaged PAR files and I've no way to detect them currently." The user also said: "Basically there's no way damaged Par files can be detected unless you look at the verify in real time." So, the user wants to know about damaged PAR2 files and client *is already* detecting damaged PAR2 files. The client probably prints out a warning when a PAR packet doesn't satisfy the checksum. The user wants to find out about damaged PAR files without "looking at the verify in real time". In my opinion, the user is complaining that the client doesn't warn them at the end of execution that PAR files were damaged and doesn't tell them which ones were damaged. From his other statements, I do not think this user needs to repair the PAR3 files to their original state. And doing that is not always possible --- some damage cannot be recovered. If we tell this user which PAR files are damaged, they can delete any damaged PAR3 files and create new PAR3 files to replace them. Clients can identify damaged files. When a packet doesn't satisfy a checksum, that's a damaged file. When there is padding that is not zero bytes, that is a damaged file. When there are more than 7 zero bytes in a row, that's a damaged file. I'm happy to add a recommendation to the specification. The recommendation would be for clients to print out the the list of PAR files with damaged bytes in their final summary or to alert users when they find a file with damaged bytes. The only damage that clients cannot detect is when a PAR file is truncated between packets. Or a PAR file is missing. I'm okay with those failures. Do you think either of those cases is a big concern of users? Mike

…

On Wed, Mar 12, 2025, at 12:09 AM, Yutaka Sawada wrote: I implemented sample application of PAR3 for Windows OS. While implementation, I wrote some note of updating points here <https://github.com/Parchive/par3cmdline/blob/main/windows/Appendix.txt>. Please refer it for future specification. Also, I fould a problem in current PAR3 format. Some users requested a method to repair PAR files, not input files only. There is no strict method to determine a PAR3 file is complete or not at this time. There is no way to re-construct a PAR3 file of original state. Refer my old post <#1 (comment)> for detail. Though I wrote my idea to solve this problem, it's not efficient so much. I hope someone has a nice idea than me. — Reply to this email directly, view it on GitHub <#8 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAI52PZOIEXBEGU7TRL3Z3T2T66P7AVCNFSM6AAAAABYZCO4GOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOMJWGQ4DIOJYGU>. You are receiving this because you commented.Message ID: ***@***.***> Yutaka-Sawada*Yutaka-Sawada* left a comment (Parchive/par3cmdline#8) <#8 (comment)> I implemented sample application of PAR3 for Windows OS. While implementation, I wrote some note of updating points here <https://github.com/Parchive/par3cmdline/blob/main/windows/Appendix.txt>. Please refer it for future specification. Also, I fould a problem in current PAR3 format. Some users requested a method to repair PAR files, not input files only. There is no strict method to determine a PAR3 file is complete or not at this time. There is no way to re-construct a PAR3 file of original state. Refer my old post <#1 (comment)> for detail. Though I wrote my idea to solve this problem, it's not efficient so much. I hope someone has a nice idea than me. — Reply to this email directly, view it on GitHub <#8 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAI52PZOIEXBEGU7TRL3Z3T2T66P7AVCNFSM6AAAAABYZCO4GOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOMJWGQ4DIOJYGU>. You are receiving this because you commented.Message ID: ***@***.***>

[Yutaka-Sawada]

Do you think either of those cases is a big concern of users?

No, I don't think so. Most users won't care a rare flaw in PAR files. I wrote an inefficient method to confirm & restore original state of PAR files, because it's possible technically. The evaluation needs comparison to another solution how "Par inside Par feature" works, though I didn't implement it yet.

[jordan-dlh]

The next hurdle is a reference implementation. There was work towards an implementation on Windows. I'd prefer that the reference implementation run on Unix (Linux and other variations). I got busy with other things and wasn't able to help.

Once the reference implementation has identified any problems in the specification and we've patched them, the specification will become final.

Oh, that's good to hear. So there's mainly a time issue but the specification is already well advanced if I understand correctly. How advanced would you say is the reference implementation?

Regarding the discussion about detecting faulty PAR files:

The user also said:

"Basically there's no way damaged Par files can be detected unless you look at the verify in real time."

So, the user wants to know about damaged PAR2 files and client is already detecting damaged PAR2 files. The client probably prints out a warning when a PAR packet doesn't satisfy the checksum. The user wants to find out about damaged PAR files without "looking at the verify in real time". In my opinion, the user is complaining that the client doesn't warn them at the end of execution that PAR files were damaged and doesn't tell them which ones were damaged.

Do you think either of those cases is a big concern of users?

No, I don't think so. Most users won't care a rare flaw in PAR files. I wrote an inefficient method to confirm & restore original state of PAR files, because it's possible technically. The evaluation needs comparison to another solution how "Par inside Par feature" works, though I didn't implement it yet.

I can not tell for the cited user, but working myself in digital preservation I can say this: if we were willing to use PAR files in our processes, we would certainly want to validate them, that is to say: verifying if they are actually well formed, able to fulfill their job.

From his other statements, I do not think this user needs to repair the PAR3 files to their original state. And doing that is not always possible --- some damage cannot be recovered. If we tell this user which PAR files are damaged, they can delete any damaged PAR3 files and create new PAR3 files to replace them.

Clients can identify damaged files. When a packet doesn't satisfy a checksum, that's a damaged file. When there is padding that is not zero bytes, that is a damaged file. When there are more than 7 zero bytes in a row, that's a damaged file.

I'm happy to add a recommendation to the specification. The recommendation would be for clients to print out the the list of PAR files with damaged bytes in their final summary or to alert users when they find a file with damaged bytes.

If I may suggest: I think the specification should precise how to verify the PAR3 files themselves, and clients may (should?) offer an option to do just that. In digital preservation context that would be very valuable.

[Yutaka-Sawada]

How advanced would you say is the reference implementation?

I implemented Windows version in standard C language. So, it may be possible to run on Unix / Linux with small change of incompatible points. But, I'm not sure, as I don't know about Unix / Linux.

I found a way to use Linux OS on Windows OS. It's called as Windows Subsystem for Linux. Though the emulation is slow, gcc may be enough to solve porting issue and test compatibility. If nobody makes Unix / Linux version, I will try Linux version in future. Because I'm lazy, I want someone Unix user will make reference implementation. At this time, I'm busy to make a game, hehe.

[CiteClimb]

Sorry if this is annoying, but it seems like the last push to this repository was a while ago. Is this still being actively worked on?

[Yutaka-Sawada]

Is this still being actively worked on?

I'm waiting update of PAR Specifications. After the last draft, we talked some problem and possbilities. While I implemented par3cmdline, I changed some points. Current sample implementaion is different from the draft already. But, sample implementaion should follow formal specifications. Therefore, I suspend the development until updating new PAR3 specification.

[maksverver]

@CiteClimb Could you tell us a little about your use case/the reason for your interest in the project? It might inform decisions on whether or how to move forward with the project.

[CiteClimb]

Hi @maksverver,

As there is virtually no alternative (at least that I know of) to Parchive in its niche, I think a new and improved version of the format such as par3 is important.

As jordan-dlh mentioned above, Parchive is really unique (due to there just being a lack of systems/software that implement RS codes in this specific way, and ones that optimize for reliability, as I think par2 completely discards blocks of data with non-matching hashes, instead of even trying to use RS with them). I realize that this tool is mainly focused on transferring files through the internet, but I am mainly interested in this project because I want to use it for further protection against data corruption on storage mediums that could allow bit rot to accumulate to a point where the ECC built into the storage device cannot correct for the errors. This project fixes some possible incomplete math in the RS implementation from par2 (I believe), and also speeds it up using a better parallel implementation and a lot of other possible optimizations explained in /main/src/Appendix.txt. And faster, more efficient redundancy generation is which is why I am interested in this.

However, many other users probably still use this project for transferring files through the internet, so my opinion cannot speak for all of the others.

[RedKage]

@CiteClimb Could you tell us a little about your use case/the reason for your interest in the project? It might inform decisions on whether or how to move forward with the project.

I have been tracking this issue for years.
I am mostly using par files as a way to protect against bit rot

See, I have a classic 321 backup strategy
And I want my backups to be as simple as possible to restore.
For this reason, I do not use any kind of custom filesystem, no raid, no zfs, nothing. Just files, on a fat or NTFS FS.

Once in a while, when I know some parts of my backup will not change anymore, I will par2 it with a 1% recovery. If, for any reason, my HDD is damaged, I can theoretically repair up to 1% of damaged files.

What I like about par files is that they are optional, and live side by side with my data. I can ignore them and still use my backup right away. I do not need to unzip or whatever any archive which may provide recovery blocks.
I just grab my backup, and here are the files ready for use. Broken files? Ok then let's run some par2 recovery.

I'm eagerly waiting for the par3 release as sometimes the folders I par are quite large with many files (more than 30k). And par2 fails since there is a file count limit. From what I understand this limitation is gone with v3

I'm also running all of this on linux. So linux binaries are welcomed.

Now reading the above reply, my use case is the same reason

Thanks for your hard work so far

[maksverver]

@CiteClimb, @RedKage: thanks for sharing your use cases!

To summarize, it sounds you are primarily interested in the archival case, where par files are stored next to the data. (Rather than recovery blocks being downloaded separately, as with the classic Usenet file sharing case).

I'm also running all of this on linux. So linux binaries are welcomed.

The CLI tool currently works on Linux. If you want to test, you can grab binaries compiled for Ubuntu e.g. from here: https://github.com/maksverver/par3cmdline/actions/runs/21548856604 or compile from source yourself.

However, I wouldn't recommend using it for real archival purposes until the file format is finalized and the tool is updated accordingly.

[desertwitch]

@CiteClimb Could you tell us a little about your use case/the reason for your interest in the project? It might inform decisions on whether or how to move forward with the project.

I have been tracking this issue for years. I am mostly using par files as a way to protect against bit rot

See, I have a classic 321 backup strategy And I want my backups to be as simple as possible to restore. For this reason, I do not use any kind of custom filesystem, no raid, no zfs, nothing. Just files, on a fat or NTFS FS.

Once in a while, when I know some parts of my backup will not change anymore, I will par2 it with a 1% recovery. If, for any reason, my HDD is damaged, I can theoretically repair up to 1% of damaged files.

What I like about par files is that they are optional, and live side by side with my data. I can ignore them and still use my backup right away. I do not need to unzip or whatever any archive which may provide recovery blocks. I just grab my backup, and here are the files ready for use. Broken files? Ok then let's run some par2 recovery.

I'm eagerly waiting for the par3 release as sometimes the folders I par are quite large with many files (more than 30k). And par2 fails since there is a file count limit. From what I understand this limitation is gone with v3

I'm also running all of this on linux. So linux binaries are welcomed.

Now reading the above reply, my use case is the same reason

Thanks for your hard work so far

I'm using PAR2 to the same end and turned some of these ideas into a utility using par2cmdline under the hood.
It facilitates that orchestration of interacting with a large amount par2 files across one or multiple directory trees:

https://github.com/desertwitch/par2cron

Posting it here because I thought it may help you too, seeing as you've been tracking this for years. 😉
That being said I am also following PAR3 progress and am excited to test what's in store for us in the future...

It would be great if the repository could be cleaned up a bit and some information on the current state added.
Especially only one central draft of the new specification that's always updated, rather than in multiple places...
Right now it's a bit hard to figure out exactly which parts are already usable, and what is the latest state of affairs.

[CiteClimb]

@desertwitch Thanks for developing this useful fork of PAR2! Generating redundancy for an entire directory of files is a great idea.

It would be great if the repository could be cleaned up a bit and some information on the current state added.
Especially only one central draft of the new specification that's always updated, rather than in multiple places...
Right now it's a bit hard to figure out exactly which parts are already usable, and what is the latest state of affairs.

I agree with this; some updates in a centralized place in this repository would indeed be nice.