Include version number in manifest

[mofosyne]

Was looking at your current output which looks like

{
    "files": {
        "example1.png": {
            "hash": "70740a80c43a42beecc70b4e9e1e8f2e25272e1e5fc15f691266848e9359403f",
            "size": 28778
        },
        "example2.png": {
            "hash": "cee5298da94a413da7e5e718a9c36c02feefe42904bac13f819b7021f8612f1d",
            "size": 14256
        },
        "example3.png": {
            "hash": "75f20a9723d5597404e9ffac44f8e26429f3a125898e6a4b9ea4c80aff9a1b28",
            "size": 27745
        }
    }
}

While I'm not expecting a manifest to be infinitely extensible, maybe you want to make it so you can evolve the hash?
So perhaps you would prefer this manifest format instead?

{
  "version": 1,
  "defaults": {
    "hash": "blake3"
  },
  "files": {
    "example1.png": {
      "hash": "70740a80c43a42beecc70b4e9e1e8f2e25272e1e5fc15f691266848e9359403f",
      "size": 28778
    },
    "example2.png": {
      "hash": {
        "blake3": "cee5298da94a413da7e5e718a9c36c02feefe42904bac13f819b7021f8612f1d",
        "sha256": "8d969eef6ecad3c29a3a629280e686cf"
      },
      "size": 14256
    }
  }
}

Plus versioning would help in maintenance.
Is there a file format documentation, perhaps you may also want this format to include description, timestamp, etc...

[casey]

In general, I'm a little skeptical of pluggable hash functions, and similar extensibility hooks, for a few reasons:

• If filepack encounters a manifest with a new hash function, all it can do is emit an error message and tell the user to a newer version of filepack that supports the hash function.
• Different hash functions have different security guarantees, different functionality (especially tree hashes vs non tree hashes, since tree hashes allow verifying sub-ranges of hashed content without needing everything), and different levels of maturity, so I don't think they're as interchangeable as one might think.
• Any kind of proliferation of different hash functions being used in manifests would be quite annoying, so I wouldn't like to even tacitly encourage it by providing the appearance of extensibility.
• In general, I think we're kind of past the point of major hash functions breaking and needing to be replaced.

All that being said, I do think adding a version field would be a good idea, so that if anything needs to change in a backwards incompatible way, old versions of filepack can produce a useful error message which references the new version, instead of something more inscrutable complaining about an unrecognized field.

[mofosyne]

Ah thanks. Had a watch of Rust BLAKE3 – Hashing at Speed for Large Files & Web Tokens (Jeremy Chone) and also found this research article Performance Evaluation of Hashing Algorithms on Commodity Hardware which explains that BLAKE3 pretty decent for large file.

The results of the evaluation show that Blake3 generally outperforms both SHA-256 and SHA-512 in terms of throughput and latency. However, the performance advantage of Blake3 varies depending on the specific hardware platform and the size of the input data.
REF: arXiv:2407.08284v1 [cs.CR] 11 Jul 2024

I'm interested in your bet that there is not likely any significant improvement in hash algorithms vs BLAKE3 in the foreseeable future for the application of file manifest, but yeah it's better than sha256 at least.

[casey]

I'm interested in your bet that there is not likely any significant improvement in hash algorithms vs BLAKE3 in the foreseeable future for the application of file manifest, but yeah it's better than sha256 at least.

It's definitely possible, although BLAKE3 in particular is very much designed for speed.

If there was a new hash function that was so good that it was worth adopting, it could be done with a version bump, instead of needing to add a specific way of plugging in new hash functions. And even then, it would take a few years at least for me to feel comfortable that the new hash function wasn't breakable. Even BLAKE3, which is 6 years old now, is a little "young" for a hash function 😃

[mofosyne]

That's alright then, your logic checks out. Happy with this approach :) nice job with this project.