Framework hinting standard about rust_serialization_benchmark HOT 1 OPEN

I'm really happy about this plan to improve the benchmarks! 🌟 👏

Document exactly which invariants crates are allowed to rely on. This encompasses data ranges and valid values; anything outside of the documented invariants are off-limits.

Strongly agree, but do be sure to define the phrase "rely on." I see a distinction between crashing/erroring if the invariant is broken and merely pessimizing the performance/size. For reference, bitcode hints are of the latter category (if you're wrong, you haven't broken anything except your bandwidth costs).

Make the benchmarks more comprehensive. If crates are able to turn a benchmark into a memcpy war, then their punishment will be more difficult schemas. This should be the preferred route for foiling metagaming.

Yes please, but do note there is also a compression war going on 😉 ⚔️

Compression algorithms will happily rely on any invariant they can detect, so please either remove those invariants or make them more realistic:

• Generate random strings from a much larger pool (aim for very low duplicate rates) or use enums instead
• Make the bit-width and random distribution of integers and floats match as closely as possible

Finally, don't forget to time the compression!

Allow some minor optimizations. Anything that only requires a little work is OK. My unhelpful metric would be "as much help as you can get from five minutes in a text chat".

That is a helpful metric! ❤️

For complex schemas, I'd say five minutes per type would be more scalable 👌

from rust_serialization_benchmark.