Add Reed-Solomon support about ocaml-seqbox HOT 3 CLOSED

Clarification : using Reed-Solomon as erasure code rather than error correction code for easier implementation (can translate from implementation done by Backblaze : https://github.com/Backblaze/JavaReedSolomon)

The error detection scheme remains to be decided : using a SBX block as base unit vs using smaller blocks within SBX block as base unit.

The former allows error correction of the entire file, but not useful if all blocks are a bit corrupted thus failing the CRC checks.
The latter allows error correction of each individual SBX block, but need extra CRC codes for all smaller blocks within individual SBX blocks.

from ocaml-seqbox.

Progress update

Finished Galois module for the finite field calculation stuff
Finished Reed-Solomon matrix module
Reed-Solomon module still needs a lot of work

Gonna come back later after a few days of break

Note to self

• May want to change Galois stuff from using int to using bytes/char
• Current setup requires conversion from string/bytes to int in order to do RS calculations, this may be slow
• Klaus Post's Go implementation achieves 1GB/s per core, Backblaze's Java Implementation achieves 500MB/s
• SBx encoding currently takes quite a while anyway due to CRC and crypto hashing and not optimized for speed(around 50MB/s), so no need to try to catch up really
• Add is_reedsolomon_enabled predicate for common_params in sbx_blocks, which just returns true for the RS enabled versions, and use this predicate for reworking encode, decode module
• Not much to do for rescue or show module, maybe show more metadata regarding RS stuff
• Add RSC field to metadata block to indicate percentage of redundancy
• User should express redundancy as percentage in integer in [1, 100], same as parchive2
• Calculate RS stuff per 100 blocks, let n be the percentage, then arrangement of every 100 + n blocks = 100 data blocks + n parity blocks. Do things slightly differently if fewer than 100 blocks remain, use function below

calc_parity_blocks_number (blocks_remaining : int) (redundancy_percent : int) : int =
  (* use integer ceiling *)
  ((min 100 blocks_remaining) * redundancy_percent + 100 - 1) / 100

• Look into randomizing block ordering maybe, this will complicate encode module somewhat, but encode module is well written enough to add some changes(only need to modify encoder generation part)
• Look into allowing specification of a separate metadatablock file during decoding, this will complicate current reference block logic somewhat, but shouldn't be too bad since some decoder generation code can be reused

from ocaml-seqbox.

Development effort now directed towards rsbx.

Osbx is now under maintenance mode.

Closing all issues related to adding features.

from ocaml-seqbox.