Git Product home page Git Product logo

checkseum's Introduction

Checkseum

Chekseum is a library which implements ADLER-32 and CRC32C Cyclic Redundancy Check. It provides 2 implementation, the first in C and the second in OCaml. The library is on top of optint to get the best representation of the CRC in the OCaml world.

Linking trick / variant

Then, as digestif, checkseum uses the linking trick. So if you want to use checkseum in a library, you can link with the checkseum package which does not provide an implementation. Then, end-user can choose between the C implementation or the OCaml implementation (both work on Mirage).

So, in utop, to be able to fully use checkseum, you need to write:

$ utop -require checkseum.c

or

$ utop -require checkseum.ocaml

In a dune workspace, the build-system is able to choose silently default implementation (checkseum.c) for your executable if you don't specify one of them. A dune-library is not able to choose an implementatio but still able to use the virtual library checkseum.

The trick permits us to compile checkseum with js_of_ocaml.

Compatibility with 32-bits architecture

checkseum was made to provide a Adler-32 or a CRC-32 implementation regardless the architecture - it takes the advantage of optint to get in any situation a 32-bits integer. However, it takes the advantage of the immediate int value for a 64-bits architecture (which let us to use, at most, 63 bits).

The C implementation provides the cheapest FFI between OCaml and C via annotations such as [@untagged] or [@unboxed].

Example

This is a simple example of how to use checkseum with CRC-32:

$ cat >digest.ml <<EOF
let digest ic =
  let tmp = Bytes.create 0x1000 in
  let rec go crc32 = match input ic tmp 0 0x1000 with
    | 0 -> crc32
    | len ->
      Checkseum.Crc32.digest_bytes tmp 0 len crc32 |> go
    | exception End_of_file -> crc32 in
  go Checkseum.Crc32.default

let () = match Sys.argv with
  | [| _; filename |] when Sys.file_exists filename ->
    let ic = open_in filename in
    let crc32 = digest ic in
    close_in ic ; Format.printf "%a\n%!" Checkseum.Crc32.pp crc32
  | [| _ |] ->
    let crc32 = digest stdin in
    Format.printf "%a\n%!" Checkseum.Crc32.pp crc32
  | _ -> Format.eprintf "%s [<filename>]\n%!" Sys.argv.(0)
EOF
$ cat >dune <<EOF
(executable
 (name digest)
 (libraries checkseum))
EOF
$ dune exec ./digest.exe -- digest.ml
3995955175

Benchmark

With the dune's profile benchmark (which requires bechamel), the user is able to produce a bench/main.html which compares checkseum.c with tcpip.

checkseum's People

Contributors

dinosaure avatar hannesm avatar jonahbeckford avatar misterda avatar reynir avatar rgrinberg avatar sternenseemann avatar thelortex avatar tmcgilchrist avatar vapourismo avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

checkseum's Issues

Use virtual libraries

Hi,

The custom link technique used in checkseum makes it impossible to use in certain contexts. For example, any library that depends on checkseum cannot be used with ppx_expect. Now that there is support in dune for this use case, can checkseum use virtual libraries?

Thanks

Build error with esy

looks like there are still some issues with esy, at least when esy is used for cross compilation. I attach here the logs for refernce (form https://github.com/mseri/doi2bib/runs/2255652909?check_suite_focus=true)

error: build failed with exit code: 1
  build log:
    # esy-build-package: building: @_linux.musl.x86_64/_opam_checkseum@path:.mocks/6ca89b3d/esy.json
    # esy-build-package: pwd: /home/runner/.esy/3/b/__linux.musl.x86__64__s____opam__checkseum-fadeaa0e
    # esy-build-package: running: 'cp' '-r' '-p' '/home/runner/.esy/source/i/opam__s__checkseum__opam__c__0.3.1__803db346/.' '/home/runner/.esy/3/b/__linux.musl.x86__64__s____opam__checkseum-fadeaa0e'
    # esy-build-package: running: 'mv' '/home/runner/.esy/3___________________________________________________________________/s/__linux.musl.x86__64__s____opam__checkseum-fadeaa0e' '_____install_____'
    # esy-build-package: running: 'mkdir' '-p' '/home/runner/.esy/3___________________________________________________________________/s/__linux.musl.x86__64__s____opam__checkseum-fadeaa0e/_esy'
    # esy-build-package: running: 'mv' '_____install_____' '/home/runner/.esy/3___________________________________________________________________/s/__linux.musl.x86__64__s____opam__checkseum-fadeaa0e/linux.musl.x86_64-sysroot'
    # esy-build-package: running: 'dune' 'build' '-p' 'checkseum' '-j' '4' '-x' 'linux.musl.x86_64' '@install'
            make freestanding/libcheckseum_freestanding_stubs.a [default.linux.musl.x86_64]
    make: opam: Command not found
    touch libcheckseum_freestanding_stubs.a
    # esy-build-package: running: './install/install.ml'
    Exception:
    Unix.Unix_error(Unix.ENOENT, "chmod", "_build/default/META.checkseum")
    error: command failed: './install/install.ml' (exited with 2)
    esy-build-package: exiting with errors above...
    
  building @_linux.musl.x86_64/_opam_checkseum@path:.mocks/6ca89b3d/esy.json
esy: exiting due to errors above
Error: Process completed with exit code 1.

I still have to investigate this properly, I will send a PR if I figure it out but I am very unexperienced with esy.

Cannot find file topfind. Unknown directive `require'.

Sorry if this issue is not properly addressed.
I try to add irmin-unix package to my project which is build with esy and reasonml.
But after trying to build it with esy build i get the error below.

error: build failed with exit code: 1
  build log:
    # esy-build-package: building: @opam/checkseum@opam:0.2.1
    # esy-build-package: pwd: /Users/pawelfalisz/.esy/3/b/opam__s__checkseum-opam__c__0.2.1-3b39b42e
    # esy-build-package: running: 'dune' 'build' '-p' 'checkseum' '-j' '4'
    # esy-build-package: running: './install/install.ml'
    Cannot find file topfind.
    Unknown directive `require'.
    error: command failed: './install/install.ml' (exited with 2)
    esy-build-package: exiting with errors above...

  building @opam/checkseum@opam:0.2.1
esy: exiting due to errors above

From my understanding, there is something wrong with ./install/install.ml file? But i dont have that much knowledge about ocaml and ocaml tools, to know what exactly went wrong.

Thanks in advance.

Build issues with Solo5

Best effort at integrating checkseum with a Solo5 unikernel here:

git clone https://github.com/mato/camel-service
cd camel-service/checkseum-unikernel
mirage configure --target hvt
make depend
make
mirage build
ld: cannot find -l:freestanding/liblaolao_freestanding_stubs.a
run ['ld' '-nostdlib' '-z' 'max-page-size=0x1000' '-static' '-T'
     '/home/g2p/.opam/4.07.0+fp+flambda/lib/pkgconfig/../../lib/solo5-bindings-hvt/solo5_hvt.lds'
     '/home/g2p/.opam/4.07.0+fp+flambda/lib/pkgconfig/../../lib/solo5-bindings-hvt/solo5_hvt.o'
     '_build/main.native.o'
     '-L/home/g2p/.opam/4.07.0+fp+flambda/lib/checkseum'
     '-l:freestanding/liblaolao_freestanding_stubs.a'
     '-L/home/g2p/.opam/4.07.0+fp+flambda/lib/zarith' '-lzarith-freestanding'
     '-L/home/g2p/.opam/4.07.0+fp+flambda/lib/gmp-freestanding'
     '-lgmp-freestanding' '-L/home/g2p/.opam/4.07.0+fp+flambda/lib/nocrypto'
     '-lnocrypto_stubs+mirage-freestanding'
     '-L/home/g2p/.opam/4.07.0+fp+flambda/lib/mirage-entropy'
     '-lmirage-entropy_stubs+mirage-freestanding'
     '-L/home/g2p/.opam/4.07.0+fp+flambda/lib/pkgconfig/../../lib/ocaml-freestanding'
     '/home/g2p/.opam/4.07.0+fp+flambda/share/pkgconfig/../../lib/mirage-solo5/libmirage-solo5_bindings.a'
     '-lasmrun' '-lnolibc' '-lopenlibm' '-o' 'http.hvt']: exited with 1
make: *** [Makefile:18: build] Error 1

And a similar error with the checkseum.ocaml variant.

Removing the last line from the META file fixes the build for the OCaml variant.

I'm not able to use Checkseum

Hello,

while trying to use git-unix from utop, I'm obtaining:
Error: Reference to undefined global `Checkseum'

I then tried directly with Checkseum, and obtained the same message. Unfortunately, I'm not sure what to check now? Do you have any idea?

$ utop -require checkseum

utop # Checkseum.Crc32.digest_bigstring;;
Line 1:
Error: Reference to undefined global `Checkseum'
─( 17:45:38 )─< command 1 >────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────{ counter: 0 }─

Here are some more informations:

$ opam --version
2.0.5
$ opam switch
#  switch          compiler                       description
→  4.08.1          ocaml-base-compiler.4.08.1     4.08.1
$ opam show checkseum

<><> checkseum: information on all versions <><><><><><><><><><><><><><><><><><>
name                   checkseum
all-installed-versions 0.1.1 [4.08.1]
all-versions           0.0.1  0.0.2  0.0.3  0.0.9  0.1.0  0.1.1

<><> Version-specific details <><><><><><><><><><><><><><><><><><><><><><><><><>
version       0.1.1
repository    default
url.src:      "https://github.com/mirage/checkseum/releases/download/v0.1.1/checkseum-v0.1.1.tbz"
url.checksum: "sha256=a3913487f951c5bebc8e44cce41878ef3e2263c9c81a1ed963b0a86268c84229"
              "sha512=f97c2e353c2bebdef2ed61cc6a32994fc9028c70e8c96b134ea92d178a29e2eda6c8bad1099b0a6d05971a69c94da2d5d476aeb4ddd9e802bb4bebe07f034579"
homepage:     "https://github.com/mirage/checkseum"
bug-reports:  "https://github.com/mirage/checkseum/issues"
dev-repo:     "git+https://github.com/mirage/checkseum.git"
authors:      "Romain Calascibetta <[email protected]>"
maintainer:   "Romain Calascibetta <[email protected]>"
license:      "MIT"
depends:      "ocaml" {>= "4.07.0"}
              "dune" {>= "1.9.2"}
              "optint" {>= "0.0.3"}
              "base-bytes"
              "bigarray-compat"
              "fmt"
              "rresult"
              "cmdliner"
              "alcotest" {with-test}
depopts:      "ocaml-freestanding" "mirage-xen-posix"
conflicts:    "mirage-xen-posix" {< "3.1.0"}
              "ocaml-freestanding" {< "0.4.3"}
              "mirage-runtime" {< "4.0.0"}
synopsis      Adler-32, CRC32 and CRC32-C implementation in C and OCaml
description   Checkseum is a library to provide implementation of Adler-32, CRC32 and CRC32-C in C and OCaml.
              This library use the linking trick to choose between the C implementation (checkseum.c) or the OCaml implementation (checkseum.ocaml).
              This library is on top of optint to get the best representation of an int32.

In "src-c/native/", 'STDC' is never ever defined

STDC' is defined in "zconf.h" (of zlib) that is not included in src-c/native/ therefore some code is very probably dead (and does not compile on my machine)

As an experiment, I replaced STDC by the more common __STDC__ (but actually the cases of "definition" in zlib are more complex of course) and I had to apply the extra

diff --git a/src-c/native/crc32.c b/src-c/native/crc32.c
index 53f01f5..a62d26c 100644
--- a/src-c/native/crc32.c
+++ b/src-c/native/crc32.c
@@ -532,7 +532,7 @@ static unsigned long crc32_big(unsigned long crc, const unsigned char * buf, siz
     len--;
   }
 
-  buf4 = (const crc_t *)(cost void *)buf; /* Obj.magic */
+  buf4 = (const crc_t *)(const void *)buf; /* Obj.magic */
 
   while (len >= 32) {
     DOBIG32;

to make it compile.

(I discovered that by trying to cross compile checkseum for windows in the context of ocaml-cross/opam-cross-windows where "src-c/native/size_t.h" is a burden because WIN32 is defined but BaseTsd.h does not exists as we are using a (cross compiler) gcc)

Confusion in how to (de)serialize Checkseum.Crc32.t

I am using Checkseum.Crc32, and I serialized using Optint.to_int32 and deserialized using Optint.of_int32, and my code didn't work. First I used Optint.pp to print the two checksums and indeed they were different:

crc -1407441036
crc' 2887526260

Then later I switched to using Printf.printf "Back checksum: %ld vs %ld" (Optint.to_int32 crc) (Optint.to_int32 crc'): and got Bad CRC -1407441036 vs -1407441036 which is confusing. Then I dug into the checkseum tests and discovered they were using Optint.unsigned_of_int32.

I suggest perhaps Checkseum.S.{of,to}_int32 to the interface so others don't make the same mistake.

Release of 0.2.0

  • CRC-24 implementation (see #17)
  • xxHash implementation (see #26)

I think we are ready to do a release of checkseum to prepare the next release of decompress. (/cc @cfcs)

Opam installation fails with dune cache on

Using the dune cache to speed up builds, I end up with this build failure with checkseum:

#=== ERROR while compiling checkseum.0.2.0 ====================================#
# context              2.1.0~alpha | linux/x86_64 | ocaml-base-compiler.4.10.0 | file:///home/opam/opam-repository
# path                 ~/.opam/4.10.0/.opam-switch/build/checkseum.0.2.0
# command              /bin/sh -exc echo "xen_linkopts = \"-l:laolao/xen/liblaolao_xen_stubs.a\"" >> _build/default/META.checkseum
# exit-code            2
# env-file             ~/.opam/log/checkseum-6-c5e94c.env
# output-file          ~/.opam/log/checkseum-6-c5e94c.out
### output ###
# /bin/sh: 1: cannot create _build/default/META.checkseum: Permission denied
# + echo xen_linkopts = "-l:laolao/xen/liblaolao_xen_stubs.a"

This is because files taken from the dune cache will not allow writes

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.