mgsloan / store Goto Github PK

Fast binary serialization in Haskell

License: MIT License

Haskell 100.00%

store's Introduction

store

The 'store' package provides efficient binary serialization. There are a couple features that particularly distinguish it from most prior Haskell serialization libraries:

Its primary goal is speed. By default, direct machine representations are used for things like numeric values (Int, Double, Word32, etc) and buffers (Text, ByteString, Vector, etc). This means that much of serialization uses the equivalent of memcpy.

We have plans for supporting architecture independent serialization - see #36 and #31. This plan makes little endian the default, so that the most common endianness has no overhead.
- Another way that the serialization behavior can vary is if integer-simple is used instead of GHC's default of using GMP. Integer serialized with the integer-simple flag enabled are not compatible with those serialized without the flag enabled.
Instead of implementing lazy serialization / deserialization involving multiple input / output buffers, peek and poke always work with a single buffer. This buffer is allocated by asking the value for its size before encoding. This simplifies the encoding logic, and allows for highly optimized tight loops.
store can optimize size computations by knowing when some types always use the same number of bytes. This allows us to compute the byte size of a Vector Int32 by just doing length v * 4.

It also features:

Optimized serialization instances for many types from base, vector, bytestring, text, containers, time, template-haskell, and more.
TH and GHC Generics based generation of Store instances for datatypes.
TH generation of testcases.
Utilities for streaming encoding / decoding of Store encoded messages, via the store-streaming package.

Gotchas

Store is best used for communication between trusted processes and local caches. It can certainly be used for other purposes, but the builtin set of instances have some gotchas to be aware of:

Store's builtin instances serialize in a format which depends on machine endianness.
Store's builtin instances trust the data when deserializing. For example, the deserialization of Vector will read the vector's length from the first 8 bytes. It will then allocate enough memory to store all the elements. Malicious or malformed input could cause allocation of large amounts of memory. See issue #122.
Serialization may vary based on the version of datatypes. For example, Text serialized from text < 2 will not be compatible with Text from text >= 2, because the internal representation switched from UTF-16 to UTF-8.

Blog posts

Initial release announcement
Benchmarks of the prototype
New 'weigh' allocation benchmark package, created particularly to aid optimizing store.

store's People

Stargazers

Watchers

store's Issues

Split up store typeclass

What if we split up the store typeclass into separate poke and peek portions? There are a couple reasons for this:

store is already useful for defining serialization and deserialization for existing binary formats, even though this wasn't our immediate goal. For these applications, you might only care about reading or writing some particular data, and not both. I think @chrisdone in particular will agree this should be split up.
I suspect that the approach for speeding up compilation time described in #5 is not just specific to generics, and may apply to any multi method typeclass (maybe only in the presence of aggressive inlining?). This remains to be measured.

So, to me, it seems like we should split it up. The only downside is that it takes a bit more boilerplate to define both instances, even with generics. However, the TH magic should be able to be just as concise.

All that's left is bikeshedding. What do we call these classes? Maybe something like:

-- Constraint synonym
type Store a = (Peekable a, Pokeable a)

class Peekable a where peek :: Peek a
class Pokeable a where
    size :: Size a
    poke :: a -> Poke ()

Other possibilities are StorePeek and StorePoke, or instead perhaps HasPeek and HasPoke.

It may even be worthwhile to split out the size method, but I think I prefer it bundled with poke, since that's what it's used for.

@snoyberg @chrisdone @kantp @bitonic Thoughts on this bikeshed? :)

store-0.1.* misses upper bound on th-utilities, please consider editing bounds on Hackage

This allows cabal-based CI builds of stack to fail on CI (when not masked by caching as in commercialhaskell/stack#2428).

Because of the API change in th-utilities@1e23895fe90cd28dd1f232bc0ba4137dcdacfb6e, store-0.1.0.1 with th-utilities >= 0.2 doesn't compile. Instances on https://travis-ci.org/Blaisorblade/stack/jobs/149203734 and https://travis-ci.org/Blaisorblade/stack/jobs/149203736.

Relevant messages in the first case:

Downloading th-utilities-0.2.0.1...

Configuring store-0.1.0.1...
Building store-0.1.0.1...
Preprocessing library store-0.1.0.1...
src/Data/Store/TH/Internal.hs:47:18:
Could not find module ‘TH.ReifyDataType’
Use -v to see a list of the files searched for..

store-0.2.0.0 test fail: Data.Store, Manually listed polymorphic store instances, Roundtrips

In Stackage Nightly build for stackage-curator:

Failures:

  test/Data/StoreSpec.hs:287: 
  1) Data.Store, Manually listed polymorphic store instances, Roundtrips (GHC.Real.Ratio GHC.Int.Int8)
       uncaught exception: ArithException (arithmetic overflow)

  test/Data/StoreSpec.hs:287: 
  2) Data.Store, Manually listed polymorphic store instances, Roundtrips (GHC.Real.Ratio GHC.Int.Int64)
       uncaught exception: ArithException (arithmetic overflow)

Rename `Peek` and `Poke` to binary / cereal naming

We are already considering some breaking changes: #21

So this is a good time to consider other ones. While considering #31, it occurred to me that it may be convenient to make store's naming more consistent with binary. This would make it more straightforward to port code from binary to store. Some changes would need to change, but most of the time the custom implementations of put or get would not.

Compilation for big datatypes with Generics is quite slow

Currently, Data.Store.Instances uses GHC generics to define Store instances for all TH datatypes. This makes the module take 1 minutes 40 seconds to compile instead of 15 seconds.

I'm thinking we should do the following:

Add a clear warning to the docs that using generics for this will slow down your compile times quite a lot
Implement TH definition of Store instances

Add routines for peeking/poking from/to file

Currently Peek/Poke operations can be run on strict ByteStrings or Ptrs. I suggest adding routines that run them given a FilePath, and implement it using mmap, which avoids copying and should have good performance.

Add Liquid Haskell to our travis CI

Pinging @kantp , I think we should add Liquid Haskell to our CI, since we now have code that uses it!

Support selecting endianness for numeric types

Make it possible to specify the endianness of the serialization for numeric types.

This could be implemented as newtype wrappers LitteEndian, BigEndian, and HostEndian. Or it could be a bunch of separate combinators, as in binary and cereal.

Strange decode issue in flag cache and build cache with store branch of stack

The current blocker to releasing store publicly is that I ran into some funky behavior with stack ported to store. Not yet sure if it's a store issue.

Repro:

Checkout https://github.com/commercialhaskell/stack/tree/wip-store next to your store dir. (so that ../store/ exists from the stack dir)
stack build --fast within the stack directory
stack exec bash to enter into a shell with modified PATH
Run stack build -v twice and for the 2nd run observe a log like. It needs to be run twice as decode failures are expected the first run. The caches should be populated after that, though.

mgsloan@computer:~/fpco/stack$ stack build -v
Version 1.1.1, Git revision 10ba9938db538ef734d5b8385b5f84e8ff35c65e (dirty) (3605 commits) x86_64 hpack-0.14.0
2016-05-20 03:22:24.252525: [debug] Checking for project config at: /home/mgsloan/fpco/stack/stack.yaml @(stack_I9M2eJwnG6d3686aQ2OkVk:Stack.Config src/Stack/Config.hs:813:9)
2016-05-20 03:22:24.252868: [debug] Loading project config file stack.yaml @(stack_I9M2eJwnG6d3686aQ2OkVk:Stack.Config src/Stack/Config.hs:831:13)
2016-05-20 03:22:24.256728: [debug] Checking whether stack was built with libgmp4 @(stack_I9M2eJwnG6d3686aQ2OkVk:Stack.Config src/Stack/Config.hs:328:5)
2016-05-20 03:22:24.256900: [debug] Run process: ldd /home/mgsloan/fpco/stack/.stack-work/install/x86_64-linux/lts-5.15/7.10.3/bin/stack @(stack_I9M2eJwnG6d3686aQ2OkVk:System.Process.Read src/System/Process/Read.hs:283:3)
2016-05-20 03:22:24.270410: [debug] Stack was not built with libgmp4 @(stack_I9M2eJwnG6d3686aQ2OkVk:Stack.Config src/Stack/Config.hs:332:14)
2016-05-20 03:22:24.270684: [debug] Trying to decode /home/mgsloan/.stack/build-plan-cache/x86_64-linux/lts-5.15.cache @(stack_I9M2eJwnG6d3686aQ2OkVk:Data.Store.VersionTagged src/Data/Store/VersionTagged.hs:52:5)
2016-05-20 03:22:24.368780: [debug] Success decoding /home/mgsloan/.stack/build-plan-cache/x86_64-linux/lts-5.15.cache @(stack_I9M2eJwnG6d3686aQ2OkVk:Data.Store.VersionTagged src/Data/Store/VersionTagged.hs:61:13)
2016-05-20 03:22:24.383067: [debug] Getting system compiler version @(stack_I9M2eJwnG6d3686aQ2OkVk:Stack.Setup src/Stack/Setup.hs:341:17)
2016-05-20 03:22:24.383334: [debug] Run process: ghc --info @(stack_I9M2eJwnG6d3686aQ2OkVk:System.Process.Read src/System/Process/Read.hs:283:3)
2016-05-20 03:22:24.448115: [debug] Asking GHC for its version @(stack_I9M2eJwnG6d3686aQ2OkVk:Stack.Setup.Installed src/Stack/Setup/Installed.hs:94:13)
2016-05-20 03:22:24.448505: [debug] Run process: ghc --numeric-version @(stack_I9M2eJwnG6d3686aQ2OkVk:System.Process.Read src/System/Process/Read.hs:283:3)
2016-05-20 03:22:24.482177: [debug] Getting Cabal package version @(stack_I9M2eJwnG6d3686aQ2OkVk:Stack.GhcPkg src/Stack/GhcPkg.hs:165:5)
2016-05-20 03:22:24.482435: [debug] Run process: ghc-pkg --no-user-package-db field --simple-output Cabal version @(stack_I9M2eJwnG6d3686aQ2OkVk:System.Process.Read src/System/Process/Read.hs:283:3)
2016-05-20 03:22:24.503890: [debug] Resolving package entries @(stack_I9M2eJwnG6d3686aQ2OkVk:Stack.Setup src/Stack/Setup.hs:221:5)
2016-05-20 03:22:24.504386: [debug] Getting global package database location @(stack_I9M2eJwnG6d3686aQ2OkVk:Stack.GhcPkg src/Stack/GhcPkg.hs:48:5)
2016-05-20 03:22:24.504478: [debug] Run process: ghc-pkg --no-user-package-db list --global @(stack_I9M2eJwnG6d3686aQ2OkVk:System.Process.Read src/System/Process/Read.hs:283:3)
2016-05-20 03:22:24.561320: [debug] Error while decoding /home/mgsloan/fpco/stack/.stack-work/dist/x86_64-linux/Cabal-1.22.7.0/stack-build-cache: PeekException {peekExBytesFromEnd = 30381, peekExMessage = "Didn't consume all input."} (this might not be an error, when switching between stack versions) @(stack_I9M2eJwnG6d3686aQ2OkVk:Data.Store.VersionTagged src/Data/Store/VersionTagged.hs:76:18)
2016-05-20 03:22:24.701233: [warn] There were multiple candidates for the Cabal entry "Test.hs" (/home/mgsloan/fpco/stack/Test.hs), picking /home/mgsloan/fpco/stack/src/test/Test.hs @(stack_I9M2eJwnG6d3686aQ2OkVk:Stack.Package src/Stack/Package.hs:1037:5)
2016-05-20 03:22:24.724766: [debug] Error while decoding /home/mgsloan/fpco/store/.stack-work/dist/x86_64-linux/Cabal-1.22.7.0/stack-build-cache: PeekException {peekExBytesFromEnd = 5788, peekExMessage = "Didn't consume all input."} (this might not be an error, when switching between stack versions) @(stack_I9M2eJwnG6d3686aQ2OkVk:Data.Store.VersionTagged src/Data/Store/VersionTagged.hs:76:18)
2016-05-20 03:22:24.769037: [debug] Run process: ghc-pkg --global --no-user-package-db dump --expand-pkgroot @(stack_I9M2eJwnG6d3686aQ2OkVk:System.Process.Read src/System/Process/Read.hs:283:3)
2016-05-20 03:22:24.814132: [debug] Ignoring package Cabal due to wanting version 1.22.8.0 instead of 1.22.7.0 @(stack_I9M2eJwnG6d3686aQ2OkVk:Stack.Build.Installed src/Stack/Build/Installed.hs:189:5)
2016-05-20 03:22:24.814300: [debug] Ignoring package haskeline due to wanting version 0.7.2.3 instead of 0.7.2.1 @(stack_I9M2eJwnG6d3686aQ2OkVk:Stack.Build.Installed src/Stack/Build/Installed.hs:189:5)
2016-05-20 03:22:24.814378: [debug] Ignoring package terminfo due to wanting version 0.4.0.2 instead of 0.4.0.1 @(stack_I9M2eJwnG6d3686aQ2OkVk:Stack.Build.Installed src/Stack/Build/Installed.hs:189:5)
2016-05-20 03:22:24.814754: [debug] Ignoring package Cabal due to wanting version 1.22.8.0 instead of 1.22.5.0 @(stack_I9M2eJwnG6d3686aQ2OkVk:Stack.Build.Installed src/Stack/Build/Installed.hs:189:5)
2016-05-20 03:22:24.815047: [debug] Run process: ghc-pkg --user --no-user-package-db --package-db /home/mgsloan/.stack/snapshots/x86_64-linux/lts-5.15/7.10.3/pkgdb dump --expand-pkgroot @(stack_I9M2eJwnG6d3686aQ2OkVk:System.Process.Read src/System/Process/Read.hs:283:3)
2016-05-20 03:22:24.993441: [debug] Ignoring package th-orphans, from (InstalledTo Snap,"/home/mgsloan/.stack/snapshots/x86_64-linux/lts-5.15/7.10.3/pkgdb/"), due to wrong location: (Just (InstalledTo Snap),Local) @(stack_I9M2eJwnG6d3686aQ2OkVk:Stack.Build.Installed src/Stack/Build/Installed.hs:189:5)
2016-05-20 03:22:24.993615: [debug] Ignoring package th-reify-many, from (InstalledTo Snap,"/home/mgsloan/.stack/snapshots/x86_64-linux/lts-5.15/7.10.3/pkgdb/"), due to wrong location: (Just (InstalledTo Snap),Local) @(stack_I9M2eJwnG6d3686aQ2OkVk:Stack.Build.Installed src/Stack/Build/Installed.hs:189:5)
2016-05-20 03:22:24.998228: [debug] Run process: ghc-pkg --user --no-user-package-db --package-db /home/mgsloan/fpco/stack/.stack-work/install/x86_64-linux/lts-5.15/7.10.3/pkgdb dump --expand-pkgroot @(stack_I9M2eJwnG6d3686aQ2OkVk:System.Process.Read src/System/Process/Read.hs:283:3)
2016-05-20 03:22:25.067477: [debug] Trying to decode /home/mgsloan/.stack/indices/Hackage/00-index.cache @(stack_I9M2eJwnG6d3686aQ2OkVk:Data.Store.VersionTagged src/Data/Store/VersionTagged.hs:52:5)
2016-05-20 03:22:26.118330: [debug] Success decoding /home/mgsloan/.stack/indices/Hackage/00-index.cache @(stack_I9M2eJwnG6d3686aQ2OkVk:Data.Store.VersionTagged src/Data/Store/VersionTagged.hs:61:13)
2016-05-20 03:22:26.393979: [debug] Error while decoding /home/mgsloan/fpco/stack/.stack-work/install/x86_64-linux/lts-5.15/7.10.3/flag-cache/hpack-0.14.0-78bcd052466e0eb71e90498de3df0500: PeekException {peekExBytesFromEnd = 1, peekExMessage = "Attempted to read too many bytes for Char. Needed 4, but only 1 remain."} (this might not be an error, when switching between stack versions) @(stack_I9M2eJwnG6d3686aQ2OkVk:Data.Store.VersionTagged src/Data/Store/VersionTagged.hs:76:18)
2016-05-20 03:22:26.396649: [debug] Error while decoding /home/mgsloan/fpco/stack/.stack-work/install/x86_64-linux/lts-5.15/7.10.3/flag-cache/path-io-1.1.0-ce75195d16585f47735842b54d8af5b1: PeekException {peekExBytesFromEnd = 1, peekExMessage = "Attempted to read too many bytes for Char. Needed 4, but only 1 remain."} (this might not be an error, when switching between stack versions) @(stack_I9M2eJwnG6d3686aQ2OkVk:Data.Store.VersionTagged src/Data/Store/VersionTagged.hs:76:18)
2016-05-20 03:22:26.399590: [debug] Error while decoding /home/mgsloan/fpco/stack/.stack-work/install/x86_64-linux/lts-5.15/7.10.3/flag-cache/th-lift-instances-0.1.6-f0c252f6ee851126bd2e88464c35455d: PeekException {peekExBytesFromEnd = 3, peekExMessage = "Attempted to read too many bytes for Char. Needed 4, but only 3 remain."} (this might not be an error, when switching between stack versions) @(stack_I9M2eJwnG6d3686aQ2OkVk:Data.Store.VersionTagged src/Data/Store/VersionTagged.hs:76:18)
2016-05-20 03:22:26.402859: [debug] Error while decoding /home/mgsloan/fpco/stack/.stack-work/install/x86_64-linux/lts-5.15/7.10.3/flag-cache/th-reify-many-0.1.6-09dad3e1a00454f9216c8e45fbcd2618: PeekException {peekExBytesFromEnd = 1, peekExMessage = "Attempted to read too many bytes for Char. Needed 4, but only 1 remain."} (this might not be an error, when switching between stack versions) @(stack_I9M2eJwnG6d3686aQ2OkVk:Data.Store.VersionTagged src/Data/Store/VersionTagged.hs:76:18)
...

(Note the PeekExceptions)

Benchmark use of INLINABLE rather than INLINE

As mentioned in this comment, currently we have only benchmarked the differences between having various functions marked INLINE vs no annotation. It may well be worthwhile to benchmark INLINABLE and see if it's better.

In particular once there are more comprehensive benchmarks - #39 . It would be interesting to see if INLINABLE produces more compact code with similar performance for large complex ADT types. It's possible that we'd still get inlining for the smaller types. So, part of this might be looking into variation in compiled size.

Gain more clarity on when store outperforms cereal and why

We currently have benchmarks where store is much faster than cereal (5-10x), mostly for large payloads, and smaller benchmarks where performance is equivalent.

It would be good to understand what is going on -- one possibility is that fixed costs in starting to serialize/deserialize dominate the short benchmarks. Another would be that some cache/memory behavior kicks in only when the data is large enough.

Split out streaming

Streaming is a special case of serialization and not everyone wants to pull in conduits. Please pull this out as a separate package.

More comprehensive benchmarks

Some possibilities:

Serialization of complicated datatypes with TH / Generics (e.g. an AST). Since we already have instances for TH, that seems like a good candidate.
Serialization of unusual datatypes with TH / Generics (a hundred fields or a hundred constructors. both?!)
Comparisons with other libraries for these complicated datatypes
Comparisons with other libraries for architecture-independent serialization (e.g. compare encoding big endian with cereal and binary)
Extend the comparison benchmark to cbor, packer, etc.

Adequately document that Data.Store.Core is an internal module + avoid IO?

As mentioned here, it might be nice to restrict our IO to only memory accesses. I'd only be in favor of doing that if there is no impact to performance.

If not this resolution, we should document that Data.Store.Core is an internal module, and so you can use it to violate invariants (like "doesn't read files") .

`stack test` fails during compilation

I'm using Stack Version 1.1.2 (Git revision c6dac65e3174dea79df54ce6d56f3e98bc060ecc (3647 commits) x86_64 hpack-0.14.0) on Windows 7.

This is to do with unix-compat, but I don't understand anything beyond that.

Steps:

git clone --recursive https://github.com/fpco/store.git
stack test

Output:

$ stack test
store-0.2.1.0: unregistering (missing dependencies: cereal, cereal-vector, criterion, vector-binary-instances, weigh)
abstract-par-0.3.3: using precompiled package
abstract-deque-0.3: using precompiled package
ansi-wl-pprint-0.6.7.3: using precompiled package
Glob-0.7.5: download
cereal-0.5.1.0: using precompiled package
erf-2.0.0.0: using precompiled package
Glob-0.7.5: configure
ieee754-0.7.8: using precompiled package
Glob-0.7.5: build
blaze-builder-0.4.0.1: configure
blaze-builder-0.4.0.1: build
cereal-vector-0.2.0.1: configure
cereal-vector-0.2.0.1: build
monad-par-extras-0.3.3: configure
Glob-0.7.5: copy/register
monad-par-extras-0.3.3: build
optparse-applicative-0.12.1.0: using precompiled package
parallel-3.2.1.0: using precompiled package
cereal-vector-0.2.0.1: copy/register
parsec-3.1.9: configure
blaze-builder-0.4.0.1: copy/register
monad-par-extras-0.3.3: copy/register
parsec-3.1.9: build
scientific-0.3.4.6: configure
monad-par-0.3.4.7: download
hastache-0.6.1: configure
scientific-0.3.4.6: build
hastache-0.6.1: build
monad-par-0.3.4.7: configure
monad-par-0.3.4.7: build
scientific-0.3.4.6: copy/register
attoparsec-0.13.0.1: configure
monad-par-0.3.4.7: copy/register
attoparsec-0.13.0.1: build
vector-binary-instances-0.2.3.1: download
vector-binary-instances-0.2.3.1: configure
parsec-3.1.9: copy/register
vector-binary-instances-0.2.3.1: build
vector-th-unbox-0.2.1.4: download
vector-th-unbox-0.2.1.4: configure
vector-th-unbox-0.2.1.4: build
vector-binary-instances-0.2.3.1: copy/register
weigh-0.0.1: configure
vector-th-unbox-0.2.1.4: copy/register
math-functions-0.1.5.2: download
weigh-0.0.1: build
math-functions-0.1.5.2: configure
hastache-0.6.1: copy/register
math-functions-0.1.5.2: build
weigh-0.0.1: copy/register
math-functions-0.1.5.2: copy/register
attoparsec-0.13.0.1: copy/register
cassava-0.4.5.0: configure
cassava-0.4.5.0: build
aeson-0.9.0.1: configure
aeson-0.9.0.1: build
cassava-0.4.5.0: copy/register
aeson-0.9.0.1: copy/register
statistics-0.13.2.3: download
statistics-0.13.2.3: configure
statistics-0.13.2.3: build
statistics-0.13.2.3: copy/register
criterion-1.1.1.0: configure
criterion-1.1.1.0: build
criterion-1.1.1.0: copy/register
store-0.2.1.0: configure (lib + test)
store-0.2.1.0: build (lib + test)

--  While building package store-0.2.1.0 using:
      C:\Users\Eitan\AppData\Roaming\stack\setup-exe-cache\x86_64-windows\setup-Simple-Cabal-1.22.5.0-ghc-7.10.3.exe --builddir=.stack-work\dist\2672c1f3 build lib:store test:store-test test:store-weigh --ghc-options " -ddump-hi -ddump-to-file"
    Process exited with code: ExitFailure 1
    Logs have been written to: C:\cygwin64\home\Eitan\store\.stack-work\logs\store-0.2.1.0.log

    Configuring store-0.2.1.0...
    Preprocessing library store-0.2.1.0...
    [ 1 of 10] Compiling System.IO.ByteBuffer ( src\System\IO\ByteBuffer.hs, .stack-work\dist\2672c1f3\build\System\IO\ByteBuffer.o )
    [ 2 of 10] Compiling Data.Store.Impl  ( src\Data\Store\Impl.hs, .stack-work\dist\2672c1f3\build\Data\Store\Impl.o )
    [ 3 of 10] Compiling Data.Store.TH    ( src\Data\Store\TH.hs, .stack-work\dist\2672c1f3\build\Data\Store\TH.o )
    [ 4 of 10] Compiling Data.Store.TH.Internal ( src\Data\Store\TH\Internal.hs, .stack-work\dist\2672c1f3\build\Data\Store\TH\Internal.o )
    [ 5 of 10] Compiling Data.Store.Internal ( src\Data\Store\Internal.hs, .stack-work\dist\2672c1f3\build\Data\Store\Internal.o )
    [ 6 of 10] Compiling Data.Store.Version ( src\Data\Store\Version.hs, .stack-work\dist\2672c1f3\build\Data\Store\Version.o )
    [ 7 of 10] Compiling Data.Store       ( src\Data\Store.hs, .stack-work\dist\2672c1f3\build\Data\Store.o )
    [ 8 of 10] Compiling Data.Store.Streaming ( src\Data\Store\Streaming.hs, .stack-work\dist\2672c1f3\build\Data\Store\Streaming.o )
    [ 9 of 10] Compiling Data.Store.TypeHash.Internal ( src\Data\Store\TypeHash\Internal.hs, .stack-work\dist\2672c1f3\build\Data\Store\TypeHash\Internal.o )
    [10 of 10] Compiling Data.Store.TypeHash ( src\Data\Store\TypeHash.hs, .stack-work\dist\2672c1f3\build\Data\Store\TypeHash.o )
    In-place registering store-0.2.1.0...
    Preprocessing test suite 'store-test' for store-0.2.1.0...
    [1 of 5] Compiling Data.StoreSpec.TH ( test\Data\StoreSpec\TH.hs, .stack-work\dist\2672c1f3\build\store-test\store-test-tmp\Data\StoreSpec\TH.o )
    [2 of 5] Compiling System.IO.ByteBufferSpec ( test\System\IO\ByteBufferSpec.hs, .stack-work\dist\2672c1f3\build\store-test\store-test-tmp\System\IO\ByteBufferSpec.o )
    [3 of 5] Compiling Data.Store.StreamingSpec ( test\Data\Store\StreamingSpec.hs, .stack-work\dist\2672c1f3\build\store-test\store-test-tmp\Data\Store\StreamingSpec.o )
    [4 of 5] Compiling Data.StoreSpec   ( test\Data\StoreSpec.hs, .stack-work\dist\2672c1f3\build\store-test\store-test-tmp\Data\StoreSpec.o )

    C:\cygwin64\home\Eitan\store\test\Data\StoreSpec.hs:85:27:
        Not in scope: type constructor or class `CUid'
        Perhaps you meant `CPid' (imported from System.Posix.Types)
        In the Template Haskell quotation ''CUid

    C:\cygwin64\home\Eitan\store\test\Data\StoreSpec.hs:87:43:
        Not in scope: type constructor or class `CTcflag'
        In the Template Haskell quotation ''CTcflag

    C:\cygwin64\home\Eitan\store\test\Data\StoreSpec.hs:87:64:
        Not in scope: type constructor or class `CRLim'
        In the Template Haskell quotation ''CRLim

    C:\cygwin64\home\Eitan\store\test\Data\StoreSpec.hs:88:25:
        Not in scope: type constructor or class `CNlink'
        In the Template Haskell quotation ''CNlink

    C:\cygwin64\home\Eitan\store\test\Data\StoreSpec.hs:88:52:
        Not in scope: type constructor or class `CGid'
        Perhaps you meant `CPid' (imported from System.Posix.Types)
        In the Template Haskell quotation ''CGid

Make the package unbuildable on architectures which suffer from alignment issues

Just opening an issue to track a change making store buildable only on a few architectures:

    - condition: (!arch(I386) && !arch(X86_64) && !arch(IA64) && !impl(ghcjs))
      buildable: false

Magic constant for streaming frames

Currently, if input to streaming decode is invalid, it can interpret misc bytes as the length of the next chunk of data. This length can be very large and could cause progress to halt.

While there isn't anything we can do about this in general, we can catch most cases by adding a magic constant and checking it. If the constant is a Word64, then we have only a miniscule probability, 1 / 2^64, of random data causing the bad behavior.

Use unsafeDupablePerformIO in pure encode / decode functions?

See the recent conversation here. Previously the use of unsafePerformIO + INLINE was discussed, but choosing something other than unsafePerformIO didn't enter the discussion. It may be worthwhile to switch to a use of unsafeDupablePerformIO to avoid the cost of eager blackholing.

Type system information for statically known data size

It would be really cool to have compiletime info about datatype size. This has the potential to make both the generics and TH derivations more efficient. Certainly for TH it would cause the compiler to do less overall work. It'd also allow us to use functions that demand ConstSize and have them fail statically. This way the user can't accidentally make their values VarSize (can be much less efficient).

We might need to use some TH hacks to reify runtime Storable info as compiletime type info.

I'm imagining something like the following, using type nats, data kinds, and type families:

-- Can I use Nat here?  You know what I mean.
data StaticSize = KnownSize Nat | UnknownSize

class Store a where
    type StoreSize :: a -> StaticSize
    type StoreSize a = UnknownSize
    -- ...

type instance StoreSize () = 'KnownSize 0
type instance StoreSize Int32 = 'KnownSize 4
-- etc

Probably good to have it as an associated type family, so that it can be given the default of

We'd also likely want the Size a type be a GADT that enforces that ConstSize is yielded when StoreSize a ~ KnownSize n. Marking this "breaking change" as such.

Benchmark a different Poke implementation

@snoyberg and @chrisdone found that having Peek use fewer arguments and not using CPS yielded performance improvements. This change happened here: b941758

We should try a similar change for Poke. I'm imagining something like the following:

newtype Poke a = Poke
    { runPoke :: forall byte.
        Ptr byte
     -> Offset
     -> IO a
    }
    deriving Functor

Whereas this is the old definition:

newtype Poke a = Poke
    { runPoke :: forall byte r.
        Ptr byte
     -> Offset
     -> (Offset -> a -> IO r)
     -> IO r
    }
    deriving Functor

Safe support for architecture independent serialization

As described by Chris Done, here, we've gotten a lot of mileage out of using store on existing binary formats. However, that only works well when we can assume a machine architecture (endianness etc).

See #31 for ideas on how to support declaring machine independent serialization. The focus of this issue is on making it safe. Ideally, store would allow you to declare architecture independent serializations in such a way that you have static checking that your serialization is architecture independent. Here's the best scheme I can come up with for this:

Choose the behavior on x86_64 machines to be the canonical serialization behavior. So, machine representations will be used on the most common platform.
Add new class(es) and types for machine independent serialization. It just occurred to me to name these after the binary types - #35 . This makes sense, as these are good names for machine independent serialization, and will make it easier to port code from binary / cereal.

class StoreCrossArch where
    putSize  :: Size a
    put :: a -> Put ()
    get :: Get ()

newtype Put a = Put (Poke a)
newtype Get a = Get (Peek a)

(probably split up - #21)

On little endian machines with no alignment issues, StoreCrossArch should get implemented and Store defined in terms of it (on a per instance basis). This way the serialization performance should be the same on x86 whether or not you are using the machine independent interface.
On big endian machines, Store would be implemented separately from StoreCrossArch. The StoreCrossArch instances would handle endianness flipping. So, put :: Int -> Put () would use a little endian int even though we're on a big endian architecture. poke :: Int -> Poke () would use a big endian int, as that's the machine representation.

Not sure if this will be implemented in the immediate future, but it's rather fun to think about. PRs appreciated!

Improve code coverage of Data.Store.Impl

I noticed that there are some fundamental functions that are not touched by the tests. In particular, Poke f <*> Poke g, Poke x >>= f, liftIO f and pokePtr are not tested, which could potentially lead to nasty surprises. It might be prudent to write some tests to cover those.

Move Data.Store.Impl to store-core?

For package writers who prefer to hand-write their own Store instances, importing the store package is an overkill. A lot of extra dependencies aren't needed in this case.

Peek: Benchmark Ptr-Length-Offset instead of Ptr-Ptr

If the type of Peek was

type Offset = Int
type Length = Int
newtype Peek = Peek { runPeek :: forall byte. Ptr byte -> Length -> IO (Offset, a) }

then most pointer subtractions and additions could be avoided; for example:

peekStorableTy ty = Peek $ \ptr blen ->
    let needed = sizeOf (undefined :: a)
     in do
        when (needed > blen) $
            tooManyBytes needed blen ty
        x <- Storable.peek (castPtr ptr)
        return (needed, x)

peekToPlainForeignPtr ty len =
    Peek $ \sourcePtr sourceLen -> do
        when (len > sourceLen) $
            tooManyBytes len sourceLen ty
        fp <- BS.mallocByteString len
        withForeignPtr fp $ \targetPtr ->
            BS.memcpy targetPtr (castPtr sourcePtr) len
        return (len, castForeignPtr fp)

They would move into the Peek monad:

    pure x = Peek (\_ _ -> return (0, x))
    Peek x >>= f = Peek $ \ptr blen -> do
        (off, x') <- x ptr blen
        runPeek (f x') (ptr `plusPtr` off) (blen - off)

The performance of the two Peek types should be close. But I thought I should ask before setting up the benchmark, in case this approach has already been considered.

TypeHash includes package names and so gets invalidated when used with ghci

The issue is that code loaded into ghci is treated as if it comes from the main package. This can cause unexpected issues! For example, if stack ghci is used to run stack, it will invalidate the binary caches stored by the compiled version.

Another issue is that TypeHash is sensitive to changes in the TH API. For the original usecase, this was fine, but for the usecase in stack, this isn't so good. It means that stack built with ghc-7.8 will invalidate the caches stored by stack built with ghc-7.10. Rather inconvenient!

I have changes in progress addressing both of these, I'm mostly writing this to document the existence of the issue.

One nice side effect of this is that the store instances for TH AST types can get dropped from the package. They add quite a bit to the compile time.

Linear regression in benchmarks does not work well

@kantp found out that the runtime of the benchmarks does not seem to be linear with regards to how many times you execute them. This is probably due to their extremely small size (in terms of how much work they do).

performance anomaly and possible related bug

Great package!

I've coded and benchmarked two functions that decode a ByteString of simple packed encoded structures. { -O2 }

decodeMessages' :: Int -> S.ByteString -> V.Vector Trade
decodeMessages' n b = V.generate n (\i -> decodeEx (S.drop (i * tradeTotalBytes ) b ) :: Trade)

-- | Peek a sequenced unit.
decodeMessages'' :: Int -> S.ByteString -> V.Vector Trade
decodeMessages'' n  = decodeExWith (replicateM n (Data.Store.peek :: Peek Trade))

The first function is about 2-3x faster. Question 1: why? I would have guessed that the second would be faster.
Question 2: If you uncomment the line

--           ,  bench "1000'' " $ whnf functionToTime100'' 1000

You get
benchmarking instance/Store/1000''
sbug: PeekException 0 "Attempted to read too many bytes for Int. Needed 8, but only 0 remain."

My guess is that replicateM is overflowing a stack with 1000...but that seems bad and also the error message might be misleading.

Question 3: Any suggestions on making this a fast as possible with Store?
Question 4: In general if you want to encode and decode N simple structures without length field what is recommended approach?

Thanks!

{-# LANGUAGE DefaultSignatures     #-}
{-# LANGUAGE DeriveGeneric         #-}
{-# LANGUAGE FlexibleContexts      #-}
{-# LANGUAGE GADTs                 #-}
{-# LANGUAGE MultiParamTypeClasses #-}
{-# LANGUAGE OverloadedStrings     #-}
{-# LANGUAGE TypeOperators         #-}

import           Control.Monad         (join, replicateM_, (>=>))
import qualified Data.ByteString.Char8 as S
--import Data.Int (Int64)
import           Data.Monoid           ((<>))
-- import System.IO.Streams (InputStream)
import           Prelude               hiding (head)
import           System.IO
import           System.IO.Streams
import qualified System.IO.Streams     as Streams

import           Data.Store
import           Data.Vector           as V
import           GHC.Generics

import           Data.Int
import           System.TimeIt
import           Control.DeepSeq
import           Criterion.Main

benchmarks :: [Benchmark]
benchmarks =
  [
       bgroup "Store"
          [
               bench "10' " $ whnf  functionToTime10' 10
            ,  bench "100' " $ whnf functionToTime100' 100
            ,  bench "1000' " $ whnf functionToTime100' 1000
            ,  bench "10'' " $ whnf functionToTime10'' 10
            ,  bench "100'' " $ whnf functionToTime100'' 100
 --           ,  bench "1000'' " $ whnf functionToTime100'' 1000
          ]
  ]

main :: IO ()
main = --do
        defaultMain
        [
          bgroup "instance" benchmarks
        ]

-- import Control.Monad

data Trade = Trade !Int !Int !Int  deriving (Show, Generic)
instance Store Trade
instance NFData Trade

-- -- | Peek a sequenced unit.
-- peekVectorN :: Store a => Int -> Peek (Vector a)
-- peekVectorN n = replicateM n (Data.Store.peek :: Store a => Peek a)

tradeTotalBytes::Int
tradeTotalBytes = S.length $! encode (Trade 1 2 3)

generateTradeByteStringN::Int->S.ByteString
generateTradeByteStringN n = S.concat $ Prelude.map (\x -> encode (Trade x 2 3) ) [1..n]

decodeMessages' :: Int -> S.ByteString -> V.Vector Trade
decodeMessages' n b = V.generate n (\i -> decodeEx (S.drop (i * tradeTotalBytes ) b ) :: Trade)

-- | Peek a sequenced unit.
decodeMessages'' :: Int -> S.ByteString -> V.Vector Trade
decodeMessages'' n  = decodeExWith (replicateM n (Data.Store.peek :: Peek Trade))

-- | Peek a sequenced unit.
peekVectorMessage :: Int -> Peek (Vector Trade)
peekVectorMessage n = replicateM n (Data.Store.peek :: Peek Trade)

b10 = Control.DeepSeq.force  $ generateTradeByteStringN 10
b100 = Control.DeepSeq.force  $ generateTradeByteStringN 100
b1000 = Control.DeepSeq.force  $ generateTradeByteStringN 1000

functionToTime10'::Int -> Int
functionToTime10' n = V.length $ V.force $ decodeMessages' n b10

functionToTime100'::Int -> Int
functionToTime100' n = V.length $ V.force $ decodeMessages' n b100

functionToTime1000'::Int -> Int
functionToTime1000' n = V.length $ V.force $ decodeMessages' n b1000

functionToTime10''::Int -> Int
functionToTime10'' n = V.length $ V.force $ decodeMessages'' n b10

functionToTime100''::Int -> Int
functionToTime100'' n = V.length $ V.force $ decodeMessages'' n b100

functionToTime1000''::Int -> Int
functionToTime1000'' n = V.length $ V.force $ decodeMessages'' n b1000

Optimize peekSequence

#26 suggests something similar might be beneficial for peekSequence. However this is a little tricky as right now we are relying on https://hackage.haskell.org/package/mono-traversable-0.10.2/docs/Data-Sequences.html#v:replicateM . We need something like replicateM but that does folding. Could use StateT, but will it optimize better?

Also, implement peekMap and peekSet in terms of it.

Optimize enums and more via dataToTag# / tagToEnum#

See the relevant part of ghc-prim.

We could consider optimizing the serialization of sum types via dataToTag#. This way there's less overall code as the tag pokes don't get duplicated.

inaccurate version bounds in store.cabal (0.2.0.0 release)

most likely a missing upper bound

Resolving dependencies...
Failed to install store-0.2.0.0
Build log ( /tmp/extra29507647803/.cabal-sandbox/logs/store-0.2.0.0.log ):
Configuring store-0.2.0.0...
Flags chosen: small-bench=False, comparison-bench=False
Dependency array ==0.5.1.1: using array-0.5.1.1
Dependency base ==4.9.0.0: using base-4.9.0.0
Dependency base-orphans ==0.5.4: using base-orphans-0.5.4
Dependency bytestring ==0.10.8.1: using bytestring-0.10.8.1
Dependency conduit ==1.2.6.6: using conduit-1.2.6.6
Dependency containers ==0.5.7.1: using containers-0.5.7.1
Dependency cryptohash ==0.11.9: using cryptohash-0.11.9
Dependency deepseq ==1.4.2.0: using deepseq-1.4.2.0
Dependency fail ==4.9.0.0: using fail-4.9.0.0
Dependency ghc-prim ==0.5.0.0: using ghc-prim-0.5.0.0
Dependency hashable ==1.2.4.0: using hashable-1.2.4.0
Dependency hspec ==2.2.3: using hspec-2.2.3
Dependency hspec-smallcheck ==0.4.1: using hspec-smallcheck-0.4.1
Dependency integer-gmp ==1.0.0.1: using integer-gmp-1.0.0.1
Dependency lifted-base ==0.2.3.8: using lifted-base-0.2.3.8
Dependency monad-control ==1.0.1.0: using monad-control-1.0.1.0
Dependency mono-traversable ==1.0.0.1: using mono-traversable-1.0.0.1
Dependency primitive ==0.6.1.0: using primitive-0.6.1.0
Dependency resourcet ==1.1.7.4: using resourcet-1.1.7.4
Dependency safe ==0.3.9: using safe-0.3.9
Dependency semigroups ==0.18.2: using semigroups-0.18.2
Dependency smallcheck ==1.1.1: using smallcheck-1.1.1
Dependency store-core ==0.2.0.0: using store-core-0.2.0.0
Dependency syb ==0.6: using syb-0.6
Dependency template-haskell ==2.11.0.0: using template-haskell-2.11.0.0
Dependency text ==1.2.2.1: using text-1.2.2.1
Dependency th-lift ==0.7.6: using th-lift-0.7.6
Dependency th-lift-instances ==0.1.10: using th-lift-instances-0.1.10
Dependency th-orphans ==0.13.2: using th-orphans-0.13.2
Dependency th-reify-many ==0.1.6: using th-reify-many-0.1.6
Dependency th-utilities ==0.2.0.1: using th-utilities-0.2.0.1
Dependency time ==1.6.0.1: using time-1.6.0.1
Dependency transformers ==0.5.2.0: using transformers-0.5.2.0
Dependency unordered-containers ==0.2.7.1: using unordered-containers-0.2.7.1
Dependency vector ==0.11.0.0: using vector-0.11.0.0
Dependency void ==0.7.1: using void-0.7.1
Using Cabal-1.24.0.0 compiled by ghc-8.0
Using compiler: ghc-8.0.1
Using install prefix: /tmp/extra29507647803/.cabal-sandbox
Binaries installed in: /tmp/extra29507647803/.cabal-sandbox/bin
Libraries installed in:
/tmp/extra29507647803/.cabal-sandbox/lib/x86_64-linux-ghc-8.0.1/store-0.2.0.0-5RpHFU4LGH44roUx0pIug8
Private binaries installed in: /tmp/extra29507647803/.cabal-sandbox/libexec
Data files installed in:
/tmp/extra29507647803/.cabal-sandbox/share/x86_64-linux-ghc-8.0.1/store-0.2.0.0
Documentation installed in:
/tmp/extra29507647803/.cabal-sandbox/share/doc/x86_64-linux-ghc-8.0.1/store-0.2.0.0
Configuration files installed in: /tmp/extra29507647803/.cabal-sandbox/etc
Using alex version 3.1.4 found on system at: /opt/alex/3.1.4/bin/alex
Using ar found on system at: /usr/bin/ar
Using c2hs version 0.26.1 found on system at: /home/hvr/.cabal/bin/c2hs
Using cpphs version 1.19.2 found on system at: /home/hvr/.cabal/bin/cpphs
Using gcc version 5.3.1 found on system at: /usr/bin/gcc
Using ghc version 8.0.1 given by user at: /opt/ghc/8.0.1/bin/ghc
Using ghc-pkg version 8.0.1 found on system at: /opt/ghc/8.0.1/bin/ghc-pkg
No ghcjs found
No ghcjs-pkg found
No greencard found
Using haddock version 2.17.2 found on system at: /opt/ghc/8.0.1/bin/haddock
Using happy version 1.19.5 found on system at: /opt/happy/1.19.5/bin/happy
Using haskell-suite found on system at: haskell-suite-dummy-location
Using haskell-suite-pkg found on system at: haskell-suite-pkg-dummy-location
No hmake found
Using hpc version 0.67 found on system at: /opt/ghc/8.0.1/bin/hpc
Using hsc2hs version 0.68 found on system at: /opt/ghc/8.0.1/bin/hsc2hs
No hscolour found
No jhc found
Using ld found on system at: /usr/bin/ld
No lhc found
No lhc-pkg found
Using pkg-config version 0.29.1 found on system at: /usr/bin/pkg-config
Using strip version 2.26 found on system at: /usr/bin/strip
Using tar found on system at: /bin/tar
No uhc found
Component build order: library
creating dist/dist-sandbox-435e38a1/build
creating dist/dist-sandbox-435e38a1/build/autogen
Building store-0.2.0.0...
/opt/ghc/8.0.1/bin/ghc-pkg init dist/dist-sandbox-435e38a1/package.conf.inplace
Preprocessing library store-0.2.0.0...
Building library...
creating dist/dist-sandbox-435e38a1/build
/opt/ghc/8.0.1/bin/ghc --make -fbuilding-cabal-package -O -static -dynamic-too -dynosuf dyn_o -dynhisuf dyn_hi -outputdir dist/dist-sandbox-435e38a1/build -odir dist/dist-sandbox-435e38a1/build -hidir dist/dist-sandbox-435e38a1/build -stubdir dist/dist-sandbox-435e38a1/build -i -idist/dist-sandbox-435e38a1/build -isrc -idist/dist-sandbox-435e38a1/build/autogen -Idist/dist-sandbox-435e38a1/build/autogen -Idist/dist-sandbox-435e38a1/build -optP-include -optPdist/dist-sandbox-435e38a1/build/autogen/cabal_macros.h -this-unit-id store-0.2.0.0-5RpHFU4LGH44roUx0pIug8 -hide-all-packages -no-user-package-db -package-db /tmp/extra29507647803/.cabal-sandbox/x86_64-linux-ghc-8.0.1-packages.conf.d -package-db dist/dist-sandbox-435e38a1/package.conf.inplace -package-id array-0.5.1.1 -package-id base-4.9.0.0 -package-id base-orphans-0.5.4-5IQvrjd7gNP548VkOOyIq6 -package-id bytestring-0.10.8.1 -package-id conduit-1.2.6.6-H4st8iXCEueA82FD9YMKXS -package-id containers-0.5.7.1 -package-id cryptohash-0.11.9-5YfYrn05cQxCyV3jPzlQjg -package-id deepseq-1.4.2.0 -package-id fail-4.9.0.0-FGE8TmYPyfd2ZUpoMVx9sD -package-id ghc-prim-0.5.0.0 -package-id hashable-1.2.4.0-Ctl752zbguF6QanxurLOm2 -package-id hspec-2.2.3-4M9w4AygLMBLxptQkYAZlg -package-id hspec-smallcheck-0.4.1-4Y8oBgFu0gWKdVOFJs9nGZ -package-id integer-gmp-1.0.0.1 -package-id lifted-base-0.2.3.8-LSXKdE75JIl3uzD4Y2GaXO -package-id monad-control-1.0.1.0-1xoC3YihUKYHLar1SsWtYe -package-id mono-traversable-1.0.0.1-I3jr9YVvPyWvKDQ0lRn6T -package-id primitive-0.6.1.0-Ip44DqhfCp21tTUYbecwa -package-id resourcet-1.1.7.4-4e3ja0mbEa19TzW51oui6c -package-id safe-0.3.9-Hqo3JhJes6h6KTrVZw55Pb -package-id semigroups-0.18.2-Abbk7pV5dwErRVpafzN0b -package-id smallcheck-1.1.1-1P6TVEcRFpCt9hJwyUv6k -package-id store-core-0.2.0.0-JpGFrymQg7PLPkdEPDM9JZ -package-id syb-0.6-C65vWCsht6A8uLstpQIXyj -package-id template-haskell-2.11.0.0 -package-id text-1.2.2.1-9Yh8rJoh8fO2JMLWffT3Qs -package-id th-lift-0.7.6-3YgW6AFnN6SEjgDav6Czfy -package-id th-lift-instances-0.1.10-5ikYvZEf5WO6h6Tp6MOrZd -package-id th-orphans-0.13.2-8NRsqJ9JVt9CwA8ypd5FeU -package-id th-reify-many-0.1.6-dFWe01olzSD1PLLNPi6UM -package-id th-utilities-0.2.0.1-LC1xXMyt1OXBYx64D1QGjQ -package-id time-1.6.0.1 -package-id transformers-0.5.2.0 -package-id unordered-containers-0.2.7.1-5INwdG7O5Jdakf1CqKoOB -package-id vector-0.11.0.0-6uB77qGCxR6GPLxI2sqsX3 -package-id void-0.7.1-6M5lJJScsPq7tf0u22sAj1 -XHaskell2010 Data.Store Data.Store.Internal Data.Store.Streaming Data.Store.TH Data.Store.TH.Internal Data.Store.TypeHash Data.Store.TypeHash.Internal System.IO.ByteBuffer Data.Store.Impl -Wall -fwarn-tabs -fwarn-incomplete-uni-patterns -fwarn-incomplete-record-updates -O2
[1 of 9] Compiling System.IO.ByteBuffer ( src/System/IO/ByteBuffer.hs, dist/dist-sandbox-435e38a1/build/System/IO/ByteBuffer.o )
[2 of 9] Compiling Data.Store.Impl  ( src/Data/Store/Impl.hs, dist/dist-sandbox-435e38a1/build/Data/Store/Impl.o )
[3 of 9] Compiling Data.Store.TH    ( src/Data/Store/TH.hs, dist/dist-sandbox-435e38a1/build/Data/Store/TH.o )
[4 of 9] Compiling Data.Store.TH.Internal ( src/Data/Store/TH/Internal.hs, dist/dist-sandbox-435e38a1/build/Data/Store/TH/Internal.o )

src/Data/Store/TH/Internal.hs:48:1: error:
    Failed to load interface for ‘TH.ReifyDataType’
    Use -v to see a list of the files searched for.
xcabal: Error: some packages failed to install:
store-0.2.0.0 failed during the building phase. The exception was:
ExitFailure 1

Better streaming support

At the moment, for decoding streams there is an incremental function and a conduit, and for encoding there is only a conduit.

It would be better to provide a good generic streaming API, perhaps using a CPS approach as in streaming-commons, and then split off separate packages store-conduit, store-pipes, etc. That would also cut the dependencies down significantly.

Functions for peeking sequence-like types with runtime known size

For some reason, you may already know the count of your serialized [a] / Map k a / etc. We could offer variants of peekSequence / peekSet / peekMap / etc. Perhaps something like peekSequenceWithSize.

Cryptohash is deprecated in favor of cryptonite

DEPRECATED: this library is still fully functional, but please use cryptonite for new projects and convert old one to use cryptonite. This is where things are at nowadays.

Deferred serialization / deserialization holes

packer has a really cool idea where during serialization you can create known-sized holes, and later fill them. I haven't really considered the details, but it would be cool to generalize this to deserialization as well. A hole in deserialization would essentially be explicit decoding laziness. It'd probably use a data frame much like the streaming support.

Another thought on this is that for serialization it'd be good to be able to run some assertion function to check that all the holes have been filled.

Test suite does not compile

With the current master, the test suite does not compile.

It looks like it broke around 417b6df, with the changes in the TH code.

Remove testing dependencies from library dependencies

Thanks for this awesome library!

Are there any plans to remove hspec and friends from the library dependencies?

Status of GHCJS support

By using stack test --compiler ghcjs-0.2.0.20160315_ghc-7.10.2, I can actually run the test suite with ghcjs!!

I just had to comment out the instance Integer and $(mkManyHasTypeHash [ [t| Int32 |] ]) in the test. The Integer stuff needs to be re-worked to use GHCJS's representation. The TypeHash stuff requires a sha1 C function.

It immediately fails due to using unimplemented C in the streaming test.

store-0.1.0.1: test (suite: store-test)

Progress: 1/2
Data.Store.Streaming
  conduitEncode and conduitDecode
    Roundtrips ([Int]).
uncaught exception in Haskell thread: ReferenceError: h$realloc is not defined
ReferenceError: h$realloc is not defined
    at h$$gl7 (/home/mgsloan/fpco/store/.stack-work/dist/x86_64-linux/Cabal-1.22.4.0_ghcjs/build/store-test/store-test:347056:13)
    at h$runThreadSlice (/home/mgsloan/fpco/store/.stack-work/dist/x86_64-linux/Cabal-1.22.4.0_ghcjs/build/store-test/store-test:9788:11)
    at h$runThreadSliceCatch (/home/mgsloan/fpco/store/.stack-work/dist/x86_64-linux/Cabal-1.22.4.0_ghcjs/build/store-test/store-test:9742:12)
    at h$mainLoop (/home/mgsloan/fpco/store/.stack-work/dist/x86_64-linux/Cabal-1.22.4.0_ghcjs/build/store-test/store-test:9737:9)
    at /home/mgsloan/fpco/store/.stack-work/dist/x86_64-linux/Cabal-1.22.4.0_ghcjs/build/store-test/store-test:10300:13
    at /home/mgsloan/fpco/store/.stack-work/dist/x86_64-linux/Cabal-1.22.4.0_ghcjs/build/store-test/store-test:23046:17
    at FSReqWrap.strWrapper (fs.js:568:5)
thread blocked indefinitely in an ST             
store-0.1.0.1: test (suite: store-weigh)

events.js:85
      throw er; // Unhandled 'error' event
            ^
Error: spawn a.jsexe ENOENT
    at exports._errnoException (util.js:746:11)
    at Process.ChildProcess._handle.onexit (child_process.js:1054:32)
    at child_process.js:1145:20
    at process._tickCallback (node.js:355:11)

Completed 2 action(s).
Test suite failure for package store-0.1.0.1
    store-test:  exited with: ExitFailure 1
    store-weigh:  exited with: ExitFailure 1
Logs printed to console

I tried commenting out the streaming tests, and got:

store-0.1.0.1: test (suite: store-test)

Progress: 1/2
Data.Store
  Store on all monomorphic instances
    Roundtrips (GHC.Word.Word8)
    Roundtrips (GHC.Word.Word64)
    Roundtrips (GHC.Word.Word32) FAILED [1]
    Roundtrips (GHC.Word.Word16)
    Roundtrips (GHC.Types.Word) FAILED [2]
    Roundtrips (Data.Void.Void)
uncaught exception in Haskell thread: unhandled primop: CopyByteArrayToAddrOp (0,5)
thread blocked indefinitely in an ST             
store-0.1.0.1: test (suite: store-weigh)

events.js:85
      throw er; // Unhandled 'error' event
            ^
Error: spawn a.jsexe ENOENT
    at exports._errnoException (util.js:746:11)
    at Process.ChildProcess._handle.onexit (child_process.js:1054:32)
    at child_process.js:1145:20
    at process._tickCallback (node.js:355:11)

Completed 2 action(s).
Test suite failure for package store-0.1.0.1
    store-test:  exited with: ExitFailure 1
    store-weigh:  exited with: ExitFailure 1
Logs printed to console

There, it's failing on the serialization of Data.Vector.Unboxed.Base.Vector GHC.Word.Word64.

So there are at least a few missing things: CopyByteArrayToAddrOp, sha1, and realloc. Not sure if it's even possible to implement these.

We have a few choices for what to do about this:

Leave it, GHCJS users will need to be sure to test! (current approach)
CPP to remove the things not supported by GHCJS. Seems fiddly
Get the requisite FFI / primops added to GHCJS

Known issue: instances missing on haddocks in hackage

I think this would be addressed by updating hackage's version of haddock - haskell/haddock#449 .
The reason for all the orphans is a combination of the TH stage restriction and https://ghc.haskell.org/trac/ghc/ticket/1012 . It's a safe use of orphans because you can't import the Storeclass from any exposed module without getting them. Soo, bizarrely enough, the only way to fix this is merge the Data.Store.Impl and Data.Store.Internal modules into one and use the crazy "make your own global TH names" hack described by Richard in the comments.

I'm inclined to wait for the new version of haddock to be on hackage :)

AArch64 support

AArch64 is currently unsupported because of #37. However, I compiled store for this architecture and didn't encounter any problems. Tests included in the package are passing.
Could you enable building for this platform?

Safe implementation from a single spec

Continuing along the line of thought for making store safer for some uses (#36), it also makes sense to consider a mechanism for implementing custom serialization safely.

Currently, you really have to be careful that your peek / poke / size methods are implemented consistently. peek and poke need to agree upon the serialization format. You can even cause seg faults if your size function is wrong! In practice, this isn't a big deal due to a combination of generics / TH and testing.

It feels a bit un-haskell-ey to do things in such an unsafe way. How can we make it safer? I think it's possible!

JsonGrammar

Take, for example, the JsonGrammar project: https://github.com/MedeaMelana/JsonGrammar2

It leverages some fancy lensy constructions to allow you to specify json encoding and decoding with a single method definition. Thankfully, I think this variety of binary serialization can be done without quite as much fanciness. This is because we can expect a particular order of input, whereas JsonGrammar needs to allow you descend into particular fields by name.

Sum types are still quite tricky, though. JsonGrammar uses "stack prisms" to handle this. That may well be the most viable approach. However, I have a suspicion that something like dependent-sum could be sufficient to get a solution that isn't too hairy, and allows us to use case statements. Since I haven't quite figured out this bit, omitting the speculative code I have for it.

Rough sketch

For product types, I'm imagining something roughly like:

-- A serialization which can be used to implement size / peek / poke
data Serialize a
serializeSize :: Serialize a -> Size a
serializePeek :: Serialize a -> Peek a
serializePoke :: Serialize a -> a -> Poke ()

-- Applicative serialization has a hope of being 'ConstSize'
data SerializeA a
instance Applicative SerializeA
mkSerializeA :: (a -> SerializeA a) -> Serialize a
serializeA :: Store a => SerializeA a

-- Monadic serialization is always 'VarSize'
data SerializeM a
instance Monad SerializeM
mkSerializeM :: (a -> SerializeM a) -> Serialize a
serializeM :: Store a => SerializeM a

-- Example use

serializeTuple :: (Store a, Store b) => Serialize (a, b)
serializeTuple = mkSerializeA $ \(a, b) -> (,) <$> serializeA a <*> serializeA b

instance Store (a, b) where
    size = serializeSize serializeTuple
    peek = serializePeek serializeTuple
    poke = serializePoke serializeTuple

Together with Instantiators provided by th-utilities, we could have quite concise declaration of Store instances. Instead of the -- Example use above, we could just write:

$($(derive [d|
    instance StoreSafe X where
        storeSafe = mkSerializeA $ \Prod a b -> Prod <$> serialize a <*> serialize b
    |]))

Better approach to running tests on many generated values

I picked smallcheck to generate values for the tests, for the following reasons:

Easy derivation for ADTs (compared to quickcheck)
Simply conceptual model for the 'serial' instances

However, it isn't quite ideal for this, for the following reasons:

No way to limit the time or number of things tested, just an arbitrary depth
Doesn't seem to report the value used when there's an exception
We don't really use much of the fancy features. We do care about some form of exhaustiveness (using every constructor), but beyond that depth, we mostly care about not taking a long time to test
The default numeric instances focus on values near 0, and don't include minBound or maxBound.
Error cases for some instances - Bodigrim/smallcheck#34

The main thing is that there doesn't seem to be any reasonable way for me to pick a depth that works for all tests.

Bounds checks are vulnerable to overflow.

The bounds checks in the peek function are performed like this: ⁠⁠⁠⁠plusPtr startPtr len > endPtr.
This is vulnerable to overflow if len is large (which is likely to happen, for instance, when we read corrupt data).
We should change this to minusPtr endPtr startPtr < len.

Possible multiple exceptions for the same decoding error

In Data.Store.Impl:

decode = unsafePerformIO . try . decodeIO
{-# INLINE decode #-}

The reason this is done in IO is so that throwIO can be used when a decoding error occurs.

This INLINE is unsafe. If GHC actually does inline this code in multiple places, you could generate the same decode error multiple times. It should be changed to NOINLINE.

A cleaner way to structure this would be to run the entire calculation in Either and then throw any asynchronous exception in one place at the top level.

Have an option to avoid copies and instead pin the input BS memory

It'd be nice to support this without baking it directly into the Store instance for types. This is because whether you want the buffers in the input pinned or not is really a consideration of usage. Values that are the same type may vary quite a bit w.r.t. how much of them consists of shareable buffers.

This would add a bit of overhead, due to the runtime check of some pinBuffers :: Bool that gets passed around all over the place :/ . This could be avoided. If we assume deserialization all happens on one thread, then this global could be hacked on as a global pinBufferMap :: IORef (IntMap Bool) where the key is the ThreadId.

Getting more pie-in-the-sky, it could also be possible to do this intelligently by:

Figuring out the ratio of shareable buffers while encoding. (the user would ask for this via newtype AutoShare a or something)
On decoding, read out the ratio of shareable buffers and decide whether to pin or not.

Optional checks to make 'poke' safer

As documented on the size method, if size reports a different byte size than is actually used by poke, memory corruption, segfaults, and/or PokeExceptions may occur. This is freaky! It's done in the name of performance, in order to avoid checking it all the time.

One approach to this is to provide a lot of ways to safely define Store instances. The good news is that current store is pretty good for this. You just need to:

Use the well-tested instanced provided by the library
Use Generics / TH for deriving instances for your data types
Use the testing utilities on any of your custom instances (and perhaps the derived ones too)

The need for (3) could potentially be reduced by implementing #41

Potential plan for supporting processors that care about alignment

On various architectures, particularly RISC, alignment faults get thrown on unaligned access. This either causes software emulation of unaligned access (with a lot of overhead due to the interrupt) or crashes.

It's easy to find the places where we need to add alignment support - it's all the use of Storable instances, which come with alignment info. To support this, we can probably just have the encoder / decoder for these skip forward enough padding bytes to reach an aligned address. A pretty easy fix!

It relies on all of the alignment values being a divisor of the buffer's alignment. In other words, bufferOffsetmodalignment x == 0 must be true for all of the alignments used. Otherwise, we're going to have trouble with reliable encoding / decoding. While we can add some asserts for this, I don't think this assumption will be violated in practice.

This alignment stuff gets more complicated with support for machine independent serialization. See #36 . We'd want the machine independent instances to know how to use aligned accesses to simulate an unaligned access.

PowerPC Specific

Not sure how much demand there is for PowerPC support, but we probably already support it. store is currently buildable: false on PowerPC, because there are a few things with the potential to have pathologically poor performance.

In this comment, hvr points out an interesting article about devastating performance with unaligned access:

Here's a good couple paragraphs from the article:

The PowerPC takes a hybrid approach. Every PowerPC processor to date has hardware support for unaligned 32-bit integer access. While you still pay a performance penalty for unaligned access, it tends to be small.
On the other hand, modern PowerPC processors lack hardware support for unaligned 64-bit floating-point access. When asked to load an unaligned floating-point number from memory, modern PowerPC processors will throw an exception and have the operating system perform the alignment chores in software. Performing alignment in software is much slower than performing it in hardware.

So it looks like for the specific case of PPC, we could continue doing unaligned access of most things, but ensure aligned access of Double. What about Int64? It's surprising to me that this would affect Double but not Int64

Adopt compact normal forms from GHC HEAD?

Preliminary support for compact regions was added to GHC on the 15 July. It is based on the approach presented in "Efficient Communication and Collection with Compact Normal Forms", Edward Z. Yang et.al. from ICFP 2015. The top level API is in Data.Compact: https://github.com/ghc/ghc/blob/cf989ffe490c146be4ed0fd7e0c00d3ff8fe1453/libraries/compact/Data/Compact.hs

Might this store library adopt compact normal forms support at some point?

Make instances list in docs clearer

The problem is that Data.Store.Internal defines instances, but they don't get listed in the docs of Data.Store even though it imports them. Why does Data.Store.Internal have orphans? Because the code in Data.Store.TH needs to reference the typeclasses defined in the hidden Data.Store.Impl module. These modules cannot be combined, due to the TH stage restriction, keeping the Data.Store.Internal module and the Data.Store.TH module apart.

Note: these are not "real" orphans, in that you cannot access the Store type class without also getting these instances. (lets ignore the TH tricks that could allow working around this ;) )

Note: due to a bug we cannot use hs-boot files to resolve the circularity - https://ghc.haskell.org/trac/ghc/ticket/1012

Open haddock issue about it
Either:
- Fix it in haddock and upload the docs generated by the fixed version + use fixed version in stackage
- Hardcode global names into the TH code, and use them generate non-orphan instances. This workaround is described by goldfire, in that ticket.

Tests for Ratio fail.

The tests for Ratio Int8 and Ratio Int64 fail with an arithmetic overflow. This is rather strange, given that the implementations of poke, peek, and size do not seem to be performing any kind of arithmetic.

Failures:

  test/Data/StoreSpec.hs:237:
  1) Data.Store, Manually listed polymorphic store instances, Roundtrips (GHC.Real.Ratio GHC.Int.Int8)
       uncaught exception: ArithException (arithmetic overflow)

  test/Data/StoreSpec.hs:237:
  2) Data.Store, Manually listed polymorphic store instances, Roundtrips (GHC.Real.Ratio GHC.Int.Int64)
       uncaught exception: ArithException (arithmetic overflow)

Make exceptions thrown by streaming more deterministic in their content

I have no reason to believe that there are any cases where streaming fails with valid input. However, it seems like the particular details of the thrown exceptions do vary run-to-run in the test-suite. For now, the test-suite just passes these test if an exception of the correct type is thrown.

I believe this is the result of merging #49 and #50