pmunch / binaryparse Goto Github PK

I have a file which consists of a lot of entries, each of them has 4 int32 fields (so file content is like this: i32 i32 i32 i32 i32 i32 i32 i32 - in this example there's two entries), but there's no thing like "size" in the file.

Of course I can easily read this file with streams module, but maybe it would be good for binaryparse to have that functionality. Maybe it can be possible to implement "loop" section or something like that.

when reading in uint16s it appear to be reading them in backwards as all of the numbers I am getting seem to be bit shifted 8 to the left instead of where they should be. Maybe I am doing something wrong. Here is the code I am testing it on.

import streams
import binaryparse

createParser(las):
    s4:sig = "LASF"
    u16:id
    u16:encoding
    u32:data1
    u16:data2
    u16:data3
    s8:data4
    s1:major
    s1:minor
    s32:sysID
    s32:software
    u16:daycreated
    u16:yearcreated
    u16:headersize





var strm = newFileStream("NEONDSSampleLiDARPointCloud.las", fmread)

https://www.asprs.org/wp-content/uploads/2010/12/LAS_1_4_r13.pdf is the specifications for parsing.
and the dataset I am testing it on can be found here https://figshare.com/articles/dataset/NEON_Teaching_Data_LiDAR_Point_Cloud_las_Data/4307750

when I try manually parsing the code I am able to get correct numbers but the numbers I am getting from the generated parser are wrong. for ID the value should be 101 but i am getting 25856 instead. I can share the manual parser if that is needed. Thank you

basic test: (in x86 arch)

import binaryparse, streams

createParser(simple):
  lu32: size

var t: typeGetter(simple)
t.size = 1
let s = newStringStream("")
simple.put(s, t)
echo repr s.data
s.setPosition 0
echo simple.get(s)

got:

0000000000960060"\0\0\0\1"
(size: 16777216)

When creating a custom parser with more parameters than just the stream, binaryparse expects an encoder signature with only 2 parameters - the stream and the input. If you try to add more parameters it doesn't work. For example:

import streams
import binaryparse

proc parseCustom(stream: Stream; extra: int): tuple[a: int] =    
  result = (1,)

proc encodeCustom(stream: Stream; input: var tuple[a: int], extra: int) =    
  discard

let custom = (get: parseCustom, put: encodeCustom)

createParser(x):
  *custom(1): y

This errors with:

test.nim(12, 13) template/generic instantiation of `createParser` from here
binaryparse.nim(358, 15) Error: type mismatch: got <Stream, tuple[a: int]>
but expected one of:
proc (stream: Stream, input: var tuple[a: int], extra: int){.noSideEffect, gcsafe, locks: 0.}

This is a proposal for extending binaryparse's syntax to support more features

#13, https://github.com/sealmove/binaryparse

To make my proposal more appealing, here is a realistic example of how parsing could look like.

Hello, I'm trying a simple exmple with the following code:

import streams, binaryparse

var strm = newFileStream("demodado.tem",fmRead)

createParser(simple):
  u16: field1[2]

echo simple.get(strm)

When introducing any count on field1 the following error occurs:

\binaryparse-0.2.2\binaryparse.nim(476, 22) Error: internal error: environment misses: :tmp

Omitting the "[2]" the compilation runs fine.

I'm comparing the use of binaryparse and using streams directly.
Why does the following code does not result in the same output?

import streams, binaryparse

var strm = newFileStream("demodado.tem",fmRead)
strm.setPosition(0)

var datehour : array[7,uint16]

for i in 0..6:
 datehour[i] = strm.readUint16()

echo datehour

strm.setPosition(0)
createParser(simple):
  u16:field1[7]

echo simple.get(strm)
strm.close()

The echo results are:

[20, 2, 1990, 21, 44, 52, 22]
(field1: @[5120, 512, 50695, 5376, 11264, 13312, 5632])

I have the following in python:

import struct

a = 0x95006c08 
print( a & 0x3FF )           # First: 10 bits  --> 8
print( (a >> 10) & 0x1FFF )  # Second: 13 bits --> 27

And I am trying to do the equialent with binaryparse:

import binaryparse
import streams

createParser(p):
  10: a
  13: b 

var 
  a = 0x95006C08 
  s = newStringStream()

s.write(a)
s.setPosition(0)

var tmp = p.get(s)
echo tmp.a  # --> 33 (instead of 8)
echo tmp.b  # --> 2688 (instead of 27)

What am I doing wrong? I am looking to replicate python's results.

By the way, binaryparse is AWESOME. I love it.

ran into it while trying this library for today's aoc:

import binaryparse, streams

block:
  let test1 = newStringStream("\xD2\xFE\x28")

  createParser(packet):
    u3: version
    u3: typeId
  
  var data = packet.get(test1)
  dump data

outputs:

data = (version: 6, typeId: 7)

but typeId should be 4.

For comparison binarylang has the correct output:

import binarylang

block:
  let test1 = newStringBitStream("\xD2\xFE\x28")

  struct(packet):
    u3: version
    u3: typeId

  var data = packet.get(test1)
  # data is an object without a $ proc defined
  echo data.version
  echo data.typeId

outputs:

6
4

it will only read the first byte, since peek won't move the position...

Is there a way of using a socket connection directly as a stream?
I only see examples of StringStream and FileStream but don't know how to get data from a socket