foliojs / restructure Goto Github PK

Hi Devon,

I couldn't find email address for you. I'm currently traveling but when I'm back to Bay Area mid August, let's definitely meet and talk font formats, etc!

Cheers

There's an issue in pdfmake which is said to be caused by restructure.
Can you please confirm whether that is fixed in the current version or not yet? Thank you.

Hello,

FIY the .parcel-cache directory got packaged in the npm packages, I think that was not expected ?

My editor automatically removes trailing whitespace on save, which might end up as commit noise for pull requests, making it difficult to spot the actual changes.

Would you be open to a pull request removing trailing whitespace across the codebase?

(please pardon the length and difficulty of this report, but it is a complicated data structure they've defined in Opentype, and it just may be beyond what restructure was defined to do, at least directly?)

I am interested in working up a parser for Opentype STAT Style Attributes Tables for use in Fontkit, and thus I am using restructure in the structure definitions.

I'm stuck on a complication with one part of the STAT specification and need suggestions, and/or the restructure code may need fixes or new features to handle this scenario:

• an array size plus pointer to an array of pointers to structures

I have everything working in one self-contained Mocha test (attached) versionSTAT01.js.txt for running within a restructure git clone, using sample data embedded in the file, but with one remaining problem.

STAT field axisValueCount has the length of the linked array. The field offsetToAxisValueOffsets points to an array. That array then contains pointers to the desired axis value tables. So there are three levels of data - initial STAT table, intermediate separate array of pointers, and separate axis value tables.

All that would seem to translate to be something like:

Pointer(uint, Array( Pointer(uint, valueStruct), 'theSize') )

But I am unable to use Pointer() twice in the same line. I am also unable to acquire the array count from the initial table to size the array. I can't find any combination of definitions that allows for describing
the situation as found in the STAT table.

.

Below I describe (at length) the things I've tried. It's a lot of description. You may want to instead first peek at the STAT table spec and play with the attached Mocha test file.

.

In the same STAT structure there is another pointer, to an array of structs, and the below code works for that:

    DesignAxisCount:    r.uint16,
    OffsetToDesignAxes:  
        new r.Pointer(r.uint32, 
                new r.Array(
                        AxisRecord, 
                        'DesignAxisCount')),

For the above "pointer to array of structs" the array count is accessed correctly and the definition for this simpler setup works great.

The problem is with a pointer that links to an array, but that array itself contains pointers to the desired final structures. The array size comes from the original struct:

    AxisValueCount:     r.uint16,
    OffsetToAxisValueOffsets: 
        new r.Pointer(r.uint32, 
                new r.Array( 
                        new r.Pointer(r.uint16, AxisValue), 
                        'AxisValueCount')),

For a "pointer to array of pointers to structs" the array count can't be picked up from the initial struct.

If I try to reference the array size using the preceding field,

    AxisValueCount:     r.uint16,
    OffsetToAxisValueOffsets: 
        new r.Pointer(r.uint32, 
                new r.Array( 
                        new r.Pointer(r.uint16, AxisValue), 
                        'AxisValueCount')
        ),

I get the error reason: "Error: Unknown version 6"

C:\Toms\Study\fonts\javascript\restructure-0.5.4\src\VersionedStruct.js:37
        throw new Error("Unknown version " + res.version);

If I try to hard-code the array count to 6, this code also fails

    OffsetToAxisValueOffsets: 
        new r.Pointer(r.uint32, 
                new r.Array( 
                        new r.Pointer(r.uint16, AxisValue), 
                        6)
        ),

with the exact same reason. From these two tests it looks like mentioning r.Pointer() twice in one line causes confusion...

If I separate the pointer and array definitions, like so:

    AxisValueArray = new r.Struct( {
      AxisValues: new r.Array( new r.Pointer(r.uint16, AxisValue), 6),
    })
          . . . . . 
    OffsetToAxisValueOffsets: 
        new r.Pointer(r.uint32, AxisValueArray),

the correct result is obtained (it works). But note I'm hard-coding the array size to 6.

If I try to reference the array size count from the original struct,

    AxisValueArray = new r.Struct( {
      AxisValues: new r.Array( new r.Pointer(r.uint16, AxisValue), 'AxisValueCount'),
    })
          . . . . . 
    OffsetToAxisValueOffsets: 
        new r.Pointer(r.uint32, AxisValueArray),

this fails with "Error: Not a fixed size"

C:\Toms\Study\fonts\javascript\restructure-0.5.4\src\utils.js:19
      throw new Error('Not a fixed size');

Continuing to research possibilities I thought to check using functions for array size:

    AxisValueArray = new r.Struct( {
      AxisValues: new r.Array( new r.Pointer(r.uint16, AxisValue), 
          function(parent){
            console.log("**D this:    %j", this)
            console.log("**D parent:  %j", parent)
            return 6
          }),
    })

as maybe I could chain upwards to find the array size value in the original first-level structure. But this displayed the astonishing:

**D this: {}
**D parent: {}
No parent?

I'm stuck. At this point it looks like extra code in two steps would be needed to handle the "pointer to array to pointers" situation. It looks like I'd have to use

    OffsetToAxisValueOffsets: r.uint32,

then figure out the correct data offset to the intermediate array, and then again call restructure to separately parse the "array of pointers to structs", stuffing that result back into the first result structure. Foo.

Is this impossible given the existing restructure feature set? Ideas?

Attached test file: versionSTAT01.js.txt

Good job. Thank you

Would it be convenient to have some sort of assertion type?

new r.Struct(
  version: new r.Assert(uint8, 3, 'Only version 3.x is supported')
)

Or should that be handled in userland for structs using the postProcess hook?

The repository, homepage and bugs links in package.json result in 404s.

Might be able to drop homepage and bugs properties, and roll with just:

"repository": "devongovett/restructure"

As npm is capable of inferring these when dealing with GitHub repositories.

Hello,

When working on network streams, it happens that binary protocol chunks are split randomly by routers or whatever is on the path between the client or the server.

I understand that restructure decoding works via DecodeStream(buffer) and that the buffer needs to match the structure boundaries (maybe my understanding is not correct)

The closest explanation/partial solution to what I mean is described on https://stackoverflow.com/questions/52267098/whats-the-fastest-way-to-parse-node-js-buffer-stream-chunks-binary-data-into-s/52333431

Is there a plan to make DecodeStream work with streams and be aware of boundary issues ?

Is there a way to optionally exclude fields depending on other values?

Record = new r.Struct({
  name: new r.String(),
  flags: r.uint32,
  age: r.uint8, // optional if flags & 0x50
  friends: new r.Array(..)
})

Does restructure have support for NULL-terminated C strings?

This example is taken from the README:

var fs = require('fs');
var r = require('restructure');

var Person = new r.Struct({
  name: r.String(r.uint8, 'utf8'),
  age: r.uint8
});

var stream = new r.EncodeStream();
stream.pipe(fs.createWriteStream('out.bin'));

Person.encode(stream, {
  name: 'Devon',
  age: 21
});

stream.end();

Running it in various io.js/Node.js versions throws:

/tmp/restructure-test/node_modules/restructure/src/Struct.js:77
        if (type.size != null) {
                ^
TypeError: Cannot read property 'size' of undefined
    at Struct.size (/tmp/restructure-test/node_modules/restructure/src/Struct.js:77:17)
    at Struct.encode (/tmp/restructure-test/node_modules/restructure/src/Struct.js:99:45)
    at Object.<anonymous> (/tmp/restructure-test/x.js:12:8)
    at Module._compile (module.js:426:26)
    at Object.Module._extensions..js (module.js:444:10)
    at Module.load (module.js:351:32)
    at Function.Module._load (module.js:306:12)
    at Function.Module.runMain (module.js:467:10)
    at startup (node.js:117:18)
    at node.js:946:3

In c++, the int is 4 bytes.
And here it starts at uint8, is there any way around it?

Am currently dealing with formats that deal exclusively with little endian and it'd be nifty exposing int16le, uint32le and the likes alongside its big endian cousins.

Thoughts?

I found 2 license-related problems:

• missing LICENSE file at top of repo
  • it should contain your copyright info
  • you've indicated MIT: https://opensource.org/licenses/MIT
• missing LICENSE file in the release tarball (under https://registry.npmjs.org/)

I have the restructure 2.0.0 package as part of my package.json file. When I attempt to run my app, I get the error below. I am creating a React app using Typescript and Ionic. How can I fix a dependency problem like this which is inside the node_modules folder itself? In package.json I have restructure set as version ^2.0.0 but I have also tried ^3.0.0 with the same results.

Uncaught ReferenceError: Buffer is not defined
at ./node_modules/restructure/src/DecodeStream.js

Actually, we should implement some kind of markup to signal that a certain vector table can optimize the structure size by having pointers pointing to the same sub-structures.

In particular, that happens in OpenType GPOS/GSUB sub-tables.

Right now I'm using r.uintx to represent booleans, but is there a boolean type I may have overlooked?

Not entirely sure about the added benefit, but it would be easier to spot booleans within structs as they would stand out.

The utf16/ucs2 support for null-terminated strings assumes a single-byte null terminator, instead of two zero bytes. This affects encode(), size(), and decode() for these string types.

Hello,

For some reason I get different params format when encoding and decoding data.

For example;

data: new r.Optional(new r.String('size', 'utf8'), (parent) => {
  console.log(parent)
  return !isBinaryType(parent.val?.type ?? parent.type)
}),

When encoding I get :

But when decoding I get :

So when I decoding I get the .val that I get in encoding

I that expected ?

Now that BigInt is getting popular, maybe we can use it for int64/uint64 support?

since dd343b0 has been merged, it would be highly appreciated if you can cut a new release

Hi, thanks for making restructure!

It looks great for our needs, but I have one small concern: Struct takes an object, and according to the spec, object iteration order is not guaranteed.

I assume nobody has run into this problem yet because implementations do in fact guarantee order.

Any suggestions for working around this? Would you be interested in a patch to make it possible to supply Struct with an array of key/value pairs instead?

Version: 1.0.0
Issue: I'm using restructure to unpack/pack objects for use in a binary protocol. Given "C-like structures" are a supported feature, I expected the "encode" function to align fields based on the spec provided.

If the spec states a string is 10 bytes long, shouldn't a 3-byte string be null-padded when encoded?

For example:

let stream = new r.EncodeStream();
stream.on('data', console.log);

let encoder = new r.Struct({
  foo: new r.Buffer(10),
  bar: new r.String(10)
});

encoder.encode(stream, {
  foo: Buffer.from('foo'),
  bar: 'bar'
});
stream.end();

Expected:

<Buffer 66 6f 6f 00 00 00 00 00 00 00>
<Buffer 62 61 72 00 00 00 00 00 00 00>

Actual:

<Buffer 66 6f 6f>
<Buffer 62 61 72>

Analysis of a stream according to frames according to #38

Because I haven't done Nodejs-only coding in a while, this somehow escaped me on first and second doc read-throughs. The lowest-level data access here is using the Nodejs Buffer APIs, such as readUint32BE() rather than the other cross-platform possibilities like DataView and Typed Arrays.

The readme needs to be updated to mention the use of Buffer APIs, and thus that additional shims like buffer might be needed when targeting browsers. I see mention of Browserify in package.json which when used would automatically add the shim, but I haven't used Browserify in ages either.

Ref to issue foliojs/pdfkit#757

This line evaluates generated code and thus cannot be used on projects relying on CSP. (without adding the unsafe-eval directive)

I'm attempting to parse a format that's very similar to the Interchange File Format which has arbitrarily sized chunks.

It seems Array has support for decoding using parent._length, which would allow decoding unknown amounts of chunks based on number of bytes. However, a struct's length seems to default to 0 unless explicitly passed into decode, is that by design?

It's sometimes useful to be able to know a relative offset of a specific field of a Struct.
Currently this works for me, but I would expect to see such API available out-of-the box ;)
Thanks!

r.Struct.prototype.offsetOf = function(field) {
  let keys = Object.keys(this.fields);
  let offset = 0;
  for (let i=0; i<keys.length; i++) {
    let key = keys[i];
    if (key == field) {
      return offset;
    }

    offset += this.fields[key].size();
  }
  return -1;
}

Running restructure in node throws ReferenceError: TextEncoder is not defined.
I think it's because TextEncoder is exported via util.TextEncoder
Currently blocking react-pdf migration to mainline fontkit repo here

First of all, great module!

I tried to do this:

var Record = new r.Struct({....});
...
...
var Packet = new r.Struct({
      number_of_records: r.uint8,
      records: new r.Array(Record, 'number_of_records')
});

var stream = new r.DecodeStream(buf);
var object = Packet.decode(stream);

var object returns this error:

TypeError: type.decode is not a function
      at Struct._parseFields (node_modules/restructure/src/Struct.js:53:22)
      at Struct.decode (node_modules/restructure/src/Struct.js:18:12)
      at StructTest.parse (main.js:79:23)
      at Context.<anonymous> (test/protocol.js:14:26)

Does this mean I can't do array of structs? :(

Set a magic byte with a fixed check of the value for the start of the packet and optionally for the end.

Example: https://github.com/keichi/binary-parser#common-options option assert