- 'C like structures' - TypeScript library
- Read/make/write buffer, Buffer <=> Object/Array
- Pack / unpack, compress / decompress data to minimize size.
- Convert buffer of struct/array to TypeScript object/array and back
- Able to pin start point from any offset in the buffer (read/write)
- Can return whole buffer from your data Object/Array (make)
- Little endian - LE
- Big endian - BE
- TypeScript's decorators for classes and properties
- Added predefined types
- Added predefined aliases
- Fixed issue in one function where we use endian BE -> LE
- Added more tests to cover that issue and predefined types
- Added support for wstring (UTF-16LE) type. Thanks to Sorunome.
For wstring/utf16le trailing character is/must be 16bit zero '\u0000'.
npm i @mrhiden/cstruct
Atom | Type | Size [B] | Aliases | Notes |
---|---|---|---|---|
b8 | boolean | 1 | bool8 bool BOOL | |
b16 | boolean | 2 | bool16 | |
b32 | boolean | 4 | bool32 | |
b64 | boolean | 8 | bool64 | |
u8 | unsigned char | 1 | uint8 uint8_t BYTE | |
u16 | unsigned int | 2 | uint16 uint16_t WORD | |
u32 | unsigned long | 4 | uint32 uint32_t DWORD | |
u64 | unsigned long long | 8 | uint64 uint64_t LWORD QWORD | |
i8 | signed char | 1 | int8 int8_t SINT | |
i16 | signed int | 2 | int16 int16_t INT | |
i32 | signed long | 4 | int32 int32_t DINT | |
i64 | signed long long | 8 | int64 int64_t LINT QINT | |
f | float | 4 | float float32 float32_t single REAL F F32 | |
d | double | 4 | double float64 float64_t LREAL D F64 | |
sN | string | N | string | N= 0+ |
wsN | wstring (UTF-16LE) | N * 2 | wstring WS WSTR WSTRING | N= 0+ |
bufN | buffer | N | buffer BUF BUFFER | N= 1+ |
jN | json | N | json any J JSON ANY | N= 0+ |
The main concept is to first create a model of your data structure and then utilize it to read from and write to a buffer.
- To achieve this, you can precompile the model and optional types when creating a CStructBE or CStructLE object.
- After this step, you can use the object to efficiently access the buffer and perform read/write operations on your data.
For exchanging data between JavaScript and C/C++ you can use the following classes and their methods:
- CStructBE - Big Endian
- CStructLE - Little Endian
Both classes uses the same methods and have the same functionality.
Dynamic methods uses Object/Array/String model/types to exchange data.
Static methods are designed to use decorators to define model/types.
MAKE
When using make
method it returns { buffer, offset, size }
object.
make(struct: T): CStructWriteResult;
WRITE
When using write
method it returns { buffer, offset, size }
object.
write(buffer: Buffer, struct: T, offset?: number): CStructWriteResult;
Write is different from make because it uses existing buffer and writes data to it.
Also write allows to pass offset
to pin start point from any offset in the buffer.
READ
When using read
method it returns { struct, offset, size }
object.
read<T>(buffer: Buffer, offset?: number): CStructReadResult<T>;
Read uses existing buffer and reads data from it.
And also read allows to pass offset
to pin start point from any offset in the buffer.
OFFSET
Offset can be used in different scenario as we want to read/write from/to buffer from any offset.
Which allows binary parser to split data into different parts and read/write them separately.
DECORATORS
Decorators are used to define model/types for static methods.
@CStructClass
- defines model/types for class.
@CStructProperty
- defines type for property.
Static make
creates new instance of provided class and fills it with parsed data.
NOTE
When using @CStructClass
decorator with {model: ... }
it can override @CStructProperty
decorators.
Atomic types instead pure string types
const {CStructBE,CStructLE, hexToBuffer, AtomTypes} = require('@mrhiden/cstruct');
const {U16, I16, STRING} = AtomTypes; // You can also use 'u16', 'i16', 'string' / 's' as before
// JavaScript Example 1
{
// Model with two fields.
const model = {a: U16, b: I16}; // = {a: 'u16', b: 'i16'};
// Create CStruct from model. Precompile.
const cStruct = CStructBE.fromModelTypes(model);
// Data to transfer. Make buffer from data. Transfer buffer.
const data = {a: 10, b: -10};
// Buffer made.
const buffer = cStruct.make(data).buffer;
console.log(buffer.toString('hex'));
// 000afff6
// {a: 000a, b: fff6}
// Read buffer. Receive data.
const result = cStruct.read(buffer);
console.log(result.struct);
// { a: 10, b: -10 }
}
// JavaScript Example 2
{
// Sensor type. ID and value.
const types = {
Sensor: {id: U16, value: I16}, // Sensor: {id: 'u16', value: 'i16'},
}
// Model with IOT name and two sensors.
const model = {
iotName: STRING(0), // iotName: 's0',
sensors: 'Sensor[2]',
};
// Create CStruct from model and types. Precompile.
const cStruct = CStructBE.fromModelTypes(model, types);
// Data to transfer. Make buffer from data. Transfer buffer.
const data = {
iotName: 'iot-1',
sensors: [
{id: 1, value: -10},
{id: 2, value: -20}
]
};
// Buffer made.
const buffer = cStruct.make(data).buffer;
console.log(buffer.toString('hex'));
// 696f742d31000001fff60002ffec
// '696f742d31'00 [{0001, fff6}, {0002, ffec}]
// Read buffer. Receive data.
const result = cStruct.read(buffer);
console.log(result.struct);
// { iotName: 'iot-1', sensors: [ { id: 1, value: -10 }, { id: 2, value: -20 } ] }
}
Make example
import { CStructBE, CStructProperty } from '@mrhiden/cstruct';
class MyClass {
@CStructProperty({type: 'u8'})
public propertyA: number;
@CStructProperty({type: 'i8'})
public propertyB: number;
}
const myClass = new MyClass();
myClass.propertyA = 10;
myClass.propertyB = -10;
const bufferMake = CStructBE.make(myClass).buffer;
console.log(bufferMake.toString('hex'));
// 0af6
// 0a f6
Read example
import { CStructBE, CStructProperty } from '@mrhiden/cstruct';
class MyClass {
@CStructProperty({type: 'u8'})
public propertyA: number;
@CStructProperty({type: 'i8'})
public propertyB: number;
}
const buffer = Buffer.from('0af6', 'hex');
const myClass = CStructBE.read(MyClass, buffer).struct;
console.log(myClass);
// MyClass { propertyA: 10, propertyB: -10 }
console.log(myClass instanceof MyClass);
// true
CStructClass example
import { CStructBE, CStructClass } from '@mrhiden/cstruct';
@CStructClass({
model: {
propertyA: 'u8',
propertyB: 'i8',
}
})
class MyClass {
public propertyA: number;
public propertyB: number;
}
const buffer = Buffer.from('0af6', 'hex');
const myClass = CStructBE.read(MyClass, buffer).struct;
console.log(myClass);
// MyClass { propertyA: 10, propertyB: -10 }
console.log(myClass instanceof MyClass);
// true
Read example with offset, and model and types described as string in CStructClass decorator
import { CStructBE, CStructClass } from '@mrhiden/cstruct';
@CStructClass({
model: `{propertyA: U8, propertyB: I8}`,
types: '{U8: uint8, I8: int8}',
})
class MyClass {
public propertyA: number;
public propertyB: number;
}
const buffer = Buffer.from('77770af6', 'hex');
const myClass = CStructBE.read(MyClass, buffer, 2).struct;
console.log(myClass);
// MyClass { propertyA: 10, propertyB: -10 }
console.log(myClass instanceof MyClass);
// true
Off course types: '{U8: uint8, I8: int8}'
shows only some idea how to use types.
Types can be much more complex.
import { CStructBE } from '@mrhiden/cstruct';
// Make BE buffer from struct based on model
const model = { a: 'u16', b: 'i16' };
const cStruct = CStructBE.fromModelTypes(model);
const data = { a: 10, b: -10 };
const buffer = cStruct.make(data).buffer;
console.log(buffer.toString('hex'));
// 000afff6
// 000a fff6
import { CStructBE } from '@mrhiden/cstruct';
// Make BE buffer from struct based on model
const cStruct = CStructBE.fromModelTypes({ error: {code: 'u16', message: 's20'} });
const { buffer, offset, size } = cStruct.make({ error: { code: 10, message: 'xyz' } });
console.log(buffer.toString('hex'));
// 000a78797a0000000000000000000000000000000000
console.log(offset);
// 22
console.log(size);
// 22
import { CStructBE } from '@mrhiden/cstruct';
// Read BE buffer to struct based on model
const buffer = hexToBuffer('000F 6162630000_0000000000_0000000000');
console.log(buffer.toString('hex'));
// 000f616263000000000000000000000000
const cStruct = CStructBE.fromModelTypes({ error: {code: 'u16', message: 's20'} });
const struct = cStruct.read(buffer).struct;
console.log(struct);
// { error: { code: 15, message: 'abc' } }
import { CStructBE } from '@mrhiden/cstruct';
// Read BE buffer to struct based on model
const buffer = hexToBuffer('000F 6162630000_0000000000_0000000000');
console.log(buffer.toString('hex'));
// 000f616263000000000000000000000000
const cStruct = CStructBE.fromModelTypes({ error: {code: 'u16', message: 's20'} });
const { struct, offset, size } = cStruct.read(buffer);
console.log(struct);
// { error: { code: 15, message: 'abc' } }
console.log(offset);
// 17
console.log(size);
// 17
import { CStructBE } from '@mrhiden/cstruct';
class MyClass {
public propertyA: number;
public propertyB: number;
}
const myClass = new MyClass();
myClass.propertyA = 10;
myClass.propertyB = -10;
const cStruct = CStructBE.from({
model: `{propertyA: U16, propertyB: I16}`,
types: '{U16: uint16, I16: int16}',
});
const make = cStruct.make(myClass);
console.log(make.buffer.toString('hex'));
// 000afff6
// 000a fff6
import { CStructBE } from '@mrhiden/cstruct';
@CStructClass({
model: `{propertyA: U16, propertyB: I16}`,
types: '{U16: uint16, I16: int16}',
})
class MyClass {
public propertyA: number;
public propertyB: number;
}
const myClass = new MyClass();
myClass.propertyA = 10;
myClass.propertyB = -10;
const cStruct = CStructBE.from(MyClass);
const make = cStruct.make(myClass);
console.log(make.buffer.toString('hex'));
// 000afff6
// 000a fff6
import { CStructBE } from '@mrhiden/cstruct';
// Model and Types for Sender & Receiver
const types = {
Sensor: {
id: 'u32',
type: 'u8',
value: 'd',
timestamp: 'u64',
}
};
const iotModel = {
iotName: 's20',
sensor: 'Sensor',
};
// IOT Sender
const sender = CStructBE.fromModelTypes(iotModel, types);
const senderData = {
iotName: 'IOT-1',
sensor: {
id: 123456789,
type: 0x01,
value: 123.456,
timestamp: 1677277903685n,
}
};
const { buffer: senderFrame } = sender.make(senderData);
// Transmitting frame
console.log(senderFrame.toString('hex'));
// IOT Receiver
const receiver = CStructBE.fromModelTypes(iotModel, types);
const { struct: receiverData } = receiver.read(senderFrame);
console.log(receiverData);
// {
// iotName: 'IOT-1',
// sensor: {
// id: 123456789,
// type: 1,
// value: 123.456,
// timestamp: 1677277903685
// }
// }
import { CStructBE } from '@mrhiden/cstruct';
// Make buffer from struct based on model and types
const cStruct = CStructBE.fromModelTypes(`{errors: [Error, Error]}`, `{Error: {code: u16, message: s10}}`);
const {buffer, offset, size} = cStruct.make({
errors: [
{code: 0x12, message: 'message1'},
{code: 0x34, message: 'message2'},
]
});
console.log(buffer.toString('hex'));
// 00126d65737361676531000000346d657373616765320000
console.log(offset);
// 24
console.log(size);
// 24
import { CStructBE } from '@mrhiden/cstruct';
// Mixed approach for model and types
const cStruct = CStructBE.fromModelTypes({errors: `[Error, Error]`}, {Error: `{code: u16, message: s10}`});
const {buffer, offset, size} = cStruct.make({
errors: [
{code: 0x12, message: 'message1'},
{code: 0x34, message: 'message2'},
]
});
console.log(buffer.toString('hex'));
// 00126d65737361676531000000346d657373616765320000
console.log(offset);
// 24
console.log(size);
// 24
import { CStructBE } from '@mrhiden/cstruct';
// C-kind fields {u8 a,b;} into {a:u8,b:u8}
const model = `{u8 a,b;}`;
const cStruct = CStructBE.fromModelTypes(model);
const makeStruct = { a: 1, b: 2 };
const { buffer: structBuffer } = cStruct.make(
makeStruct
);
console.log(structBuffer.toString('hex'));
// 0102
const { struct: readStruct } = cStruct.read(structBuffer);
console.log(readStruct);
// { a: 1, b: 2 }
import { CStructBE } from '@mrhiden/cstruct';
// Dynamic (length) array with types
const model = {
ab: "Ab[i16]",
};
const types = {
Ab: {a: 'i8', b: 'i8'}
};
const cStruct = CStructBE.fromModelTypes(model, types);
console.log(cStruct.modelClone);
// { 'ab.i16': { a: 'i8', b: 'i8' } }
const data = {
ab: [
{a: '-1', b: '+1'},
{a: '-2', b: '+2'},
]
};
const {buffer} = cStruct.make(data);
console.log(buffer.toString('hex'));
// 0002ff01fe02
const {struct: extractedData} = cStruct.read(buffer);
console.log(extractedData);
// { ab: [ { a: -1, b: 1 }, { a: -2, b: 2 } ] }
import { CStructBE } from '@mrhiden/cstruct';
// Dynamic (length) string
const model = {
txt1: "s[i16]",
txt2: "string[i16]",
};
const cStruct = CStructBE.fromModelTypes(model);
console.log(cStruct.modelClone);
// { 'txt1.i16': 's', 'txt2.i16': 's' }
const data = {
txt1: "ABCDE",
txt2: "AB"
};
const {buffer} = cStruct.make(data);
console.log(buffer.toString('hex'));
// 0005414243444500024142
// 0005_4142434445 0002_4142
const {struct: extractedData} = cStruct.read(buffer);
console.log(extractedData);
// { txt1: 'ABCDE', txt2: 'AB' }
import { CStructBE } from '@mrhiden/cstruct';
const model = {b: 'BYTE', w: 'WORD', f: 'BOOL'};
const cStruct = CStructBE.fromModelTypes(model);
console.log(cStruct.modelClone);
// { b: 'BYTE', w: 'WORD', f: 'BOOL' }
const struct = {b: 0x12, w: 0x3456, f: true};
const {buffer} = cStruct.make(struct);
console.log(buffer.toString('hex'));
// 12345601
// 12 3456 01
const {struct: extractedData} = cStruct.read(buffer);
console.log(extractedData);
// { b: 18, w: 13398, f: true }
// { b: 0x12, w: 0x3456, f: true }
import { CStructBE } from '@mrhiden/cstruct';
// C struct types
const model = {
xyzs: "Xyx[2]",
};
const types = `{
typedef struct {
uint8_t x;
uint8_t y;
uint8_t z;
} Xyx;
}`;
const cStruct = CStructBE.fromModelTypes(model, types);
const data = {
xyzs: [
{x: 1, y: 2, z: 3},
{x: 4, y: 5, z: 6},
]
};
const {buffer: makeBuffer} = cStruct.make(data);
console.log(makeBuffer.toString('hex'));
// 010203040506
const {struct: readStruct} = cStruct.read(makeBuffer);
console.log(readStruct);
// { xyzs: [ { x: 1, y: 2, z: 3 }, { x: 4, y: 5, z: 6 } ] }
import { CStructBE } from '@mrhiden/cstruct';
// C struct types
const types = `{
// 1st approach
typedef struct {
u8 x,y,z;
} Xyz;
// 2nd approach
struct Ab {
i8 x,y;
};
// As you noticed, comments are allowed
}`;
const model = `{
ab: Ab,
xyz: Xyz,
// As you noticed, comments are allowed
}`;
const cStruct = CStructBE.fromModelTypes(model, types);
const data = {
ab: { x: -2, y: -1 },
xyz: { x: 0, y: 1, z: 2 }
};
const {buffer: makeBuffer} = cStruct.make(data);
console.log(makeBuffer.toString('hex'));
// feff000102
const {struct: readStruct} = cStruct.read(makeBuffer);
console.log(readStruct);
// { ab: { x: -2, y: -1 }, xyz: { x: 0, y: 1, z: 2 } }
import { CStructBE } from '@mrhiden/cstruct';
// Value, static array, dynamic array
const model = `[
i8, // 1 byte
i8[2], // 2 bytes static array
i8[i16] // i16 bytes dynamic array
]`;
const cStruct = CStructBE.fromModelTypes(model);
console.log(cStruct.jsonModel);
// ["i8","i8.2","i8.i16"]
console.log(cStruct.modelClone);
// [ 'i8', 'i8.2', 'i8.i16' ]
const data = [
0x01,
[0x02, 0x03],
[0x04, 0x05, 0x06, 0x07],
];
const {buffer} = cStruct.make(data);
console.log(buffer.toString('hex'));
// 010203000404050607
// 01 02_03 0004_04_05_06_07
const {struct: extractedData} = cStruct.read(buffer);
console.log(extractedData);
// [ 1, [ 2, 3 ], [ 4, 5, 6, 7 ] ]
import { CStructBE } from '@mrhiden/cstruct';
class Undecorated {
a: number;
b: number;
}
const undecorated = new Undecorated();
undecorated.a = -1;
undecorated.b = -2;
const undecoratedStruct = CStructBE.from({
model: '{a:float,b:double}',
});
const undecoratedBuffer = undecoratedStruct.make(undecorated).buffer;
console.log(undecoratedBuffer.toString('hex'));
// bf800000c000000000000000
// bf800000 c000000000000000
TypeScript's decorators to serialize/deserialize class object to/from binary
NOTE Take a look on '/examples/decorators.ts'
MUST enable in tsconfig.json
or jsconfig.json
{
"compilerOptions": {
"experimentalDecorators": true
}
}
import { CStructBE, CStructClass, CStructModelProperty } from '@mrhiden/cstruct';
// Decorators - Serialize any class with CStructClass and CStructModelProperty decorator, model and type
class MyClass {
public a: number;
public b: number;
}
@CStructClass({
types: {MyClass: {a: 'u16', b: 'i16'}}
})
class MyData {
@CStructModelProperty('MyClass')
public myClass: MyClass;
}
const myData = new MyData();
myData.myClass = new MyClass();
myData.myClass.a = 10;
myData.myClass.b = -10;
// MAKE
const bufferMake = CStructBE.make(myData).buffer;
console.log(bufferMake.toString('hex'));
// 000afff6
// 000a fff6
// READ
const myDataRead = new MyData();
myDataRead.myClass = new MyClass();
CStructBE.read(myDataRead, bufferMake);
console.log(myDataRead);
// MyData { myClass: MyClass { a: 10, b: -10 } }
// WRITE
const bufferWrite = Buffer.alloc(4);
CStructBE.write(myData, bufferWrite);
console.log(bufferWrite.toString('hex'));
// 000afff6
// 000a fff6
import { CStructBE } from '@mrhiden/cstruct';
import * as fs from "fs";
interface GeoAltitude {
lat: number;
long: number;
alt: number;
}
@CStructClass({
types: `{ GeoAltitude: { lat:double, long:double, alt:double }}`
})
class GeoAltitudesFile {
@CStructProperty({type: 'string30'})
public fileName: string = 'GeoAltitudesFile v1.0';
@CStructProperty({type: 'GeoAltitude[i32]'})
public geoAltitudes: GeoAltitude[] = [];
}
(async () => {
// Make random data
const geoAltitudesFile = new GeoAltitudesFile();
for (let i = 0; i < 1e6; i++) {
let randomLat = Math.random() * (90 - -90) + -90;
let randomLong = Math.random() * (180 - -180) + -180;
let randomAlt = 6.4e6 * Math.random() * (8e3 - -4e3) + -4e3;
const geo = {lat: randomLat, long: randomLong, alt: randomAlt};
geoAltitudesFile.geoAltitudes.push(geo);
}
console.log('Write data length,', geoAltitudesFile.geoAltitudes.length);
// Make buffer
console.time('make');
const writeFile = CStructBE.make(geoAltitudesFile).buffer;
console.timeEnd('make');
// Write to file
console.log('Write file length,', writeFile.length);
await fs.promises.writeFile('geoAltitudesFile.bin', writeFile);
// Read from file
const readFile = await fs.promises.readFile('geoAltitudesFile.bin');
console.log('Read file length,', readFile.length);
// Read data
console.time('read');
const readGeoAltitudesFile = CStructBE.read(GeoAltitudesFile, readFile).struct;
console.timeEnd('read');
console.log('Read fileName,', readGeoAltitudesFile.fileName);
console.log('Read data length,', readGeoAltitudesFile.geoAltitudes.length);
})();
import { CStructBE } from '@mrhiden/cstruct';
@CStructClass({
model: {
any1: 'j[i8]',
any2: 'json[i8]',
any3: 'any[i8]',
}
})
class MyClass {
any1: any;
any2: any;
any3: any;
}
const myClass = new MyClass();
myClass.any1 = {a: 1};
myClass.any2 = {b: "B"};
myClass.any3 = [1, 3, 5];
const buffer = CStructBE.make(myClass).buffer;
console.log(buffer.toString('hex'));
// 077b2261223a317d097b2262223a2242227d075b312c332c355d
// 07_7b2261223a317d 09_7b2262223a2242227d 07_5b312c332c355d
const myClass2 = CStructBE.read(MyClass, buffer).struct;
console.log(myClass2);
// MyClass {
// any1: { a: 1 },
// any2: { b: 'B' },
// any3: [ 1, 3, 5 ]
// }
Trailing zero support for "string", "wstring" and "json" types ("s", "string", "ws", "wstring", "j", "json", "any")
This library has support for trailing zero for "string" and "json" types.
When you use it "json" and "string" will be written as full unknown length and '\0' will be added at the end.
That ending zero helps to read data from binary file without knowing the length of the string / json.
We can not use that trick with buffer as it may contain zeros at any place.
For wstring/utf16le trailing character is/must be 16bit zero '\u0000'.
import { CStructBE } from '@mrhiden/cstruct';
const model = {
any1: 'j[0]', // or 'json[0]' or 'any[0]'
any2: 's[0]', // or 'string[0]'
};
const cStruct = CStructBE.fromModelTypes(model);
const data = {
any1: [1, 2, 3],
any2: 'abc',
};
const buffer = cStruct.make(data).buffer;
console.log(buffer.toString('hex'));
// 5b312c322c335d0061626300
// 5b_31_2c_32_2c_33_5d_00 616263_00
// [ 1 , 2 , 3 ] \0 a b c \0
const extractedData = cStruct.read(buffer).struct;
console.log(extractedData);
// { any1: [ 1, 2, 3 ], any2: 'abc' }
console.log(JSON.stringify(data) === JSON.stringify(extractedData));
// true
If you have any questions or suggestions, please contact me at
[email protected]