I'm using GnuWin32 Make on Windows to try to compile this library, but every time I run make or make install VARIANT=release I get an error that says The syntax of the command is incorrect.

I haven't changed anything in the Makefile or the repository. I'd be happy to provide additional information upon request.

Thanks!

Currently the "<<" merge key for mappings is not supported. It's not part of any current YAML specification, but it's widely supported anyway.

https://yaml.org/type/merge.html

https://github.com/tlsa/libcyaml/blob/master/src/util.h#L132

The "fmt" parameter needs to be declared "const char *" if you want to compile with the "-Wwrite-strings" flag in gcc.

It seems this library is limited to loading only structured maps into specific C structs with known (static) keys and types. It would be useful to allow a map with unknown (dynamic) keys to be loaded into a runtime hash table data structure. Or at the very least provide an example of how to escape the libcyaml binding mechanism and be able to work at the libyaml level for a specific schema.

• Build system.
• Way for clients to retrieve the libcyaml version number.
• Distinguish release from development builds.

Bitfield support has been available for some time now. I have been using it and it works fine.

Can we get a new tag/release out please with the bitfield support? I want to package it and also I want to add a proper version dependency to my project. since i use the bitfield support, I would have something like dependency ('cyaml', version: '>= 0.2')

check is the standard target for running tests

https://www.gnu.org/software/make/manual/make.html#Standard-Targets

Some implicit casts to smaller types are not reported by gcc.
This could be fixed by adding the -Wconversion option.
Doing this, build generates a lot of conversion-related warnings.

Hi tlsa:

In a previous issue, I utilized the following example regarding memory allocation for dynamic arrays, which you mentioned would be messy to support.

struct sample {
uint32_t size_of_books;
struct book_t books[];
};

Is there any way we can initialize a fixed array books (using SEQUENCE_FIXED) using a size defined by the variable size_of_books. Also in this method, is it possible to handle the memory allocation based on the size defined by 'size_of_books'. Understandably, size_of_books would need to come before the list. Any help would be greatly appreciated.

Have you considered tagging the repo with versions? It would be helpful for projects that depend on this library.

there are cases that I have fixed-size arrays that always have N pointers, and if any of those is NULL cyaml fails to serialize them.

a use case is I have a mixer that always has N slots, and when the slot is supposed to be empty I simply set array[x] to NULL.

right now I am doing a workaround where I aggregate the non-NULL elements of the array into another array before serializing, and putting them back where they belong after deserializing.

I found this which says that ~ or null can be used for null values
https://stackoverflow.com/a/34089604

It seems that CYAML_FLAG_OPTIONAL is not supported for serialization (save).

I would expect that serializing an optional pointer field would mean the field is skipped when NULL. For a scalar optional field, I'd expect fields to be skipped when their value is the default, e.g. 0, false, or empty/NULL string.

What happens is the save.c code tries to dereference a NULL pointer and write its scalar value out which results in a EXC_BAD_ACCESS on Mac OS.

The error starts near save.c:1044 in cyaml__write_mapping. The field should've been skipped just like the field->value.type == CYAML_IGNORE case.

It is very slow to load large files, for example, it takes more than 3 minutes for a 30 MB file. Is this expected? Do you have any suggestion to improve the performance? This is with an x86/Linux machine. Thanks!

Yaml supports lists in its values. Is there a way to use libcyaml to parse something like this?

games:
game1:

• id: 0
  name: Player1
  data:
  • 'x', 1, 0, 9

I have union data type field inside my C structure and YAML data for the same. I am unable to use libcyaml library in this case. LIBCYAML does not have corresponding CYAML_FIELD and CYAML_VALUE for union data type in C.

iMac27:~/downloads/libcyaml-main-2 % make
cc -o build/debug/libcyaml.so.1 build/debug/shared/src/mem.o build/debug/shared/src/free.o build/debug/shared/src/load.o build/debug/shared/src/save.o build/debug/shared/src/util.o build/debug/shared/src/utf8.o -L/usr/local/lib/ -lyaml -Wl,-soname=libcyaml.so.1 -shared
ld: unknown option: -soname=libcyaml.so.1
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [build/debug/libcyaml.so.1] Error 1

Hi,

I noticed that cyaml_load_file and cyaml_free have slightly different interfaces but they could basically use the same type for data.

You could have:

cyaml_err_t cyaml_free(
		const cyaml_config_t *config,
		const cyaml_schema_value_t *schema,
		cyaml_data_t **data,
		unsigned seq_count)
{
    ...
}

and, instead of

cyaml_load_file(filename, &settings, &schema, &data, 0)
cyaml_free(&settings, &schema, data, 0);

we can use:

cyaml_load_file(filename, &settings, &schema, &data, 0)
cyaml_free(&settings, &schema, &data, 0);

Best,

related to #70

If the struct I am serializing has only one pointer inside, and that pointer is NULL it crashes (even though I set it to optional). I think the optional gets ignored if there are no other members.

If there are any other struct members it doesn't crash.

Is there any ways to define the value range for a specific struct member using a schema.
For example, in here:

Is there any ways to restrict the user to use values between 0-12 only?

Something to document the process of authoring a schema definition is needed.

I'm trying to package your library (i know it's not released yet but i want to use it in my program) on fedora/arch linux/debian and I noticed something about your makefile.

It is common in other libraries when packaging for different distros that the makefile does mkdir -p before trying to install something somewhere. (in a fakeroot environment, you have to make the /usr/lib /usr/include etc. before installing) try doing `mkdir somedir && make DESTDIR=somedir install" to see what i mean

I can submit a pull request fixing it if you are okay with that.

PS. i'm a new packager and I might be wrong, but from my experience so far, other libraries and pretty much anything that uses autotools does that

thanks for your useful work!

At the moment libyaml is used for low level YAML read/write.

The problem with it is it not actively developed. I've had a fix for it that's been sat unmerged for over a year. On the positive side, it's well used/tested and available on all the distros already.

So, I want to have a close look at libfyaml, and consider switching to it.

Currently, the logging and memory allocation callback functions do not provide a context variable.
It would be helpful to have a context in the callback to allow for easier implementation.

I'm writing extremely large files (100s of megabytes before compression) using cyaml and naturally it starts to get laggy when saving/loading large YAML files. having the ability to provide the custom strcmp function that libcyaml uses would allow me to use an optimized string interner to speed up string comparison for things that are repeated thousands of times in my YAML files like schema keys and flags

Hi, after updating to libyaml 0.2.5, this test started failing:

================================================================================
TEST: Save tests: ptr sequence with null values
================================================================================
  FAIL: test_save_sequence_null_values_int (Bad data:
EXPECTED (38):

---
- 7
- 6
- 5
- 
- 3
- 2
- 
- 0
...


GOT (36):

---
- 7
- 6
- 5
-
- 3
- 2
-
- 0
...

)

Any idea what's going on here?

Ubuntu 18.04

C++ 17 - set(CMAKE_CXX_STANDARD 17)

line 264:
CYAML_FIELD_SEQUENCE(
"depends", CYAML_FLAG_POINTER | CYAML_FLAG_OPTIONAL,
struct task, depends,
&string_ptr_schema, 0, CYAML_UNLIMITED),

error message:
error: designator order for field ‘cyaml_schema_field::data_offset’ does not match declaration order in ‘const cyaml_schema_field_t’ {aka ‘const cyaml_schema_field’}

cause:
the CYAML_FIELD_SEQUENCE declares the struct in the wrong order

fix:
edit the CYAML_FIELD_SEQUENCE (and other macros) to declare the struct members in the right order

In "plan" structure there is a pointer to structure date (start mapping). Therefore, to correctly parse it you should use CYAML_FIELD_MAPPING_PTR instead of CYAML_FIELD_MAPPING.

Currently, the resulting structure is allocated by the library, which means there is no way to know if an optional fields was loaded from the yaml file or not, as the memory is always set to zero.

When an optional field is allocated, it could be instead set to a default value which will be supplied in the field's schema.

I can save into a yaml with no errors even when a fields schema is empty (only has CYAML_FIELD_END) but it fails to load.

I temporarily removed the only thing that I had in this particular schema because I moved it somewhere else and I thought this would still work without changing the rest of the code or removing the schema completely (I plan to add things to it in the future). Is this supposed to be supported or should users never do this?

Auto-deploying Doxygen documentation to gh-pages with Travis CI

Decide how/whether to support multi-document streams.

Maybe the current API only supports single document streams, and a new one can be added that takes a client callback that is called for each new document created from the stream.

Several non-portable arguments are given to tools in the Makefile such as ar and install and $(LDFLAGS) inclusion is missing on the shared library target.

Hello,

I am trying to use libcyaml to parse OpenAPI based YAML, which has contents like this

paths:
  /pet:       <--- this can be any path!
    post:      <-- this can  be one of post/put/patch/delete/get
      summary: Add a new pet to the store
      description: Add a new pet to the store
      operationId: addPet
      responses:
        '200':
          description: Successful operation
          content:
            application/xml:
              schema:

How can I add a schema/definition to parse this ? From reading the example, I got the parsing part
when there is like name: value and name is known in advance. How do I define rules when I dont
know the word/key in advance?

Thanks.

I did a small demo project (visual studio 2019) in C to read a yml file, and it worked fine.
I tried to use the same code in C++ project (__cplusplus flag is ON). Just by adding the libcyaml files everything is OK and compiling, but when I try to create the schema structs it raises an error in the macro functions that the enum value I chose (CYAML_FLAG_DEFAULT) is actually an int and int cannot be used to initialized an entity of type "cyaml_flag". The same exact code works fine when I replace the file extension from cpp to c.

Hello,

this is libcyaml's pkg-config:

prefix=PREFIX
exec_prefix=${prefix}
libdir=${exec_prefix}/LIBDIR
includedir=${prefix}/INCLUDEDIR

Name: libcyaml
Description: Schema-based YAML parsing and serialisation
Version: VERSION
Libs: -L${libdir} -lcyaml -lyaml
Cflags: -I${includedir}

this is libyaml's pkg-config:

prefix=@prefix@
exec_prefix=@exec_prefix@
includedir=@includedir@
libdir=@libdir@

Name: LibYAML
Description: Library to parse and emit YAML
Version: @PACKAGE_VERSION@
Cflags: -I${includedir}
Libs: -L${libdir} -lyaml

As you can see, they both provide the -lyaml linker flag. This causes problems when building some programs, as they might require libyaml even when libcyaml is exclusively used (it happened to me recently on openSUSE)!

It'd be best if -lcyaml were the only linker flag provided by pkg-config.

Sorry for the many issues.

I have a sequence with a count 0 that gets serialized as:

automation_points: []

When I try to load the same string I got when saving, I get

libcyaml: DEBUG: PUSH[0]: start
libcyaml: ERROR: libyaml: control characters are not allowed
libcyaml: DEBUG: POP[0]: start

If I remove that sequence from the schema, the rest of the fields get de-serialized with no errors.

Aside from setting CC, LD, and PREFIX, wow does one properly cross-compile the library?
Is there a env variable to define the location of includes (yaml.h) and libs (libyaml.so) if they are in non-standard locations, e.g. /usr/local/arm/include and /usr/local/arm/lib

Use libyaml for the low level YAML serialisation.

Write from the client's own data structure(s) according to their schema.

I already used libcyaml in a project and I loved it? Nevertheless today we would now like to use it to parse bigger configuration files. For this we need to parse the config file in phases and not in one shot. The reason is that we have a config for the main program and multiple custom config schema attached to each plugins that are loaded dynamically after config file is parsed. As a result on lone hand we cannot known the schema of a given plugin before this one is loaded. On the other hand we do not have the list of plugin before we started to load the main config.

As a result I wonder if it is possible to load a sub YAML attached to a specific key without parsing it and retrieve this YAML-subset as a simple string. Then when a plugin is loaded we could pass the unprocessed YAML-subset to the custom plugin we just loaded to be parsed.

As example in following pseudo YAML plugin1 & 2 may not have the same schema config. I would like main to load plugin-1 config as a string and pass it to the plugin when loaded.

main:
  options:
   - opt: xxx
     value: yyy
plugins:
 - name: plugin_1.so
    config:
     (plugin 1 YAML custom schema & config)

  -name: plugin_2.so
   config:
     (plugin 2 YAML custom schema & config)

when you have a struct with only a float field inside and you set the value to 1.55331e-40f and attempt to save it, it works. it gets printed as:

---
fval: 1.55331e-40
...

when you try to load it however it rejects it:

# cyaml-MESSAGE: Load: PUSH[0]: start
# cyaml-MESSAGE: Load: Event: STREAM_START
# cyaml-MESSAGE: Load: Handle state start
# cyaml-MESSAGE: Load: PUSH[1]: in stream
# cyaml-MESSAGE: Load: Event: DOC_START
# cyaml-MESSAGE: Load: Handle state in stream
# cyaml-MESSAGE: Load: PUSH[2]: in doc
# cyaml-MESSAGE: Load: Event: MAPPING_START
# cyaml-MESSAGE: Load: Handle state in doc
# cyaml-MESSAGE: Load: Reading value of type 'MAPPING' (pointer)
# cyaml-MESSAGE: Load: Allocation: 0x5555555c1030 (0 + 4 bytes)
# cyaml-MESSAGE: Load: PUSH[3]: in mapping (key)
# cyaml-MESSAGE: Load: Event: SCALAR
# cyaml-MESSAGE: Load: Handle state in mapping (key)
# cyaml-MESSAGE: Load: [fval]
# cyaml-MESSAGE: Load: Event: SCALAR
# cyaml-MESSAGE: Load: Handle state in mapping (value
# cyaml-MESSAGE: Load: Reading value of type 'FLOAT'
# cyaml-MESSAGE: Load:   <1.55331e-40>
# cyaml-MESSAGE: Free: Top level data: 0x5555555c1030
# cyaml-MESSAGE: Free: Freeing key: fval (at offset: 0)
# cyaml-MESSAGE: Free: Freeing: 0x5555555c1030
Bail out! cyaml-FATAL-WARNING: Load: Backtrace:
Bail out! cyaml-FATAL-WARNING:   in mapping field: fval
# cyaml-MESSAGE: Load: POP[3]: in mapping (key)
# cyaml-MESSAGE: Load: POP[2]: in doc
# cyaml-MESSAGE: Load: POP[1]: in stream
# cyaml-MESSAGE: Load: POP[0]: start

cyaml error: Invalid value

It seems like it would be quite common to want to do some logic on a struct before serializing it or after deserializing it. one use case would be to fill in more information on the struct after deserializing it, another would be to "prepare" some data for serialization.

the user would register callbacks on serializable structs to be called either before cyaml starts serialization or after cyaml finished deserialization

this is kinda doable if you traverse all your structs and do it manually, but it would be nice if cyaml took care of it automatically for every struct it touches

struct test1 {
    double *value;
};

static const cyaml_schema_field_t test1_fields_schema[] = {
	CYAML_FIELD_FLOAT(
		"value", CYAML_FLAG_OPTIONAL | CYAML_FLAG_POINTER,
		struct test1, value
	),
	CYAML_FIELD_END
};

When using CYAML_FLAG_OPTIONAL | CYAML_FLAG_POINTER flags, the data type becomes double * and not double and is hence sized incorrectly. This error exists for any usage of CYAML_FLAG_POINTER flag where the field type is a pointer to a primitive value and one of the FIELD_INT or FIELD_FLOAT macros is used.

The use case here is being able to preserve optionality of the value from YAML as a nullable pointer to a primitive value.

Defining a structure with a static array and constructing the schema using CYAML_FIELD_SEQUENCE_FIXED results in incorrect data after parsing.

Consider the following structure and schema,

struct test = {
    float foo;
    int bar[10];
}

static const cyaml_schema_value_t bar_schema = {
    CYAML_VALUE_INT(CYAML_FLAG_DEFAULT, int)
};

static const cyaml_schema_field_t test_fields_schema[] = {
    CYAML_FIELD_FLOAT("foo", CYAML_FLAG_OPTIONAL, struct test, foo),
    CYAML_FIELD_SEQUENCE_FIXED("bar", CYAML_FLAG_POINTER, struct test, bar, &bar_schema, 10),
    CYAML_FIELD_END
};

static const cyaml_schema_value_t test_schema = {
    CYAML_VALUE_MAPPING(CYAML_FLAG_POINTER, struct test, test_fields_schema)
};

when tested with the following file,

foo: 123.4567
bar:
    - 0
    - 1
    - 2
    - 3
    - 4
    - 5
    - 6
    - 7
    - 8
    - 9

results in the following values in the structure test,

foo: 123.4567
bar[0]: -503343168
bar[1]: 32767
bar[2]: 0
bar[3]: 0
bar[4]: 0
bar[5]: 0
bar[6]: 0
bar[7]: 0
bar[8]: 0
bar[9]: 0

I have cloned the latest code into Ubuntu 18 LTS.
Building the code is giving me error:

make

cc -I include -DVERSION_MAJOR=0 -DVERSION_MINOR=1 -DVERSION_PATCH=0 -DVERSION_DEVEL=1   -std=c11 -Wall -Wextra -pedantic -Wconversion -O0 -g -fPIC  -c -o build/debug/shared/src/mem.o src/mem.c
cc -I include -DVERSION_MAJOR=0 -DVERSION_MINOR=1 -DVERSION_PATCH=0 -DVERSION_DEVEL=1   -std=c11 -Wall -Wextra -pedantic -Wconversion -O0 -g -fPIC  -c -o build/debug/shared/src/free.o src/free.c
cc -I include -DVERSION_MAJOR=0 -DVERSION_MINOR=1 -DVERSION_PATCH=0 -DVERSION_DEVEL=1   -std=c11 -Wall -Wextra -pedantic -Wconversion -O0 -g -fPIC  -c -o build/debug/shared/src/load.o src/load.c
src/load.c:22:10: fatal error: yaml.h: No such file or directory
 #include <yaml.h>
          ^~~~~~~~
compilation terminated.
Makefile:148: recipe for target 'build/debug/shared/src/load.o' failed
make: *** [build/debug/shared/src/load.o] Error 1

There is no such file as yaml.h in the code.

• Shared library versioning: #5
• Add install target: #46

I was wondering if you could look at this issue:

struct book_t {
uint32_t page;
};

struct sample {
struct book_t books[10];
};

This is a bit of an unusual use case, where a dynamic array is declared inside the structure, as opposed to simply declaring a pointer to a separately initialized array. I was wondering if you could add support for this particular case? I am trying to use your library with some other pre-existing code structures which cannot change. Any help in this regard would be appreciated.

I think the two functions should use the same types for consistency

Hi,

I was passing through and saw the planner.c example has June and July in the wrong order in the enum and the string to enum mapping table.

Matt

It seems that the parser crash when the reality doesn't match the schema (for instance, if one field is defined as a string in the schema but the file contains a map).

How to reproduce

Parsing the following file (bug.yaml):

global:
  path: {{ .DbnamePath | default "/tmp/index" }}

With the following code:

#include <stdio.h>
#include <stdlib.h>

#include <cyaml/cyaml.h>

struct section {
    const char* path;
};

static const cyaml_schema_field_t section_fields_schema[] = {
    CYAML_FIELD_STRING_PTR(
            "path", CYAML_FLAG_POINTER,
            struct section, path, 0, CYAML_UNLIMITED),
    CYAML_FIELD_END
};

struct conf {
    struct section* global;
};

static const cyaml_schema_field_t conf_fields_schema[] = {
    CYAML_FIELD_MAPPING_PTR(
            "global", CYAML_FLAG_POINTER,
            struct conf, global, section_fields_schema),
    CYAML_FIELD_END
};

static const cyaml_schema_value_t conf_schema = {
    CYAML_VALUE_MAPPING(CYAML_FLAG_POINTER, struct conf, conf_fields_schema),
};


static cyaml_config_t cyaml_cfg = {
    .log_level = CYAML_LOG_DEBUG,
    .log_fn = cyaml_log,
    .mem_fn = cyaml_mem,
};


int main(void)
{
    struct conf config;

    cyaml_err_t err = cyaml_load_file(
            "bug.yaml", &cyaml_cfg, &conf_schema, (void **) &config, NULL
    );

    if (err != CYAML_OK) {
        fprintf(stderr, "Error: %s", cyaml_strerror(err));
        return EXIT_FAILURE;
    }
    return EXIT_SUCCESS;
}

Produces:

% ./bug
libcyaml:   DEBUG: Load: PUSH[0]: start
libcyaml:   DEBUG: Load: Event: STREAM_START
libcyaml:   DEBUG: Load: Handle state start
libcyaml:   DEBUG: Load: PUSH[1]: in stream
libcyaml:   DEBUG: Load: Event: DOC_START
libcyaml:   DEBUG: Load: Handle state in stream
libcyaml:   DEBUG: Load: PUSH[2]: in doc
libcyaml:   DEBUG: Load: Event: MAPPING_START
libcyaml:   DEBUG: Load: Handle state in doc
libcyaml:   DEBUG: Load: Reading value of type 'MAPPING' (pointer)
libcyaml:   DEBUG: Load: Allocation: 0x555a306e7460 (0 + 8 bytes)
libcyaml:   DEBUG: Load: PUSH[3]: in mapping (key)
libcyaml:   DEBUG: Load: Event: SCALAR
libcyaml:   DEBUG: Load: Handle state in mapping (key)
libcyaml:    INFO: Load: [global]
libcyaml:   DEBUG: Load: Event: MAPPING_START
libcyaml:   DEBUG: Load: Handle state in mapping (value)
libcyaml:   DEBUG: Load: Reading value of type 'MAPPING' (pointer)
libcyaml:   DEBUG: Load: Allocation: 0x555a306e74e0 (0 + 8 bytes)
libcyaml:   DEBUG: Load: PUSH[4]: in mapping (key)
libcyaml:   DEBUG: Load: Event: SCALAR
libcyaml:   DEBUG: Load: Handle state in mapping (key)
libcyaml:    INFO: Load: [path]
libcyaml:   DEBUG: Load: Event: MAPPING_START
libcyaml:   DEBUG: Load: Handle state in mapping (value)
libcyaml:   DEBUG: Load: Reading value of type 'STRING' (pointer)
zsh: segmentation fault (core dumped)  ./bug

Backtrace

#0  0x00007fa7c5e9a6f5 in __strlen_avx2 () from /usr/lib/libc.so.6
#1  0x000056026beb115e in cyaml__data_handle_pointer (ctx=0x7ffde18b62b0, schema=0x56026beb87d8 <section_fields_schema+24>, event=0x7ffde18b62c0, value_data_io=0x7ffde18b6198) at src/load.c:1078
#2  0x000056026beb245e in cyaml__read_value (ctx=0x7ffde18b62b0, schema=0x56026beb87d8 <section_fields_schema+24>, data=0x56026d6644e0 "", event=0x7ffde18b62c0) at src/load.c:1892
#3  0x000056026beb294f in cyaml__map_value (ctx=0x7ffde18b62b0, event=0x7ffde18b62c0) at src/load.c:2129
#4  0x000056026beb2d5a in cyaml__load_event (ctx=0x7ffde18b62b0, event=0x7ffde18b62c0) at src/load.c:2317
#5  0x000056026beb2ea8 in cyaml__load (config=0x56026beb9100 <cyaml_cfg>, schema=0x56026beb88c0 <conf_schema>, data_out=0x7ffde18b6600, seq_count_out=0x0, parser=0x7ffde18b63f0) at src/load.c:2373
#6  0x000056026beb30aa in cyaml_load_file (path=0x56026beb5010 "bug.yaml", config=0x56026beb9100 <cyaml_cfg>, schema=0x56026beb88c0 <conf_schema>, data_out=0x7ffde18b6600, seq_count_out=0x0) at src/load.c:2429
#7  0x000056026beaf2d7 in main () at bug.c:44

My guess

Problems seems to come from cyaml__read_value:

            if (!cyaml__is_sequence(schema)) {
                    /* Since sequences extend their allocation for each entry,
                     * they're handled in the sequence-specific code.
                     */
                    err = cyaml__data_handle_pointer(ctx, schema, event, &data);
                    if (err != CYAML_OK) {
                            return err;
                    }
            }

The condition here only relies on the schema (with cyaml__is_sequence, here schema.type is CYAML_STRING) regardless of what's really happening (content of cyaml_event is CYAML_EVT_MAP_START) and 💥 .

Hi Tlsa:
I want to use double type in libcyaml since I want to read some double type data from my .yml. However,I only find cyml_float in cyaml.h. How can I achieve that? Thank you so much!