http://json-schema.org seems to be a good basis. The only difference is that ucl may define multiple values for an object's key. This should be reflected in schema as well.

See #51
Would a parser option/flag that creates explicit arrays in cases of two objects with the same key be useful?
so:

disk { device = /dev/ad0; }
disk { device = /dev/ad1; }

would be:
disk [
{ device = /dev/ad0; },
{ device = /dev/ad1; }
]

This is already what happens if the output type is JSON

It might be handy to have attributes for meta-information that attaches to AST nodes or files, e.g.:

#![version=0.1, unsafe]

#[uses_network]
stuff {
  url: foo://bar/baz/wibble
}

The following configuration:

ALIAS             :
  all-depends: query %dn-%dv
  annotations: info -A
  build-depends: info -qd
  download: fetch
  iinfo: info -i -g -x
  isearch: search -i -g -x
  leaf: query -e '%a == 0' '%n-%v'
  list: info -ql
  origin: info -qo
  provided-depends: info -qb
  raw: info -R
  required-depends: info -qr
  shared-depends: info -qB
  show: info -f -k
  size: info -sq


repos_dir : [
   /home/bapt,
   /usr/local/etc
]

Alias is silently ignored instead of failing and not available in the object when parsed.

I would have expected here either an error, or an object with alias as a key (what ever type)

A new macro, named .contents or .file or something

the args would include file=/path/to/file
and the 'data' would be the key name
so:

.contents(file=/etc/motd) "motd";

would result in:
motd = <<<EOD
my motd
goes
here
EOD;

Looking for some feedback before I wrote the code for this.

A canonical way to have an included config file override the configuration of the existing config file

example:
the pkg(8) tool in FreeBSD gets its configuration from all of the files in /etc/pkg and /usr/local/etc/pkg

/etc/pkg/FreeBSD.conf:
FreeBSD: {
url: "pkg+http://pkg.FreeBSD.org/${ABI}/latest",
mirror_type: "srv",
signature_type: "fingerprints",
fingerprints: "/usr/share/keys/pkg",
enabled: yes
}

/usr/local/etc/pkg/repos/FreeBSD.conf:
FreeBSD: { enabled: no }

Results in a final config with the FreeBSD repo disabled.

This is a relatively simple example, and seems to already work in FreeBSD, but codifying this as a feature would be useful.

Is there a way how to do something if the configuration is encountering an unknown variable?

I want to use libucl for configuring a build tool. So I would need to look, if variable $CC is not defined, that it will be defined.

Kind regards, Ingwie.

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

It would be nice to have ucl_lookup_path("parent.child.key") and ucl_lookup_path("parent.child[index]") to walk the tree and return the object if found.

While packaging libucl for a distribution, I noticed there was already a libucl packaged:

http://www.oberhumer.com/opensource/ucl/

Hello, Vsevolod!

Since I don't see your e-mail address, I decided to use DTS to post my question - sorry about that. I need your advice on what's the optimal route to proceed. Currently I'm working on dissertation, which includes empirical part. There I implement data collection and analysis framework for my research in R statistical language and environment.

I felt the need to have means of flexible configuration for my research framework implementation. Initially I thought I need something very simple and started implementing this via JSON configuration files. However, later I realized that I might need to use variables and parameters in a configuration file and maybe other more complex features to automate the research workflow. I decided not to re-invent the wheel and look around to see what's available. I found some sed scripts, but they seemed not being flexible. On the other hand, I looked at m4 pre-processor, but it looks like an overkill for my needs. Then I found your libucl, which seems like an optimal approach. However, there some issues that call for your clarification.

First of all, as I said, most of my code is in R. It's my understanding that libucl is (only?) a C library. To the best of my knowledge, there exist a (not recommended) direct interface between C and R (http://adv-r.had.co.nz/C-interface.html) and an (recommended) indirect interface between C++ and R in a form of R package (Rcpp). I have experience with neither of them. I suspect the former is pretty complex and there might be issues with type conversions between C, C++ and R. But, I might take a quick look at it to estimate the learning curve. What's your opinion on that? Secondly, is current implementation of libucl capable of handling variables that refer to JSON elements' values in the same configuration file? I'd appreciate if you could reply at your earliest convenience. I look forward to hearing from you!

Kind regards,
Aleksandr Blekh

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

Hey,
could you possibly create a tag for libucl 0.4.0? I'm keen on including it in gentoo. Thanks.

If there are any implicit arrays in files when you use includes, libucl crashes:

if you have this input file:

16.in:
.include(priority = 1) "${CURDIR}/16.inc"

section = {
value = "test";
}

overrided = {
value = "not-to-be-shown";
}
overrided = {
value2 = "implicit-array";
}

if you ignore the include statement, the output is rendered as:
section {
value = "test";
}
overrided [
{
value = "not-to-be-shown";
}
{
value2 = "implicit-array";
}
]

But with the include:
16.inc:
overrided {
key = "overrided";
}

the output is:
overrided {
key = "overrided";
}
section {
value = "test";
}

Program received signal SIGBUS, Bus error.
0x0000000800838f20 in ucl_parser_free (parser=0x801806080) at ucl_util.c:461
461 LL_FOREACH_SAFE (parser->trash_objs, tr, trtmp) {

Comments are lost when the UCL is emitted as UCL

In the case of a tool that reads a config file, applies edits, and writes the config file out again, this is a fairly large issue.

Keeping the positioning of the comments may be quite challenging

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

#0 ucl_unescape_json_string (str=0x801815160 "an array anymore", len=18446744073705443697) at ucl_util.c:267

    t = 0x801bfffff ""
    h = 0x801c00000 <Error reading address 0x801c00000: Bad address>
    uval = 17

#1 0x000000080083084a in ucl_parse_value (parser=0x8018060f0, chunk=) at ucl_parser.c:528

    p = (const unsigned char *) 0x7fffffffed7e "an array anymore\\"
    c = <value optimized out>
    obj = (ucl_object_t *) 0x80180a480
    stripped_spaces = <value optimized out>
    str_len = <value optimized out>
    need_unescape = true
    ucl_escape = false
    var_expand = false

Hi, first of all, thanks for a nice software!

Could you add LICENSE file?
This link might help for choosing a license.
http://choosealicense.com/

UCL lacks support of binary data, so adding something like this could be useful.

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

I am building libucl in Linux targeting a Win32 platform using MXE (https://github.com/mxe/mxe). This uses an internal i686-w64-mingw32 toolchain. The same issues also apply when building using Qt's built in mingw toolchain in a Windows environment. I am using the provided cmake CMakeLists.txt in both cases, which makes it very easy to compile to target Windows and generate a static library.

Here is what I'm seeing:

1. ucl_util.c fails to build out of the box because the NBBY macro is not recognized in either of these toolchains, or any mingw based toolchain I've come across. Thus, I'd propose adding something like this:

#ifdef _WIN32
#include <limits.h>
#define NBBY CHAR_BIT
#endif

2. ucl_util.c again fails to build because most Windows compatible toolchains also do not support globbing due to missing glob.h (it must be added specially to certain toolchains) so I had to comment out the dependency for glob.h and comment out related globbing code in ucl_util.c code. I'd propose adding in #ifdef's around these globbing sets so we can build on Win32 (albeit with no glob support, which is fine for most users anyway), or handle this differently if there is a better way.

You can see it in .log (see below) but make check reporting what everything is fine.

root@az-testnode1:~/libucl/tests # ../tests/test_schema < schema/enum.json
Segmentation fault (core dumped)

schema.log

running schema test suite ../tests/test_schema additionalItems.json...Fail
running schema test suite ../tests/test_schema additionalProperties.json...OK
running schema test suite ../tests/test_schema allOf.json...OK
running schema test suite ../tests/test_schema anyOf.json...OK
running schema test suite ../tests/test_schema definitions.json...Fail
running schema test suite ../tests/test_schema dependencies.json...OK
running schema test suite ../tests/test_schema enum.json...Segmentation fault (core dumped)
Fail
running schema test suite ../tests/test_schema items.json...Fail
running schema test suite ../tests/test_schema maxItems.json...OK
running schema test suite ../tests/test_schema maxLength.json...OK
running schema test suite ../tests/test_schema maxProperties.json...OK
running schema test suite ../tests/test_schema maximum.json...OK
running schema test suite ../tests/test_schema minItems.json...OK
running schema test suite ../tests/test_schema minLength.json...OK
running schema test suite ../tests/test_schema minProperties.json...OK
running schema test suite ../tests/test_schema minimum.json...OK
running schema test suite ../tests/test_schema multipleOf.json...OK
running schema test suite ../tests/test_schema not.json...OK
running schema test suite ../tests/test_schema oneOf.json...OK
running schema test suite ../tests/test_schema pattern.json...OK
running schema test suite ../tests/test_schema patternProperties.json...OK
running schema test suite ../tests/test_schema properties.json...OK
running schema test suite ../tests/test_schema ref.json...Fail
running schema test suite ../tests/test_schema refRemote.json...Fail
running schema test suite ../tests/test_schema required.json...OK
running schema test suite ../tests/test_schema type.json...OK
running schema test suite ../tests/test_schema uniqueItems.json...Fail
PASS schema.test (exit status: 0)

With the following configuration:

packagesite: http://site.1

ucl renders it in json:

{
   "packagesite": "http:"
}

ucl fails to read the following line:

packagesite: http://pkg-test.freebsd.org/pkg-test/${ABI}/latest

it makes it become:

packagesite: http://pkg-test.freebsd.org/pkg-test/${ABI

The API for ucl_parser_add_chunk is inconsistent, it takes a const unsigned char * while everything else takes a const char *. While the project is young I figured I'd report this if you feel you want to make the change.

It is just a bit odd.

On the same way we can remove elements from an array, having an API to remove element from an object would be nice

If your configuration has an implicit array (two objects with the same key), it is not possible to address the second (or third...) object with ucl_find_path

server {
disk {
device = /dev/ad0;
}
disk {
device = /dev/ad1;
}
}

ucl_find_path(obj, 'server.disk');

returns the first disk. You have to know that it is an implicit array (obj->type = UCL_OBJECt, obj->next != 0) and ucl_object_iterate(obj, &it, false) over the object to get to the latter items.

Any ideas?

It appears that pkgconfig is supported only when built using autotools. It would be nice to have pkgconfig support added for cmake builds and straight up Makefiles.

Related to #37

If my configuration is compiled from a number of source files, which might contain many blocks for the same key, a way to specify which block 'wins' when there is a conflict

Example:
/etc/pkg/FreeBSD.conf:
FreeBSD: { enabled=yes }

/usr/local/etc/pkg/repos/FreeBSD.conf:
FreeBSD: { enabled=no }

/usr/local/etc/pkg/repos/something.conf:
FreeBSD: { enabled=yes }

It currently does not seem to be clearly defined which block would win. It is likely just a side effect of the order the blocks are parsed in.

Having a way to specify the priority would be useful.

I can think of a number of ways:
a) in each include file (or configuration statement/block), a 'priority' parameter, possibly named __priority or something to ensure it doesn't conflict with the configuration namespace, or put in a comment

b) Add parameters to the include macro:
.include(pri=1, override=1, require=0) /usr/local/etc/pkg/repos/*.conf

The file with priority=1 would be parsed 'last', so its values would overwrite any previously loaded values. The default would be the MAX of whatever datatype is used to store the priority, so files without a priority are parsed first, and have the 'least' priority. Or maybe MAX - 1, and have the 'defaults' be parsed as MAX

Is there a way to get the number of elements in an array without an iteration? object.len always seems to return 1 despite there being multiple elements. For example, this input:

bundle "basic" {}
bundle "foo" {}
bundle "bar" {}

Emits this JSON;

[
    {
        "basic": {
        }
    },
    {
        "foo": {
        }
    },
    {
        "bar": {
        }
    }
]

And yet object.len of the "bundle" key is 1.

Would it be possible to create conditional includes, based on a variable, or even better, a key

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

Macros such as this example from the README:
.macro(key="()") "something";
result in a parse error

/* Initialize parser */
parser = ucl_parser_new(UCL_PARSER_KEY_LOWERCASE);

perr = ucl_parser_add_string(parser, ".include(try=true, something=\"broken\") \"somefile\";", 0);
if (perr == false) {
    if (ucl_parser_get_error(parser)) {
            fprintf(stderr, "Error: Parse Error occured: %s\n", ucl_parser_get_error(parser));
    }
}

Error: Parse Error occured: error while parsing : line: 1, column: 38 - 'macro arguments parsing error', character: '0x20'

utils/objdump has a name conflict with /usr/bin/objdump on FreeBSD, objdump is part of the GNU Development Tools.

I propose renaming it ucldump

Something like ucl_object_find_key_alias (obj, "name", "alt_name", "more_name", NULL)

Because ucl_object_set_priority does an OR operation on obj->flags, setting the priority on an object that already has a priority does not work as intended.

printf("root_obj.priority = %d\n", ucl_object_get_priority(root_obj));
ucl_object_set_priority(root_obj, 4);
printf("root_obj.priority = %d\n", ucl_object_get_priority(root_obj));
ucl_object_set_priority(root_obj, 2);
printf("root_obj.priority = %d\n", ucl_object_get_priority(root_obj));

produces:
root_obj.priority = 0
root_obj.priority = 4
root_obj.priority = 6

Hi,

This lib uses libucl as name of library files. However it has been used by many distros for UCL Compression Library from lzo upstream. Any idea on how to solve the name issue?

http://www.oberhumer.com/opensource/ucl/

Hey. I wanted to implement a .var macro. In fact, I sat down and wrote a small math algorithm that'd find the key and value for me (which works!). However, when I then try to call register_variable, I get a segfault. Here is what the debugger pointed me to:

* thread #1: tid = 0x584133, 0x0000000100006b6d basic`ucl_parser_register_variable(parser=0x00007fff5fbfdaf0, var=0x00007fff5fbfd8d0, value=0x00007fff5fbfd8c0) + 77 at ucl_parser.c:1834, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=EXC_I386_GPFLT)
    frame #0: 0x0000000100006b6d basic`ucl_parser_register_variable(parser=0x00007fff5fbfdaf0, var=0x00007fff5fbfd8d0, value=0x00007fff5fbfd8c0) + 77 at ucl_parser.c:1834
   1831 
   1832     /* Find whether a variable already exists */
   1833     LL_FOREACH (parser->variables, cur) {
-> 1834         if (strcmp (cur->var, var) == 0) {
   1835             new = cur;
   1836             break;
   1837         }
(lldb) bt
* thread #1: tid = 0x584133, 0x0000000100006b6d basic`ucl_parser_register_variable(parser=0x00007fff5fbfdaf0, var=0x00007fff5fbfd8d0, value=0x00007fff5fbfd8c0) + 77 at ucl_parser.c:1834, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=EXC_I386_GPFLT)
  * frame #0: 0x0000000100006b6d basic`ucl_parser_register_variable(parser=0x00007fff5fbfdaf0, var=0x00007fff5fbfd8d0, value=0x00007fff5fbfd8c0) + 77 at ucl_parser.c:1834
    frame #1: 0x000000010000143d basic`printer(data=0x000000010020049c, len=8, ud=0x00007fff5fbfdaf0) + 701 at basic.c:66
    frame #2: 0x0000000100007c6e basic`ucl_state_machine(parser=0x0000000100200000) + 2654 at ucl_parser.c:1754
    frame #3: 0x0000000100006f19 basic`ucl_parser_add_chunk(parser=0x0000000100200000, data=0x0000000100200330, len=374) + 361 at ucl_parser.c:1905
    frame #4: 0x00000001000017ff basic`main(argc=1, argv=0x00007fff5fbffb48) + 911 at basic.c:131

The function from which I am calling this:

bool
printer(const unsigned char *data, size_t len, void* ud) {
    struct ucl_parser* parser = (struct ucl_parser*)ud;
    char key[len];
    char value[len];
    bool now_value=false;
    int left_length;
    for(int i=0; i<len; i++) {
        if(now_value == false && data[i] != '=') {
            // no equal sign, write to key.
            key[i]=data[i];
        } else if(now_value == false && data[i] == '=') {
            // Equal sign. I.e.: key=value
            key[i]='\0';
            now_value=true;
            left_length=i+1; // length starts at 0, but its really above null.
        } else if(now_value == false && data[i] == ' ' && data[i+1] == '=') {
            // Space and equal. I.e.: key = value
            key[i]='\0';
            left_length=i+1; // length starts at 0, but its really above null.
            i++; // Advance till after the equal sign.
            now_value=true;
        } else if(now_value == true && data[i] == ' ' && data[i-1] == '=') {
            // Space after equal, skip.
            continue;
        } else if(now_value == true) {
            // Value, write to it.
            value[( len-(i+(len-i))+(left_length-(len-i)) )]=data[i];
            printf("i:%d, di:%c, formular:%d\n", i, data[i], ( len-(i+(len-i))+(left_length-(len-i)) ));
            /*printf(
                "( %d-(%d+(%d-%d))+(%d-(%d-%d)) )\n",
                len, i, len, i, left_length, len, i
            );*/
        }
    }
    // Make it NULL terminated.
    value[len]='\0';
    // print
    //printf("VAR: %s=%s\n", key, value);
    ucl_parser_register_variable(parser, key, value);
    return true;
}

key and value are properly NULL terminated, at least they should be! I see nothing that says different with the printfs. But during variable registration, I get the segfault.

Any soltution?

Currently there is only ucl_parser_get_error that returns const char* but in my Rust wrapper I want to use "raw" error codes (as there is enum ucl_error that is used nowhere) to allow user to "manually" provide suitable error message.

This is just a feature request. It would be really great for validation reasons if we could extract the line/col for objects. Thoughts?

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

% cat test2.ucl
section foo bar {
  key = value
}
section foo baz {
  key = value
}
section foo {
  bar = lol /* removing this line makes parsing successful */
}
% ./objdump test2.ucl
Error occured: error on line 10 at column 3: 'invalid character in a key', character: '0xff'

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

From DevSummit:

merge=error|replace|merge|append

I wish to do something like this...how can I actually do it?

        onBuild: [
            <<PHP
                Phar::createFromDir(
                    "${in}",
                    "${out}"
                );
            PHP
        ]

I am just using PHP as the delimiter, nothing else.

Kind regards, Ingwie

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

if you call ucl_object_delete_key(parent, keyname);

every object after that key disappears. The 'len' on the parent is correct, but the other objects disappear.

example:
with the ucl object: { one: 1, two: 2, three: 3, four: 4, five: 5 }

$ ./del_repro
one = 1
two = 2
three = 3
four = 4
five = 5

but if you delete "three"
$ ./del_repro three
one = 1
two = 2

then "four" and "five" disappear

Reproduction code:

include <stdio.h>

include <errno.h>

include "ucl.h"

int
main(int argc, char *argv[])
{
ucl_object_t *root_obj = NULL;
struct ucl_parser *parser = NULL;
ucl_object_iter_t it;
const ucl_object_t *cur;
bool perr = false, success = false;

/* Initialize parser */
parser = ucl_parser_new(UCL_PARSER_KEY_LOWERCASE);

perr = ucl_parser_add_string(parser, "{ one: 1, two: 2, three: 3, four: 4, five: 5 }", 0);
if (perr == false) {
exit(1);
} else {
root_obj = ucl_parser_get_object(parser);
}

if (ucl_parser_get_error(parser)) {
fprintf(stderr, "Error: Parse Error occured: %s\n",
    ucl_parser_get_error(parser));
exit(2);
}

if (argc > 1) {
success = ucl_object_delete_key(root_obj, argv[1]);
if (success == false) {
    fprintf(stderr, "failed to delete key\n");
    exit(3);
}
}

it = ucl_object_iterate_new (root_obj);
while ((cur = ucl_object_iterate_safe (it, true)) != NULL) {
printf("%s = %ld\n", ucl_object_key(cur), ucl_object_toint(cur));
}
ucl_object_iterate_free (it);

if (parser != NULL) {
ucl_parser_free(parser);
}
if (root_obj != NULL) {
ucl_object_unref(root_obj);
}

return(0);

}

Would it be possible to have automatic conversion between arrays and strings

Not sure how a user might denote when they want this to happen but, example:

flags = "a,b,c";
becomes:
"flags": [
'a',
'b',
'c',
]

and vice versa

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

Hello there!

I have to say first, that this is one very great configuration library. I really like the ngix like syntax that you choose to make usable. And, this project is highly embedable as well!

But, the C-API is a little bit confusing to me still - I cant seem to find out how to iterate over an array yet? Doxygen refused to build a nice documentation too as well, so I am stuck with the test_*.c files only.

But, will there ever be a c++ wrapper? I could imagine it quite simple actually.

UCLObject *obj = new UCLObject(buffer);
UCLObject *srcs = obj->get("sources");
for(int i=0; i<srcs->length; i++) {
    cout << srcs->get(i)->valueAsString() << endl;
}

In my case, I am about ot build a meta build system using UCL. Why? The other meta build systems require a scripting language to be preinstalled. And not everyone has Python or pearl preinstalled, especially not on windows... o.o

Kind regards!

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

kib/KiB and so on

I'd like to request a feature to add a parser flag to disable the automatic time parsing. It overlaps with Go's style of time parsing and I'd rather use Go's. Unfortunately, if I pass in the string, it automatically becomes a time!

Hi,

I'm trying to build libucl using Visual Studio 2012, and there are numerous errors (even if I try to compile as C++ code).
Have anyboy used libucl with Visual Studio?

Thanks!

Hi! I've been getting some strange segfaults in my go-libucl library, and I outputted ref counts and everything looks good. It seems like when the parent gets freed, all children are automatically freed.

Instead, shouldn't it just unref the children?

add an emitter that will export in UCL format but string entries with newline will be emitted as a preformatted block

bla: <<EOD
blabla
EOD

For specifying defaults

defaults {
key = "val"
foo = "bar"
many = "values here"
}

mything {
.inherit "defaults"
key = "newval"
}

results in:
mything {
foo = "bar"
many = "values here"
key = "newval"
}

If you try to parse this JSON, it doesn't work:

{"foo": "bar"}

But if you remove the prefixed whitespace, it works. I don't think the prefixed whitespace actually is valid JSON, but I think this should work.