jkcfg / jk Goto Github PK

(Either in #48 or once it's merged)

Add an optional format argument to read, which parses either YAML or JSON files (or decides based on their extension?), or supplies the contents verbatim.

The return value will therefore either be a string with the raw content, or a value parsed from the bytes supplied by the runtime.

This kind of enum values always ends up badly, remove it! The real behaviour I want is to have the file extension decide of the output format by default (.yaml -> produce yaml). Instead of an explicit default format value, we could have this behaviour when format is not specified.

• building in docker image on linux, /README.md
• building stdlib, on OS X, or from from scratch on linux /build/README.md

• For consistency with what people seems to use most: eg, airbnb js style guide, kubernetes documentation, ...
• Still provide an option to change the indentation with #12
• Write the test for this.

jk is easy at its core: executing a script produces an output (1 or more files). It should be easy to write e2e tests taking an input script and checking if the output matches what we expect.

When we change something, it may be nice to be able to regenerate the expected results in one go and check the diff matches the modification we just made.

This could also be used if expected results contain line numbers, say to test that exceptions in std.js are readable:

$ cat test-std-exception.js 
import std from 'std';

std.read(undefined);

$ jk run test-std-exception.js 
std:2340
    while (i < s.length) {
                 ^
TypeError: Cannot read property 'length' of undefined
    at flatbuffers.Builder.createString (std:2340:18)
    at Object.read (std:2948:29)
    at test-std-exception.js:3:5

Using gometalinter, caching it in the build container.

• Types like objects or enums should be capitalised.
• Format is shorter than outputFormat!
• We should also Object.freeze the thing.

The test is in tests/test-issue-0071.js

$ jk run test-issue-0071.js 
*** Error in `jk': double free or corruption (out): 0x00007fee98000a20 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7feea100e7e5]
/lib/x86_64-linux-gnu/libc.so.6(+0x8037a)[0x7feea101737a]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x4c)[0x7feea101b53c]
jk(_ZN2v88internal20ArrayBufferCollector15FreeAllocationsEv+0x8e)[0xf9269e]
jk(_ZN2v88internal20ArrayBufferCollector11FreeingTask11RunInternalEv+0x146)[0xf92946]
jk(_ZN2v88platform12WorkerThread3RunEv+0x36)[0x9cdf16]
jk[0xe05880]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7feea1c096ba]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7feea109e41d]

It's worth trying to be as hermetic as possible ie. not allow reading random files on the system reading /etc/resolv.conf would make the script produce two different output on two different machines.

A way to prevent that would be to prevent std.read() from reaching anywhere but the subtree the script is located.

This could be an option of the read implementation, specified through ExecuteOptions.

We should probably use an existing style guide, say airbnb.

There's definitely argument for both sides of the fence:

• relative to the current directory
• relative to the script directory

I expect that, for the configuration case, the script will want to source (script-)local assets rather than files in the current directory.

We don't want jk scripts to be able to load arbitrary files on the system. That would allow to break hermeticity a bit too easily.

I was thinking that a std.params object could hold the script input parameters.

• We could specify those parameters at run time with:
  • --params $file1 --params $file2 where $file1 and $file2 are json files with one objects each. Those object wil be merged (with something like Object.assign) before being exposed as std.params
  • Individual objects paramers could be specified on top of this eg. --number-param foo.bar=1.2

Note that we don't strictly need this mechanism. A script should be able to read JSON files with std.read, merge objects with Object.assign, ...

That said, being able to control some aspect of the script execution as run-time sounds like it could be useful. Maybe waiting until having to solve real problems with this would give us better insight whether it's needed or not.

At the moment we have:

std.write(value, "foo.yaml", std.outputFromat.YAML)

I think we should re-factor all potential parameters after the path in a options object:

std.write(value, "foo.yaml", {
  format: std.outputFromat.YAML,
  indent: "    ",
})

When running a jk program, there's often nothing written out on stdout.

We should have a way to tell the user something has happened but we don't want to pollute stdout by default, so this can be under a -v option.

There are a few things to think about for our module resolving:

1. Should the import below look into the module directory for foo.js?

import foo from 'foo';

That's what we do today. And it's not valid considering v8 will not resolve the same module specifier twice: https://github.com/ry/v8worker2/blob/master/binding.cc#L266

2. Probably more, but brain needs a pause from thinking about modules

We should find a way to publish some API docs, somewhere.

We could:

• Detect the user wants to run a .ts file
• Use https://www.npmjs.com/package/typescript to transpile it
• Run it!

At the moment we only have .expected files, but I suspect will grow .cmd for custom jk invocation, need to be able to compare trees of output files, etc.

• With a malformed js file:

$ jk run nginx.js 
Error: nginx.js:7
sd.log(deployment, { format: std.Format.YAML });
^
ReferenceError: sd is not defined
    at nginx.js:7:1

Usage:
  jk run [flags]

Flags:
  -h, --help                      help for run
  -o, --output-directory string   where to output generated files

nginx.js:7
sd.log(deployment, { format: std.Format.YAML });
^
ReferenceError: sd is not defined
    at nginx.js:7:1

• Add test!

There's a bit of work to bootstrap CI, eg. we need a container with v8worker compiled. Probably a good thing to do it sooner rather than later.

One we can generate binaries for OS X, we should think about being to use homebrew as well!

It may be nice to restricts parameters to the top level to avoid the "glog" effect. Having libraries defines all sorts of parameters without the user having a say in it.

It also has a more functional feel to it: parameters are supplied at the top level and trickle down.

Library could still expose object describing parameters and top level script expose them if it's what the user wants.

A basic web site that would:

• explain what the fuck this about
• getting started instructions
• download links
• give a couple of pertinent examples

Currently the result from deferreds is a flatbuffers string, i.e., UTF8. But we may want to have arbitrary binary data, and treat it as an ArrayBuffer (e.g., if we're doing any kind of checksumming or digests, we want the bytes).

We may want to reconsider using a default export for stdlib: https://basarat.gitbooks.io/typescript/docs/tips/defaultIsBad.html

jk run ./foo/bar.js with bar.js importing ./baz.js

We could generate XML from a JS object with https://en.wikipedia.org/wiki/JsonML

• Maybe introduce a small toy kubernetes library that can at least produce a full deployment/service/serviceaccount/...
• A nice example could be memcached+prom exporter
• 2nd example, a bit different, maybe travis file, docker file

Flatbuffers are constructed by:

• Allocating an ArrayBuffer of a certain size, by default 1kb. That buffer can grow on demand if the table serialised requires it (which of course means alloc+copy)
• The flatbuffer is built by serialising fields into that ArrayBuffer. This is actually done from the end of the array backwards. So, for instance, WriteArgs serialised to a 256 bytes ArrayBuffer looks like:

00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000030  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000040  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000050  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000060  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000070  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000080  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000090  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000000a0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000000b0  00 00 00 00 00 00 00 00  00 00 00 00 10 00 00 00  |................|
000000c0  00 00 0a 00 10 00 08 00  0c 00 07 00 0a 00 00 00  |................|
000000d0  00 00 00 02 08 00 00 00  14 00 00 00 08 00 00 00  |................|
000000e0  66 6f 6f 2e 79 61 6d 6c  00 00 00 00 0d 00 00 00  |foo.yaml........|
000000f0  7b 22 66 6f 6f 22 3a 22  62 61 72 22 7d 00 00 00  |{"foo":"bar"}...|

• The actual flatbuffer is at the end of the allocation:

00000000  10 00 00 00 00 00 0a 00  10 00 08 00 0c 00 07 00  |................|
00000010  0a 00 00 00 00 00 00 02  08 00 00 00 14 00 00 00  |................|
00000020  08 00 00 00 66 6f 6f 2e  79 61 6d 6c 00 00 00 00  |....foo.yaml....|
00000030  0d 00 00 00 7b 22 66 6f  6f 22 3a 22 62 61 72 22  |....{"foo":"bar"|
00000040  7d 00 00 00                                       |}...|

• The problem is that the js <-> go interface being an ArrayBuffer, the underlying allocation, not a view on it. That's needed for memory management. So we can't pass the result of flatbuffers' asUint8Array()
• If the whole array buffer is then passed to the go side, we lose the information about the position where the flatbuffer starts inside the bigger ArrayBuffer and we can't decode the flatbuffer
• As a (temporary!) solution, I've added a asArrayBuffer API to flatbuffers that slice (copy!) the flatbuffer into a new ArrayBuffer of exactly the right size. That's that one we give to the go side.

To remove that copy we could either:

• Have a way to compute the flatbuffers size upfront. That's complicated though because it requires quite a bit of knowledge about flatbuffers internals and fields being optional would complicate that logic even more.
• Enhance V8Worker.send() by giving (start, end) offsets to indicate the interesting part inside that array buffer. The go part could then slice the underlying buffer to the interesting part while still owning the full underlying ArrayBuffer and free it.

We could bindmount the source repo in docker and compile things with the build dependencies from the docker build image,

When an exception is being raised from the js part of the std library, it references the one line of the bundled & minimised bit of code our sdt lib is compiled down to.

More generally, we should support source maps, whether it's to decode std library exceptions or external libraries that have been minified.

We should be able to configure the indentation when write yaml/json files.

std.write(value, "foo.yaml", { indent: "     " }); // 5 spaces, because.

It's possible to use npm to publish ES2015 modules, and it'd be convenient for jk to enable it as a means of getting libraries. This would require a module resolution mechanism that reflects node's -- though without the possibility of loading non-ES2015 modules.

It may be useful for libraries built on top of jk, say to be able to create a ConfigMap object from directory ful lof files.

• do we want that?
• look at how node does it

I forgot the filesystem std functions in the std typescript shim module. We should add them!

The shim is located in std/shim.

When unit testing modules (e.g., mixins), we're using the node runtime. Inevitably, we'll want to unit test things that depend on the builtins, i.e., the std module. One solution is to provide a shim package that can be installed as a devDependency; node's module resolution can then pick it up.

When using typescript, we need to:

• export a .d.ts file with the signature of all stdlib functions.
• have an easy way for people to consume that typings file

The easiest way for 2/ is to publish this .d.ts. file in an npm module. For that we need to put the package in our scope (@jkcfg). People will then be able to add the package to their devDependencies and have the typescript compiler find the std typings automatically.