terralang / std Goto Github PK

I'll list a few things roughly in the order of how much I use them in my code. I've only written about 7K of Terra code total so far, so someone with more LOC written would probably have a better list and a different order, so FWIW:

• the ternary operator macro: but what should we name it? the operator itself would be ideal (i.e. overloading ':' in the language when the first operand is a bool)... lacking that, I would prefer something as short as possible. I personally use iif (from sql, basic), but cond could work too (form lisp). Other suggestions?

• print() and assert() -- my default debugging tools, use them all the time (tostring comes as a bonus, basically as a typed version of snprintf).

• clock() -- monotonic clock, indispensable for profiling and benchmarking -- basically terralib.currenttimeinseconds but for the Terra runtime and implement it with more accurate implementations (clock_gettime(CLOCK_MONOTONIC) for Linux, mach_absolute_time for OSX)

• typed versions of malloc, calloc, memset and memmove -- those could mimic ffi.new(), ffi.copy() and ffi.fill() semantics, so I would call them new(), copy() and fill() respectivelly. I would also add alloc() for malloc, since new() calls calloc which you may or may not want. I didn't test this but it would be interesting to know if Terra actually generates LLVM intrinsics for memmove and memset, that could be useful.

• inc() and dec() macros -- to ease the pain of not having ++ and --

These are all in my low module, which begs the question -- what should we name this module? :)

We need to figure out a good way to consume the stdlib in a separate project. This affects how the repository is structured.

I like the repository structure of having at the top level separate src/ docs/ and tests/ directories. It is nice and clean. However, it doesn't lend itself very well to inclusion in other projects.

I suspect that the easiest way to handle this is to have an init.t file at the top level to be found by require 'std' and then to have docs/ and tests/ be subfolders in the root and having additional subfolders for each module underneath root. For example, There might be a coro/ directory with a coro/init.t file which can be required by require 'std.coro' This does, however, mean that we can't have modules named docs or tests in any terra project following that project template.

In this case, we can just use a git submodule to include a version of the library, or have lit fetch dependencies based on a package manifest file.

I think there needs to be a "tier-0" lib that all other libs would depend upon. This is the lib that defines a common (preferably flat) vocabulary of the most basic tools. Unfortunately, getting two people to agree upon what exactly those tools are and what names to give them is usually very hard. OTOH, I find that not having this vocabulary makes writing any code very tedious as one has to reinvent all these little wheels all the time.

Here's my own "tier-0" stdlib: https://github.com/luapower/low/blob/master/low.t -- it contains symbols for both Lua and Terra runtimes (some names even overlap, like assert() and print()). We could maybe use it as a starting point to decide what we want to include? I have one for Lua too, just for comparison: https://luapower.com/glue

Possible stuff that comes to mind as tire-1 libs:

• a fast hash map (with a benchmark; mine: https://github.com/luapower/khash)

• a dynamic array type (mine: https://github.com/luapower/dynarray) -- here we could really benefit from an __index metamethod so we can index them with a[i] instead of a(i).

• a file type for i/o (using CreateFile on Windows instead of fopen so we can actually open users' files)

• a perfect hash function generator for O(1) lookups on static maps (mine: https://github.com/luapower/phf) -- no more clunky external tools to generate C code for that.

• a PRNG (mine is a port of LuaJIT's: https://github.com/luapower/random, twice as fast than calling math.random from Terra)

I think we should pick a naming convention. I don't want to end up with randomly mixed naming conventions. The alllowercase naming convention used by most of the terra public API is consistent, but it isn't exactly the easiest to read. the mix of underscore_case into the private code in terra as well as in a few sections of the public API isn't very appealing either. We should pick one for this and stick to it. I'm a bit partial to using camelCase or underscore_case, possibly with PascalCase for structs with camelCase for variables and functions. Does anyone else have a preference or suggestion?

I have a request for the standard library that I'm creating an issue for so that it's easier to remember.

Please avoid use of LuaJIT-specific features, particularly the FFI. Wherever possible, use features native to Terra, and if Terra can't do something, let's try to figure out if Terra can be extended so that we can have what we want.

Rationale: For various reasons, it may be necessary to move from LuaJIT to PUC Lua (i.e. the original Lua implementation). The most urgent reason is compatibility: LuaJIT supports a relatively small set of OS and architecture combinations. Notably, PPC64 is missing from this list---and if you want to use the current number 1 supercomputer in the world, it runs on PPC64. In general PUC Lua is much more portable so it less likely to run into issues in the future.

There are some promising developments on this front in raptorjit/raptorjit#199, but who knows how long this will take to mature and/or whether it ever will.

So for the moment, I'd prefer to develop in a way that won't preclude a switch to PUC Lua if we decide we need to go that route.

What testing system should we use for terra-stdlib? Busted is a nice and well developed framework for using with Lua, and is applicable to Terra-stdlib testing requirements, though it would be slightly more difficult to use since it couldn't directly interact with terra variables in the middle of terra functions. I'm also looking at the possibility of writing a new testing system designed like a cross of Spock and Busted to be able to interact with both Terra and Lua directly, and anything that Terra can abstract across through the C-ABI with ease. That is a bit further off though.

Should I just start writing tests in Busted for now and work on other things? or prioritize getting a testing platform with native support for Terra before proceding to other implementations?

Should Terra implementation of mathematical operations be included in the stdlib?

For example, quicksand has an automatic differentiation library that looks generally useful. Should that be part of the stdlib?

What about Geometric Algebra Multivectors, an FFT, Linear Algebra operators, etc.?

Basically, should any fairly general and reusable mathematical routine that isn't extremely niche be here, should there be a supplementary math library, or should they be left in separate projects?

Do we want a separate repo for discussion or do we use this one?

Here's two basic libs that we could use as a starting point for a discussion:

https://github.com/luapower/dynarray
https://github.com/luapower/arrayview

arrayview is a struct with just a pointer & length that maps the idea of a finite array over a given buffer so that you can apply bound-checked access, copying, sorting, etc. over that buffer. dynarray is a malloc'ed arrayview, so basically an arrayview with additional methods and overloads for inserting and removing elements, etc. (and it actually uses arrayview instead of duplicating its code). The APIs are documented in the source code for now for many reasons (for one, this code was already been rewritten once and it will be further modified with increased usage).

Two points I want to make on these libs.

1. The necessity of a tier-0 lib

Even though they are the most basic libs you can imagine, they are necessarily not dependency-free, as they both depend on the low module (https://github.com/luapower/low) which is my tier-0 lib as discussed in #3. Hopefully it's now easier to see why that kind of lib is needed and what's in it. Here's an incomplete checklist:

• addmethods(T, func) - pattern for declaring methods lazily for containers, see terralang/terra#348
• addproperties(T) - adds T.properties which can be quotes or macros so that t.prop redirects to t.properties.prop.
• after_getmethod(), before_entrymissing() etc. - allows a metamethod to be assigned multiple independent handlers -- this way you can have say addmethods() on a type which assigns __getmethod and still assign __getmethod afterwards for a different purpose.
• gettersandsetters() - calls t:get_<name>() for t.<name> and t:set_<name>(val) for t.<name> = <val>
• iif(), min(), max(), assert() - ...
• copy(dst, src, len) - typed memmove
• equal(a, b) - typed memcmp
• alloc() - typed realloc
• hash(uint32|uint64, buf, len, hash) - default hash function
• C headers are included with a wrapper around includec() so that 1) the calls are memoized, 2) symbols are dumped into a single table which the low module inherits so that setfenv(1, 'low') gives unprefixed access to functions like memset etc.

Some of these are one-liners that can be copy-pasted to remove the dependency on the tier-0 lib, but some are not, so we need to talk about #3.

2. Number of users is the only reliable indicator of API quality

As I mentioned before, I think the best way (IMO the only way) to make a good lib is to first have an use case for it and have the lib emerge out of that. I strongly believe that an API is only as good as the number of calls to it that are made from a variety of contexts. I can't think of a stronger indicator of quality than that. An API that's not heavily used by at least 2-3 apps/libs is not ready for prime-time no matter how much design effort is put into it. My libs only have 1-2 users so far and already have undergone major refactorings and even rewrites. I'm not sure what the action point is here, rather than the advice to not waste time with "designing" APIs for imagined use cases: nobody's going to use them, and for good reason.