tmikov / jscomp Goto Github PK

Ahead of time JavaScript compiler

License: Apache License 2.0

Makefile 1.76% JavaScript 21.97% Shell 3.57% HTML 5.25% TypeScript 1.70% CMake 0.26% C++ 2.35% C 40.75% Perl 0.09% Batchfile 0.19% Groff 4.61% Python 16.74% Assembly 0.01% Objective-C 0.11% Objective-C++ 0.01% Swift 0.01% Emacs Lisp 0.11% NSIS 0.02% M4 0.49%

jscomp's Introduction

Tzvetan Mikov <[email protected]>

An "ahead-of-time" compiler for the JavaScript programming language.

Warning

The compiler is still under heavy development and not even "beta" quality yet. Even though many things work, many still don’t. This document is mainly for potential contributors.

Goals and Motivation

JSC is a classic "ahead-of-time" compiler. At compile time it generates static executable code (currently C++). It implements ECMAScript 5 (with some minor exceptions) and provides an ever increasing level of Node.js API compatibility.

The informal goals of the project are:

Modern ECMAScript compatibility
Acceptable performance
Predictability, observability, simplicity of the execution model and small memory footprint

Try It

It has been tested on Mac and Debian Linux. Experimental Windows support is available under the cygwin branch; for Windows/Cygwin, please check-out that branch and follow the instructions in its README (note that it may be lagging behind the main development branch). Better Windows support is coming in the future (contributions are welcome).

To build JSC You need:

Node.js v0.12
a C++11 compiler (Apple Clang or gcc-4.9 both work)
CMake version at least 2.8
Clone the repository
From the root of the repository, build the runtime:

npm install
make runtime

From the root of the repository, build the compiler:

make

Run the compiler (only from the project root):

./jsc

Compile and run an example

./jsc examples/factorial.js -o factorial
./factorial

Hacking

See docs/hacking.adoc

Performance of the Compiled Code

It is not expected that the performance of compiled applications will ever rival 'v8'. JavaScript is an awful language for static compilation - it almost seems designed to foil any attempts at optimization, and so a JIT will always have a significant performance advantage.

With that said, performance still can be acceptable and even good. We have a number of potential optimizations planned, including whole program type inference.

Performance of the Compiler

JSC may feel slow at times. It generates a huge C++ source file containing all used modules and runtime. There are plans afoot to speed this process greatly, but even now it should be taken in context of the goals:

Produce executables with minimal runtime dependencies
Perform whole-program optimizations

Status

JSC is still very young. It was started on Jun 6th, after coming back from JSConf 2015, while still riding high on JavaScript enthusiasm, and yet keeping strong a life-long aversion to interpreters :-)

Syntax: it uses a mature and tested JavaScript parser (Acorn), so syntactically it is able to handle the complete language.
Currently we parse and support ECMAScript 5, with emphasis on strict mode. We do support non-strict mode, but are not putting a lot of effort into testing it. The entire application runs either in strict or non-strict mode. According to the standard, the mode can be changed on a per-function basis, but we do not intend to support that ever - the cost and complexity is not worth it. If an application absolutely needs that, it should be fixed :-), or it should use 'v8'. (Note that strict code can usually run fine in non-strict mode).
Not implemented yet:
- with : it is last on our list. We think we have come up with a reasonable way to implement it without a huge performance penalty, but since we don’t use with at all, our motivation is low.
- Object.setPrototypeOf() and \_\_proto__ : the plan is not to support prototype modification. This is not a judgement, but if you really need that you should use a different JavaScript implementation.
Differing from the spec:
- bind : it is a true implementation (not a shim), where the only difference from the spec is that it has a read-only prototype property (the spec says it shouldn’t have one at all). We need that property due to how our object allocation work and we believe that well behaved code shouldn’t be relying on its absence.
- arguments : implemented only in its 'strict mode' functionality. In other words, changes to arguments do not propagate back to the actual arguments. There is also no caller/callee, properties. We can implement all of that (non-strict mode stuff), but it is not high on our current list of priorities.
- Regular expressions: implemented using PCRE2. PCRE2 has some incompatibilities with JavaScript regexes, specifically the treatment of unicode whitespace. It is also too flexible for its own good - lots of options can be changed from within the pattern itself - which is naturally not compatible with with JavaScript. The implementation should be considered experimental and should not be used with "untrusted" regular expressions.
The object system is implemented, but some of the built-in constructors and many methods are not available. The plan is to implement as mush as possible in JavaScript.
Some built-in methods are still missing or have limited functionality.

Technical Details

C++ 11

The compiler generates C code that must be compiled with a C 11 compiler. The runtime is extremely inefficient - for example it uses std::map<> for property storage. Our excuses are:

We concentrated on correctness and wanted to get something working ASAP
We craftily left obvious optimizations opportunities alone because we wanted to get easy performance boosts later in the project (with the corresponding boosts in morale). See the TODO section.

Garbage Collector

There is a precise 'stop the world' mark and sweep garbage collector.

Node.js Compatibility

Node.js compatibility is achieved by compiling unmodified Node.js built-in JavaScript modules (we use no C/C code from either Node or v8). This can be an occasionally painfull process, as these modules rely on internal C interfaces which must be reverse engineered and recreated. Since these modules are unmodified they serve a dual purpose - validate our compiler and environment as well as provide great Node.js compatibility.

JavaScript → Native Interface

Since this is a static compiler, connecting C/C++ and JavaScript is conceptually simpler than the interfaces provided by V8 and/or Node. However we are still working on defining interfaces which would be easier to use in practice without in-depth knowledge of the internals of the compiler and runtime system.

The __asm__ built-in is conceptually similar to its equivalent in GCC. Examples of its usage can be seen all over the runtime library (e.g. in runtime/js/core).

TODO

Short term

Transition the runtime to C
Use 'hidden classes' instead of property maps.
'NaN boxing' instead of explicit tagging
Copying generattional garbage collector (we believe it is important to do this work as early as possible as it has signigicant implications on code generation and the runtime).
Better implementation of Node.js 'Buffer' - currently we are using an inefficient implementation from Browserify.
Fill in missing runtime APIs (e.g. Date).

Medium term

Speed up compilation by caching compiled modules
Better source-level debugging
Support for source maps
ES6 support
IR-level optimizations and register allocation
TypeScript integration
V8 compatibility layer for existing Node extensions

Long term

Module level eval() (by building and interpreting an AST)
REPL

Philosophical Motivation

When released, jsc will support the ECMAScript 6 standard (or later), and will be compatible with 'Node.js' libraries and extensions. Module level eval will also be supported (with performance cost). The goal is to be able to recompile most existing 'Node.js' applications without changes.

As we mentioned, a static JavaScript compiler can never rival the performance of a JIT, due to the design of the language itself. But, it can still produce binaries with 'sufficient' or 'useful' performance.

Perhaps even more importantly, the statically compiled binaries will have very predictable performance, which doesn’t change. The produced code can be trivially examined, debugged, and reasoned about - it is not hidden in a huge opaque JIT compiler. 'v8' has excellent diagnostic and visualization tools, but by its very nature it is very complex and so are its tools. Even for an experienced assembler programmer (not to say a casual JavaScript developer), it can be very difficult to decipher or predict what 'v8' is doing.

A JIT, also by its very nature, has big and somewhat unpredictable memory requirements. Different versions of code are kept around, compiled, decompiled, etc. It can get very challenging especially when running multiple ones in parallel, given that nothing can be shared between them. A static compiler avoids all of these problems.

Lastly, the biggest and more important motivation is for fun. We like making compilers, languages and runtimes. So, why not?

License and Copyright

This project (with the exception of components with different licenses, listed below) is licensed under the Apache License v2.0. See LICENSE in the project root.

Components with different licenses:

Acorn is licensed under the terms of its license in acorn/LICENSE.
pcre2 is licensed under the terms of its license in runtime/deps/pcre2/LICENSE.
dtoa and g_fmt are licensed under the terms of the license in runtime/deps/dtoa/dtoa.c and runtime/deps/dtoa/g_fmt.c.
buffer is licensed under the terms of runtime/js/modules/buffer/LICENSE
base64-js is licensed under the terms of runtime/js/modules/buffer/node_modules/base64-js/LICENSE.MIT
ieee754 is licensed under the terms of runtime/js/modules/buffer/node_modules/ieee754/LICENSE
is-array is licensed under the terms of runtime/js/modules/buffer/node_modules/is-array/Readme.md
JSON-js (from https://github.com/douglascrockford/JSON-js) is in the public domain.
Node code is licensed under the terms of its license in "runtime/js/nodelib/LICENSE+.
libuv : runtime/deps/libuv/LICENSE
gyp: runtimr/deps/gyp/LICENSE

jscomp's People

Contributors

Stargazers

Watchers

Forkers

martindale lsm minadaky ericlu88 aylusltd 0x4e38 skyschulz lezsakdomi

jscomp's Issues

Buffer.utf8Slice takes half of the running time of the compiler

It was always known that Browserify's "Buffer" implementation was a temporary fix - it was cool because it worked without modification, validating the compiler, but was probably slow.

Well, now it is known how slow it really is - it takes half the running time of the compiler when compiling anything. It appends every character to the string using two additions. This causes immense pressure on the garbage collector among other things.

Improve Array.prototype.sort()

Array.prototype.sort() is an extremely slow implementation:

it is completely generic. For example it doesn't benefit if it is run on an actual Array or a TypedArray, where it could use much faster direct access to elements.
it uses exchange sort (yes, I know, but I wanted to get something going in a very short time)

dirname and filename should change based on the directory of the executable

That behavior best matches the behavior of a Node.js app

allocate env indexes to "global vars" in non-strict mode

finally (as well as rethrowing) loses the original stack trace

For obvious reasons. We might need to store the stack trace in the exception (duh!).

parseInt() doesn't follow the spec

parseInt() returns a results for huge integers that differ from Node and JS, and they all differ from my reading of the spec.

toLower/UpperCase only converts ASCII characters

Doing it correctly in Unicode is not that easy and the current use cases didn't need it to work in Unicode. With that said, I hate leaving library functions half-implemented (String.prototype.split(), I am looking at you as well!)

Add line numbers debugging support

Associate a source location (line and file name) with every IR instruction, just like we are currently doing with the call instructions.

Awkward make issue

When running make runtime, I'm presented with this syntax error:

cd runtime/deps/libuv && ./gyp_uv.py
  File "./gyp_uv.py", line 39
    print 'Error running GYP'
                            ^
SyntaxError: Missing parentheses in call to 'print'. Did you mean print('Error running GYP')?
make: *** [Makefile:26: runtime/deps/libuv/out/Makefile] Error 1

Better exception handling

Since we are using setjmp/longjmp, non-volatile variables are (practically speaking) undefined after the longjmp(). Currently we are handling this (as well the needed code duplication for 'finally') by generating a nested function for the handler and forcing all variables it uses to be flagged as escaping.

Ideally we should track which variables are used by the handler (and by "used" I mean live values before the handler runs) and force them to memory.

seal() and preventExtensions() have no effect on array elements

New primitive type for single-character strings

It seems like it would be beneficial to specialize 1-character strings so we don't need a heap allocation for simply extracting a character (e.g. x = "aaaaa"[2]).

Compilation fails again with gcc-4.9 (and probably Linux)

incorrect code generation with ?: and ||

function badfunc (a, b)
{
return (a ? '/' : 'a') || '.';
}

autoboxing of 'this' in non-strict mode

ECMAScript 5.1 says "The thisArg value is passed without modification as the this value. This is a change from Edition 3, where a undefined or null thisArg is replaced with the global object and ToObject is applied to all other values and that result is passed as the this value."

However both Mozilla and V8 seem to obey this only in strict mode. In non-strict mode they always perform autoboxing. Probably for compatibility reasons.

This issue is simply to note this. In the future we might decide to do something about it.

Ability to wrap a native pointer

Either via a separate managed object, or directly a new tag type. If the former, possibly more data could be stored in the managed object.

The identifier of a function expression is not visible in the function

See bugs/badfuncex.js

var f = function myfunc () { myfunc(); }

doesn't compile

root@91b8df194dd4:/usr/local/lib/jscomp# ./jsc examples/factorial.js -o factorial
/usr/local/lib/jscomp/runtime/js/modules/buffer/index.js:117:7: warning: operator '==' is not recommended
/usr/local/lib/jscomp/runtime/js/modules/buffer/index.js:221:13: warning: operator '!=' is not recommended
/usr/local/lib/jscomp/runtime/js/modules/buffer/index.js:342:10: warning: condition is always true
/usr/local/lib/jscomp/runtime/js/modules/buffer/node_modules/is-array/index.js:32:20: warning: operator '==' is not recommended
runtime/release/libuv.a(signal.o): In function `uv__signal_block_and_lock':
/usr/local/lib/jscomp/runtime/deps/libuv/out/../src/unix/signal.c:102: undefined reference to `pthread_sigmask'
runtime/release/libuv.a(signal.o): In function `uv__signal_unlock_and_unblock':
/usr/local/lib/jscomp/runtime/deps/libuv/out/../src/unix/signal.c:114: undefined reference to `pthread_sigmask'
runtime/release/libuv.a(signal.o): In function `uv__signal_global_once_init':
/usr/local/lib/jscomp/runtime/deps/libuv/out/../src/unix/signal.c:67: undefined reference to `pthread_once'
runtime/release/libuv.a(signal.o): In function `uv__signal_block_and_lock':
/usr/local/lib/jscomp/runtime/deps/libuv/out/../src/unix/signal.c:102: undefined reference to `pthread_sigmask'
runtime/release/libuv.a(signal.o): In function `uv__signal_unlock_and_unblock':
/usr/local/lib/jscomp/runtime/deps/libuv/out/../src/unix/signal.c:114: undefined reference to `pthread_sigmask'
/usr/local/lib/jscomp/runtime/deps/libuv/out/../src/unix/signal.c:114: undefined reference to `pthread_sigmask'
runtime/release/libuv.a(thread.o): In function `uv_thread_create':
/usr/local/lib/jscomp/runtime/deps/libuv/out/../src/unix/thread.c:66: undefined reference to `pthread_create'
runtime/release/libuv.a(thread.o): In function `uv_thread_join':
/usr/local/lib/jscomp/runtime/deps/libuv/out/../src/unix/thread.c:80: undefined reference to `pthread_join'
runtime/release/libuv.a(thread.o): In function `uv_mutex_trylock':
/usr/local/lib/jscomp/runtime/deps/libuv/out/../src/unix/thread.c:130: undefined reference to `pthread_mutex_trylock'
runtime/release/libuv.a(thread.o): In function `uv_rwlock_init':
/usr/local/lib/jscomp/runtime/deps/libuv/out/../src/unix/thread.c:145: undefined reference to `pthread_rwlock_init'
runtime/release/libuv.a(thread.o): In function `uv_rwlock_destroy':
/usr/local/lib/jscomp/runtime/deps/libuv/out/../src/unix/thread.c:150: undefined reference to `pthread_rwlock_destroy'
runtime/release/libuv.a(thread.o): In function `uv_rwlock_rdlock':
/usr/local/lib/jscomp/runtime/deps/libuv/out/../src/unix/thread.c:156: undefined reference to `pthread_rwlock_rdlock'
runtime/release/libuv.a(thread.o): In function `uv_rwlock_tryrdlock':
/usr/local/lib/jscomp/runtime/deps/libuv/out/../src/unix/thread.c:164: undefined reference to `pthread_rwlock_tryrdlock'
runtime/release/libuv.a(thread.o): In function `uv_rwlock_rdunlock':
thread.c:(.text.uv_rwlock_rdunlock+0x5): undefined reference to `pthread_rwlock_unlock'
runtime/release/libuv.a(thread.o): In function `uv_rwlock_wrlock':
/usr/local/lib/jscomp/runtime/deps/libuv/out/../src/unix/thread.c:179: undefined reference to `pthread_rwlock_wrlock'
runtime/release/libuv.a(thread.o): In function `uv_rwlock_trywrlock':
/usr/local/lib/jscomp/runtime/deps/libuv/out/../src/unix/thread.c:187: undefined reference to `pthread_rwlock_trywrlock'
runtime/release/libuv.a(thread.o): In function `uv_rwlock_wrunlock':
/usr/local/lib/jscomp/runtime/deps/libuv/out/../src/unix/thread.c:196: undefined reference to `pthread_rwlock_unlock'
runtime/release/libuv.a(thread.o): In function `uv_once':
/usr/local/lib/jscomp/runtime/deps/libuv/out/../src/unix/thread.c:202: undefined reference to `pthread_once'
runtime/release/libuv.a(thread.o): In function `uv_sem_init':
/usr/local/lib/jscomp/runtime/deps/libuv/out/../src/unix/thread.c:268: undefined reference to `sem_init'
runtime/release/libuv.a(thread.o): In function `uv_sem_destroy':
/usr/local/lib/jscomp/runtime/deps/libuv/out/../src/unix/thread.c:275: undefined reference to `sem_destroy'
runtime/release/libuv.a(thread.o): In function `uv_sem_post':
/usr/local/lib/jscomp/runtime/deps/libuv/out/../src/unix/thread.c:281: undefined reference to `sem_post'
runtime/release/libuv.a(thread.o): In function `uv_sem_wait':
/usr/local/lib/jscomp/runtime/deps/libuv/out/../src/unix/thread.c:290: undefined reference to `sem_wait'
runtime/release/libuv.a(thread.o): In function `uv_sem_trywait':
/usr/local/lib/jscomp/runtime/deps/libuv/out/../src/unix/thread.c:302: undefined reference to `sem_trywait'
runtime/release/libuv.a(thread.o): In function `uv_cond_init':
/usr/local/lib/jscomp/runtime/deps/libuv/out/../src/unix/thread.c:334: undefined reference to `pthread_condattr_setclock'
runtime/release/libuv.a(thread.o): In function `uv_barrier_init':
/usr/local/lib/jscomp/runtime/deps/libuv/out/../src/unix/thread.c:481: undefined reference to `pthread_barrier_init'
runtime/release/libuv.a(thread.o): In function `uv_barrier_destroy':
/usr/local/lib/jscomp/runtime/deps/libuv/out/../src/unix/thread.c:486: undefined reference to `pthread_barrier_destroy'
runtime/release/libuv.a(thread.o): In function `uv_barrier_wait':
/usr/local/lib/jscomp/runtime/deps/libuv/out/../src/unix/thread.c:492: undefined reference to `pthread_barrier_wait'
runtime/release/libuv.a(thread.o): In function `uv_key_create':
/usr/local/lib/jscomp/runtime/deps/libuv/out/../src/unix/thread.c:501: undefined reference to `pthread_key_create'
runtime/release/libuv.a(thread.o): In function `uv_key_delete':
/usr/local/lib/jscomp/runtime/deps/libuv/out/../src/unix/thread.c:506: undefined reference to `pthread_key_delete'
runtime/release/libuv.a(thread.o): In function `uv_key_get':
/usr/local/lib/jscomp/runtime/deps/libuv/out/../src/unix/thread.c:512: undefined reference to `pthread_getspecific'
runtime/release/libuv.a(thread.o): In function `uv_key_set':
/usr/local/lib/jscomp/runtime/deps/libuv/out/../src/unix/thread.c:517: undefined reference to `pthread_setspecific'
runtime/release/libuv.a(linux-core.o): In function `uv__io_poll':
/usr/local/lib/jscomp/runtime/deps/libuv/out/../src/unix/linux-core.c:252: undefined reference to `pthread_sigmask'
/usr/local/lib/jscomp/runtime/deps/libuv/out/../src/unix/linux-core.c:231: undefined reference to `pthread_sigmask'
collect2: error: ld returned 1 exit status
error: child process terminated with code 1

Dockerfile here