It may make sense to add a license.

➜  tenko git:(master) ✗ ./t
./t: line 8: realpath: command not found
usage: dirname path
./tests/t.sh: line 6: realpath: command not found
usage: dirname path
./tests/t.sh: line 258: syntax error near unexpected token `;'

os: macos 11.2 Beta (20D5029f)

'new.target' must not contain escaped characters

This case 'new.t\u0061rget'.

Tenko output: Error: Parser error! Must be inside/nested a regular function to use new.target

It appears to ignore the Infinity value for the version string. Additionally, optional chaining (a?.b.c?.d;) is refused no matter what version you set it to. As per build 8c96125.

• logical assignment operators
• numeric separators

(Anything else?)

often it's hard to choose a parser based on its features in isolation. it would be useful to add a high-level comparison table of features & perf vs some others:

https://github.com/meriyah/meriyah
https://github.com/acornjs/acorn
https://github.com/jquery/esprima

you may find this useful for profiling tenko: https://meriyah.github.io/meriyah/performance/

The copyPiggies calls should not pass an assignable/destructible value. Should create a getpiggies call instead.

The call sites may be simplified by doing so since they're implying that the returned value is affected by the current value being assignable/destructible, which is not the case.

esprima and babel, estree, acorn has tokens format that has loc field:

{
    "loc": {
        "start": {
            "line": 8,
            "column": 4
         },
         "end": {
             "line": 8,
             "column": 8
         }
    }
} 

Would be great if tenko also has it because recast uses esprima if it can't find tokens field, but it works great for parsers mentioned earlier.

So compatible tokens field is crucial to avoid double parsing.

I think Tenko currently doesn't properly scope the id of a function declaration that's inside a block (or switch).

If I understand properly now then in strict mode it becomes a let that is hoisted within the block and if not strict mode then it becomes a var that is hoisted within the block.

So:

{
  f();
  function f(){}
}

In context of where f binds, is really

{
  var f = function(){}
  f();
}

(Note the discrepancies with .name and some other things. This is only supposed to illustrate how f binds.)

While

"use strict";
{
  function f(){}
  f();
}

Is really

"use strict";
{
  let f = function(){}
  f();
}

Works the same in global, in functions, and in switches.

I ran a fuzzer for a while and found the following cases which are incorrectly rejected. Other than the first two they don't make sense as programs, although that's mostly because I've tried to reduce them as much as possible.

Let me know if you would prefer that I not file bugs found with a fuzzer.

async(0,...a)

export default async=null

(class extends async function(){}{})

let[{}=class{}]=null

a[{...()=>{}}.m()]

--{_:()=>null}._

(class{static get[()=>null](){}}())

Kinda makes more sense although it probably doesn't matter much for performance.

const x = 33;
log(33 | 32 ^ 32);
log(33 & ~32);

TopLevelAwait reached stage 4

Current parsing of optional chaining appears to be buggy in some cases. And I see that estree finally settled on a spec for it so I should update Tenko to conform accordingly.

to play well with eslint, can you add a option to generate range property

refs: https://eslint.org/docs/developer-guide/working-with-custom-parsers#all-nodes

The AST specification
The AST that custom parsers should create is based on ESTree. The AST requires some additional properties about detail information of the source code.

All nodes:
All nodes must have range property.

range (number[]) is an array of two numbers. Both numbers are a 0-based index which is the position in the array of source code characters. The first is the start position of the node, the second is the end position of the node. code.slice(node.range[0], node.range[1]) must be the text of the node. This range does not include spaces/parentheses which are around the node.
loc (SourceLocation) must not be null. The loc property is defined as nullable by ESTree, but ESLint requires this property. On the other hand, SourceLocation#source property can be undefined. ESLint does not use the SourceLocation#source property.
The parent property of all nodes must be rewritable. ESLint sets each node's parent property to its parent node while traversing, before any rules have access to the AST.

Seems like tenko does not have these features.
Have you seen tree-sitter (https://www.youtube.com/watch?v=Jes3bD6P0To) ? What was the reason to create a new parser without incremental parsing and error recovery ?

Output should probably show more than just an empty object

This case

'"use strict"; (async yield => {})'

in the REPL doesn't mark 'web compat' an 'sloppy' as green. Instead they are red and indicate that this case also should fail in sloppy mode.

Nothing wrong with Tenko and if you try to parse this one in the REPL the colors are correct

'(async yield => {})'

in string literals - "\9" - is valid. Tenko throws

When CoverCallExpressionAndAsyncArrowHead is refined to async ArrowFormalParameters, the ArrowFormalParameters nonterminal is given the flag [+Await], and await is only a valid IdentifierReference when the flag is -Await. So async ({await}) => 1 should be a parse error. But it isn't. Instead the await is treated as an identifier.

Silly oversight. Also need to write a test for it.

if ($tp_next_type === $ID_in || $tp_next_type === $ID_in) {
      return THROW_RANGE('Cannot use `let` as a regular var name as the lhs of `in` or `instanceof` in a toplevel expression statement', tok_getStart(), tok_getStop()); // And why would you.
    }

"Ergonomic brand checks for Private Fields" is now stage 4.

https://github.com/tc39/proposal-private-fields-in-in

yarn install
Using globally installed version of Yarn
yarn install v1.12.1
warning package.json: No license field
info No lockfile found.
warning [email protected]: No license field
[1/4] 🔍  Resolving packages...
error Can't add "acorn": invalid package version undefined.

If I try npm, it hangs at:

⸨    ░░░░░░░░░░░░░░⸩ ⠋ fetchMetadata: sill resolveWithNewModule [email protected] checking installable status

I wonder whether this could be something in my environment (FB laptop), but other things seem to work.

Tenko doesn't parse 'async function f() { for await (async of []); }' and similar cases

I was playing around with Tenko and discovered that the parser accepted cases like [...{x} /= y].

Use of an compound operator in this case should trigger an error.

Note that this is only for array and object literals. Cases like '[...a /= y]' should not trigger any errors

does this work with pure es5 as well ?