shapesecurity / shift-java Goto Github PK

See shapesecurity/shift-parser-js#310.

Should the functional classes be split out into their own repository?

It currently only wraps a random Identifier in / characters. See estools/esfuzz/src/nodes/RegExp for a random RegExp generation example.

e.g.

var x = 'outside';
var probeParam, probeBody;

({
  *m(
    ...[_ = (eval('var x = "inside";'), probeParam = function() { return x; })]
  ) {
    probeBody = function() { return x; }
  }
}.m().next());

The Scope.Type should be passed into this function, not a boolean. Additionally, we could probably create a wrapper for this function that passes the appropriate Scope.Type given the type of the first parameter.

Provide a new mode of parsing that Instead of throwing an exception and aborting parsing on the first JS error encoutered, parser attempts to continue parsing and provides collections of all the syntax errors.

... then bump the version.

Related:
shapesecurity/shift-reducer-js#10
shapesecurity/shift-ast-js#9
shapesecurity/shift-spec#14

e.g. https://github.com/tc39/test262/blob/master/test/language/statements/for-of/scope-body-lex-open.js etc

var probeBefore = function() { return x; };
let x = 'outside';
var probeExpr, probeDecl, probeBody;

for (
    let [x, _, __ = probeDecl = function() { return x; }]
    of
    [['inside', probeExpr = function() { typeof x; }]]
  )
  probeBody = function() { return x; };

assert.sameValue(probeBefore(), 'outside');
assert.throws(ReferenceError, probeExpr);
assert.sameValue(probeDecl(), 'inside', 'reference from ForDeclaration');
assert.sameValue(probeBody(), 'inside', 'reference from statement body');

See https://github.com/shapesecurity/shift-parser-expectations

Probably this will uncover some bugs.

Also upstream tests from this package to there.

See tests from EarlyErrorTests.java:

testScriptFailure("a\\u0000", 0, "Unexpected end of input"); // TODO this should error
testScriptFailure("a\\u{0}", 0, "Unexpected end of input"); // TODO this should error

The fuzzer won't generate these any more, but the validator should still be looking for them. Same thing may apply for ReturnStatement, I haven't checked.

The way the current ReducerP is constructed is that all the state will have only one reference globally unless the user intentionally save it somewhere. It shows the possibility to treat the states linear typed and it will be a natural optimization to allow mutation in the object. But MonoidalReducer is caching the same empty object and using it everywhere. It prevent people to write bug-resistant mutational code and degrades performance.

Currently the reducer has methods which look like

public State reduceClassDeclaration(
  ClassDeclaration node,
  State name,
  Maybe<State> _super,
  ImmutableList<State> elements
)

where of course ClassDeclaration is shaped like

class ClassDeclaration(
  BindingIdentifier name,
  Maybe<Expression> _super,
  ImmutableList<ClassElement> elements
)

There is a guarantee provided by the director, but not the type system, that the fields of the node and the noninitial arguments of the method line up, in the sense that node._super.isJust() == _super.isJust(), node.elements.length == elements.length, etc.

The fact that this isn't enforced by the type system leads to a lot of unchecked .fromJust()s. We could avoid this by defining, for each node type, a node-with-associated-data type along the lines of

<T> class ClassDeclarationWithT(
  Pair<BindingIdentifierWith, T> name,
  Maybe<Pair<Expression, T>> _super,
  ImmutableList<Pair<ClassElementWith, T>> elements
)

and then making the signature of methods in Reducer<State> be

public State reduceClassDeclaration(
  ClassDeclarationWithT<State> state
)

This does have the downside that you don't get a reference to the original node, which the LazyCloneReducer among other interfaces wast - for example, you need the original node to look up its location in the data structure provided by ParserWithLocation. I suppose it could always be passed as a second argument for those cases.

You can of course automatically convert between reducers of this type and reducers of the existing type, though going from reducers of the existing type to reducers of this type requires having all those unchecked .fromJust()s.

"use strict";
var await;

is a valid script.

IE8 has problem with parsing code with single statement without braces, so the minifier can't safely have all single-statement blocks removed. For example:

if (false) do while (a) do this; while (b); while (c);

has trouble with IE8 but fine with other browsers. Putting braces around the outermost do-while statement and the problem will be gone.

if (false) do {while (a) do this; while (b); } while (c);

Suggestion is not to remove single-statement blocks.

I have commented them out for now. See https://github.com/shapesecurity/shift-java/blob/master/src/test/java/com/shapesecurity/shift/codegen/CodeGenTest.java#L506-L516.

"\u00005", which is "\0" + "5", is generated as "\05", which is a single character.

When I'm trying to parse ES5 with parser combinator, the inversed order is really annoying.
For example, when I parse ObjectProperty, I will first parse propertyName before I parse the expression, but DataProperty put the expression before the name.

It is useless to particularly define the order in an unnatural order, please fix it.

I can find any description of code formatting capabilities in the codegen API. Are there any?
https://github.com/shapesecurity/shift-java/tree/es6/src/main/java/com/shapesecurity/shift/codegen

The tokenizer splits /=3/ into a /= token and a 3/ token. It make no sense.

Provide a way to optionally parse comments

We currently only have test262-parser-tests

We get this right in JS, but apparently not here.

New version is much more performant

This test fails to parse, but shouldn't. Test262

If an entry is present in the map, it has at least one entry.

Alternatively, use a proper MultiMap type.

var a = tag`</script>`;

should become

var a = (function(){
  var literalParts = ["</\x73cript>"], computedParts = [];
  literalParts.raw = ["</\x73cript>"];
  return tag.apply(null, [literalParts].concat(computedParts));
}());

😞

Sub-typing in the constructor is suffice.

• delete this comment
• create GenericParser<AdditionalStateT>
• change Parser to extend GenericParser<Unit>
• change ParserWithLocation to extend GenericParser<SourceLocation>
• change markLocation to finishNode
• add startNode, which is also a no-op in Parser; override in ParserWithLocation to call getLocation

SourceLocation is part of the shift-ast specification but it is not implemented for the Node interface. Implementation probably requires Node interface to become an abstract class implementation but since Node interface is referenced from all the AST model classes, it is best to build this change to model generator.

That's not how line numbers work.

I don't know what the cause of this weird inconsistency is, but we should make sure we add cloning support to the CloneReducer once they are in place.

"Shift AST Constructors" is the title for many of the README.mds

path : https://github.com/shapesecurity/shift-java/blob/master/src/main/java/com/shapesecurity/shift/js/path/README.md
codegen : https://github.com/shapesecurity/shift-java/blob/master/src/main/java/com/shapesecurity/shift/js/codegen/README.md
main : https://github.com/shapesecurity/shift-java

cc @dwightmulcahy @ikarienator

#102 introduced a bug: the locations were calculated, but not stored.

Since com.shapesecurity.shift.parser.ParserWithLocation#locations was not replaced with the result of locations.put and also it was marked as final, the locations collection was not growing.

Function declarations in export default declarations have a name of *default*.

"Clone" implies memory copying, when all that is necessary is that it can reconstruct an AST node from the result of reconstructing its children.

E.g. this is rejected.

After checking out, I get the above error when running mvn test

See shapesecurity/shift-parser-js#389