blynn / nex Goto Github PK
View Code? Open in Web Editor NEWLexer for Go
Home Page: http://cs.stanford.edu/~blynn/nex/
License: GNU General Public License v3.0
Lexer for Go
Home Page: http://cs.stanford.edu/~blynn/nex/
License: GNU General Public License v3.0
By using nex to parse SQL, I encounter the following problem:
Say two statements "BETWEEN expr AND expr" and "IF( expr AND expr)", there are two "AND" token that are different. In flex we can write rule like this:
%s BTWMODE
AND { BEGIN INITIAL; return AND; }
AND { return ANDOP; }
BETWEEN { BEGIN BTWMODE; return BETWEEN; }
%s represents an inclusive-mode lexing.(%x exclusive-mode correspondingly)
What is the equivalent counterpart of it?
In addition, flex has an option "case-insensitive" to ignore case, seems nex has no such switch, so I have to write rules like /[Ss][Ee][Ll][Ee][Cc][Tt]/ to represent the "select" keyword. Is there a recommend way to achieve this?
Quite hilariously, an unpaired curly brace inside nex golang code causes lex/parse errors with nex itself!
Example:
/\$[a-z][a-z0-9]*{[0-9]+}/
{
yylex.pos(lval) // our pos
s := yylex.Text()
a := strings.Split(s, "{") // XXX: close match here: }
lval.str = a[0]
return IDENTIFIER
}
Note the comment I added with a close brace. I added that as a workaround so that this works. Remove it and you'll see nex errors:
panic: unmatched '{'
goroutine 16 [running]:
runtime.gopanic
../../../libgo/go/runtime/panic.go:493
main.$nested34
/builddir/build/BUILD/nex-5344f151fd3251726650dffd30a531d3f1bddc17/nex.go:1027
main.$nested35
/builddir/build/BUILD/nex-5344f151fd3251726650dffd30a531d3f1bddc17/nex.go:1094
main.process
/builddir/build/BUILD/nex-5344f151fd3251726650dffd30a531d3f1bddc17/nex.go:1099
main.main
/builddir/build/BUILD/nex-5344f151fd3251726650dffd30a531d3f1bddc17/main.go:81
runtime_main
../../../libgo/runtime/proc.c:606
HTH
hi,
how can i generate a syntax error using nex and add more information the the syntax error?
i tried to create a costum error function for nex:
nex -e=true lexer.nex
here my nex file:
/while/ { return WHILE }
/print/ { return PRINT }
/;/ { return END_LINE }
/+|-/ { lval.s = yylex.Text(); return ADD_OP }
/|// { lval.s = yylex.Text(); return MUL_OP }
/=/ { return ASSIGN }
/(/ { return BEGIN_EXPRESSION }
/)/ { return END_EXPRESSION }
/{/ { return BEGIN_BLOCK }
/}/ { return END_BLOCK }
/[0-9]+/ { lval.s = yylex.Text(); return NUMBER }
/[a-z][a-z0-9]/ { lval.s = yylex.Text(); return IDENTIFIER }
/\n+/ { lineno += len(yylex.Text()) }
//
package dsl
import "fmt"
var lineno int // idk if thats good to put it global?
func (yylex Lexer) Error(e string) {
yylex.p.err = fmt.Sprintf("Syntax error in line %d", lineno+1)
}
i can print the line number where the syntax error happend.
EDIT: there are those 2 variables in the type Lexer struct:
l, c int // line number and character position
are they incremented automatically and can i remove my lineno variable?
how can you add more information to the error message? i want to add the token/s that was not recognized in the parsing...
e.g.
how can i keep track of the last token that i sent to the parser, then on error just say Printf("error after %s on line %d" , humanReadable(lastToken,lastLine) ??
Are multiline comments possible with nex? I've been playing with nex quite a bit and I'm very impressed. However, I'm having trouble figuring out how to handle multiline comments. Can this been done with the nesting feature? I've tried but could not figure it out. Below is one thing I tired but I regex is greedy the nesting grabs the rest of the document.
/[0-9]+/ { println("INTEGER:", txt()) }
/[0-9]+\.[0-9]*/ { println("FLOAT:", txt()) }
/if|then|begin|end|procedure|function/
{ println( "KEYWORD:", txt()) }
/[a-z][a-z0-9]*/ { println( "ID:", txt()) }
/\+|-|\*|\// { println("OP:", txt()) }
/[ \t]+/ { /* eat up whitespace */ }
/\/\*.*|\n*\*\// < { println("BEG_MULTI:", txt()) }
/.*|\n*\*\// { println("BEG_COMMENT:", txt()) }
/\n/ { println("NEWLINE") }
> { println("END_COMMENT", txt()) }
/\n/ { println("NEWLINE") }
/./ { println("UNRECOGNIZED CHAR:", txt()) }
//
package main
import (
"os"
)
func main() {
lex := NewLexer(os.Stdin)
txt := func() string { return lex.Text() }
NN_FUN(lex)
}
Is there a way to define case insensitive regular expressions? I am making a SQL parser and I need to be able to parse correctly SELECT
, select
or any of its variants.
I do know that Flex has %option caseless
, is there something similar for Nex?
So I was trying to migrate a BISON/FLEX basic parser written in "C" to GO. Settled on goyacc and nex. But was seeing bizarre behavior, where lines were not being processed until I entered a newline again. To eliminate questions about my grammar, I tried building the rp test program, and saw the same behavior. e.g. I enter '111 111 * 3 /', and nothing is printed until I enter another newline (or ctrl-d), at which point it prints the answer. I patched yyDebug to 3 to observe the grammar actions, and saw that it wasn't completing the parse, because the terminal newline was not being returned to the parser.
Output with yyDebug==3:
reduce 1 in:
state-0
111 111 * 3 /
lex NUM(57346)
reduce 5 in:
state-5
lex NUM(57346)
reduce 5 in:
state-5
lex '*'(42)
reduce 8 in:
state-11
lex NUM(57346)
reduce 5 in:
state-5
lex '/'(47)
reduce 9 in:
state-12
(notice how it hasn't printed anything yet?) Now I enter another newline:
lex '\n'(10)
reduce 4 in:
state-6
4107
reduce 2 in:
state-2
and it finishes up.
p.s. I cloned this straight out of github...
I'm guessing that Nex is not thread safe if it uses globals, am I wrong? My program needs to be able to parse in multiple threads and I'm wondering if I will need to lock a mutex for every time Lex will be called.
I'm having a hard time understanding the behavior of the lexer in the following case:
/\(/ { fmt.Printf("-> %q\n", yylex.Text()) }
/\)/ { fmt.Printf("-> %q\n", yylex.Text()) }
/[^( ][^ ]*[^ )]/ { fmt.Printf("-> %q\n", yylex.Text()) }
//
package main
import ("fmt")
func main() {
fmt.Printf("lexing %q:\n", "(rule)")
NN_FUN(NewLexer(strings.NewReader("(rule)")))
fmt.Printf("lexing %q:\n", "( rule )")
NN_FUN(NewLexer(strings.NewReader("( rule )")))
}
Output of nex -r -s huh.nex:
lexing "(rule)":
-> "("
-> "rule"
lexing "( rule )":
-> "("
-> "rule"
-> ")"
Why is the lexer swallowing the trailing bracket when there is no space between the content and the surrounding brackets?. This looks like a bug to me. It has something to do with second character class in the content regex ([^ ]*
), when I change that to also not match )
it works.
for example flex supports something like this:
<<EOF>>
I need a way to return an EOF token to the parser.
There is currently no possibility I could find to get the shortest possible match (non-greedy behaviour).
There should be a possibility to split the following snippet:
<?php
b
?>
text
<?php
a
?>
to these 2 matches:
<?php
b
?>
and
<?php
a
?>
Currently the regex /<\?php.*\?>/
matches the whole text.
Or did I simply miss something?
Thanks
Should be as simple as renaming nex.go to main.go.
Currently ^
and $
symbols convey no special meaning and match literally. I want to be able to use these special symbols inside regular expressions, including nested ones. In the latter case I want them to match at the beginning and at the end of the substring, matched by an outer regular expression.
what if I don't want to panic(err) when there is a parsing error?
Is this possible? Currently I have to do something like:
in parser.y:
top: prog
{
langGlobal = $1.prog
}
;
in main.go:
lexer := NewLexer(os.Stdin)
yyParse(lexer) // writes the result to langGlobal
log.Printf("behold, the AST: %+v\n", langGlobal)
I know it's not directly nex related, but I was hoping someone here might know! Thanks :)
It seems that running nex multiple times on the same file generates different output sometimes. Example:
/a|A/ { return A }
//
package main
I get two different versions (filenames are their shas):
→ diff 55b29ae5b5679cab92281c8e9d3d493cb0f5be6c.go 73edf909a04d60e4f253d456fc4bf8b3ca6eee9c.go
155,156c155,156
< case 65: return 1
< case 97: return 2
---
> case 65: return 2
> case 97: return 1
I haven't looked at your code yet, but a possible source of randomness could be go's map iteration order (c.f. http://stackoverflow.com/a/9621526/220918).
Currently, if you try to put dash (-
) inside a character group (at the beginning or at the end of it) nex will panic: bad range in character class
:
/[-a-z]/ {
}
//
package main
import ("fmt";"os")
func main() {
NN_FUN(NewLexer(os.Stdin))
}
I wish to be able to use dashes like that, because else regexp becomes cumbersome if your identifiers can contain a dash.
As far as I can tell, there doesn't seem to be a way to change the func (yylex Lexer) Error(e string)
function. This function gets called automatically from a go tool yacc
generated parser, so it would be nice to be able to customize it.
In the source there is func (yylex *Lexer) Line() int
func (yylex *Lexer) Column() int
. I think it would also be useful to have func (yylex *Lexer) Size() int
which returns position in bytes relative to the start of the files.
I'd like to lex and parse a number of different files together. The approach I'm attempting is the concatenate them together with a https://golang.org/pkg/io/#MultiWriter. I also store a list of cumulative file.Size()
offsets for each file. I get this via Stat
when opening each file for the MultiWriter.
At the end of lexing/parsing if I get an error, I work backwards from position of error to line number and file.
The problem is that I'm doing the "math" with line numbers instead of Size
offsets. This means that I need to loop through each file (before lexing/parsing) and count all the newlines. If Size
was available in addition to Column
and Row
, then this would be much more direct.
I haven't yet worked out how I'd get from the correct size offset to Column and Line, but I figure that's doable somehow.
Help is appreciated if anyone can contribute. This currently works with line number alone, but as I mentioned a Size
offset might be preferable. If there's an alternative technique for lexing/parsing multiple files together as one, please let me know!
Thanks!
I haven't done super serious profiling yet, but in a rather large application we run, our lexer keeps coming up in goroutine dumps at what seems to be a disproportionately high rate, especially given that we lex very small strings as a tiny part of everything our app does. I found a bunch of routines awaiting chan receive in the generated lexer, which kind of immediately raised my eyebrow. Given what the lexer does, does it really need to use channels and goroutines to accomplish its job? I'm not familiar with the design constraints or exactly what kind of guarantees we want to make for the user code that gets embedded in the generator, but it seems like this could all be done more efficiently single-threaded.
Thoughts?
The following code
/[a-z]+: [a-z]+/ <
{ fmt.Println("BEGIN"); }
/[a-z]+:/ {
fmt.Println(1, yylex.Text())
}
/[a-z]+/ {
fmt.Println(2, yylex.Text())
}
> { fmt.Println("END"); }
//
package main
import ("fmt";"os")
func main() {
NN_FUN(NewLexer(os.Stdin))
}
(when processed with nex and executed as echo name: value | ./testcase.nn
) prints
BEGIN
1 name:
2 value
END
...as you would expect. However, if you'll change the second nested expression to /.+/
--
/[a-z]+: [a-z]+/ <
{ fmt.Println("BEGIN"); }
/[a-z]+:/ {
fmt.Println(1, yylex.Text())
}
/.+/ {
fmt.Println(2, yylex.Text())
}
> { fmt.Println("END"); }
//
package main
import ("fmt";"os")
func main() {
NN_FUN(NewLexer(os.Stdin))
}
it will print only
BEGIN
2 name: value
END
That is, in the last case the first nested expression is never matched.
10 type yySymType struct {
11 yys int n int } <---- won't compile
I was having trouble using nex with go yacc so I copied the example to test it out. I compiled, but it didn't work, as soon as I enter some text it breaks
Using:
Mac OS 10.10.1
Go version go1.3.3 darwin/amd64
panic: syntax error
goroutine 16 [running]:
runtime.panic(0xa6300, 0x2081aa300)
/usr/local/Cellar/go/1.3.3/libexec/src/pkg/runtime/panic.c:279 +0xf5
main.Lexer.Error(0x2081ae0c0, 0x2081cc240, 0x1, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0, 0xe67f0, ...)
/Users/donnanicolas/go/src/tu-carrito.com/backend/extractor/rp/rp.nn.go:239 +0x67
main.(*Lexer).Error(0x2081c21e0, 0xe67f0, 0xc)
<autogenerated>:3 +0xa5
main.yyParse(0x22081bc340, 0x2081c21e0, 0x170bc8)
/Users/donnanicolas/go/src/tu-carrito.com/backend/extractor/rp/yaccpar:155 +0x563
main.main()
/Users/donnanicolas/go/src/tu-carrito.com/backend/extractor/rp/rp.nn.go:263 +0x84
goroutine 19 [finalizer wait]:
runtime.park(0x17f90, 0x170b90, 0x16ff09)
/usr/local/Cellar/go/1.3.3/libexec/src/pkg/runtime/proc.c:1369 +0x89
runtime.parkunlock(0x170b90, 0x16ff09)
/usr/local/Cellar/go/1.3.3/libexec/src/pkg/runtime/proc.c:1385 +0x3b
runfinq()
/usr/local/Cellar/go/1.3.3/libexec/src/pkg/runtime/mgc0.c:2644 +0xcf
runtime.goexit()
/usr/local/Cellar/go/1.3.3/libexec/src/pkg/runtime/proc.c:1445
goroutine 20 [chan send]:
main.func·003(0x2081ae120, 0x2081ae0c0, 0x2081ea000, 0x3, 0x3, 0x0, 0x1)
/Users/donnanicolas/go/src/tu-carrito.com/backend/extractor/rp/rp.nn.go:136 +0x9c8
created by main.NewLexerWithInit
/Users/donnanicolas/go/src/tu-carrito.com/backend/extractor/rp/rp.nn.go:193 +0x677
hey there,
playing with nex, and using the below... The problem is I had to add an ERROR
identifier to match anything not previously caught since otherwise random typos in the lexed code are skipped silently. Any advice on how to improve this?
Thanks!
/[ \t\n]/ { /* Skip blanks and tabs. */ }
/{/ { return OPEN_CURLY }
/}/ { return CLOSE_CURLY }
/if/ { return IF }
/else/ { return ELSE }
/=/ { return EQUALS }
/\$[a-z]+/ {
s := yylex.Text()
lval.str = s[1:len(s)] // remove the leading $
return VAR_IDENTIFIER
}
/[a-z]+/ {
lval.str = yylex.Text()
return IDENTIFIER
}
/./ { return ERROR }
//
package main
import ()
Writing a little programming language, and trying to lex a "string":
Eg:
$foo = "hello, world"
This is kind of tricky, in particular in the docs:
**Matching Nuances
Among rules in the same scope, the longest matching pattern takes precedence. In event of a tie, the first pattern wins.**
This means that if I have other similar patterns in the code, this will match when I don't expect it. Additionally, is there some way to specify a string? I don't want to write in every unicode char, and if you do a \*\
match it matches too much!!
Hi,
After typing in line count example in readme, I redirected it's output to a file and issued "go build" on it. And I got "undefine os" error for NewLexer(os.Stdin)
.
Readme says generated code already imports os, but mine (1eadfa3) didn't put it. I am on windows 7.
It would be great if nex had an argument just like yacc's -p to change yy to other things. As it is, I seem to have to write a bunch of wrapper code to make things line up.
Given the nex file:
%{
package main
%}
%union {
n int
}
%token NUMBER
%token ADD SUB MUL DIV ABS
%token EOL
%%
start: term
;
term: NUMBER
| ABS term { $$ = $2 >= 0? $2 : - $2; }
;
%%
Got:
# command-line-arguments
desk.y:33[/Users/drew/src/snazzle/desk/y.go:468]: syntax error: unexpected ?
Expected:
$2 is returned as positive number
I wrote a complex regex and run nex but found an 'out-of-memory' error.
My vps has 4G memory so I doubt somewhere may went wrong.
I tried to find a simpler regex which can cause the same problem, finally, I found one like below:
/(([ \t]*)*)*/ { /* eat up whitespace */ }
It is sensed that nex may hang into inf-loop in cases which has nested klein-closures.
the crash is sth like below:
fatal error: runtime: out of memory
runtime stack:
runtime.throw(0x546c99, 0x16)
/usr/local/go/src/runtime/panic.go:566 +0x95
runtime.sysMap(0xc440200000, 0x20000000, 0xc4401a3e00, 0x5ff718)
/usr/local/go/src/runtime/mem_linux.go:219 +0x1d0
runtime.(*mheap).sysAlloc(0x5e6ec0, 0x20000000, 0x0)
/usr/local/go/src/runtime/malloc.go:407 +0x37a
runtime.(*mheap).grow(0x5e6ec0, 0x10000, 0x0)
/usr/local/go/src/runtime/mheap.go:726 +0x62
runtime.(*mheap).allocSpanLocked(0x5e6ec0, 0x10000, 0x0)
/usr/local/go/src/runtime/mheap.go:630 +0x4f2
runtime.(*mheap).allocStack(0x5e6ec0, 0x10000, 0x0)
/usr/local/go/src/runtime/mheap.go:597 +0x62
runtime.stackalloc(0xc420000000, 0xc4301a3ee0, 0x8000000, 0xc4200001a0, 0x0, 0x0)
/usr/local/go/src/runtime/stack.go:395 +0x2ed
runtime.copystack(0xc4200001a0, 0x20000000, 0x5e3d01)
/usr/local/go/src/runtime/stack.go:839 +0x83
runtime.newstack()
/usr/local/go/src/runtime/stack.go:1070 +0x370
runtime.morestack()
/usr/local/go/src/runtime/asm_amd64.s:366 +0x7f
goroutine 1 [copystack]:
main.gen.func17.1(0x2)
/home/kongdeyu/goprojs/src/github.com/blynn/nex.go:522 fp=0xc4301a42b8 sp=0xc4301a42b0
main.gen.func17.1(0x1)
/home/kongdeyu/goprojs/src/github.com/blynn/nex.go:527 +0xf8 fp=0xc4301a4308 sp=0xc4301a42b8
main.gen.func17.1(0x2)
/home/kongdeyu/goprojs/src/github.com/blynn/nex.go:527 +0xf8 fp=0xc4301a4358 sp=0xc4301a4308
main.gen.func17.1(0x1)
/home/kongdeyu/goprojs/src/github.com/blynn/nex.go:527 +0xf8 fp=0xc4301a43a8 sp=0xc4301a4358
main.gen.func17.1(0x2)
/home/kongdeyu/goprojs/src/github.com/blynn/nex.go:527 +0xf8 fp=0xc4301a43f8 sp=0xc4301a43a8
main.gen.func17.1(0x1)
I previously couldn't figure out how to match single line comments. Now solved, so I'm opening and closing this issue so it will get seen as reference for future users.
Match the single line comment:
/#[^\n]*/
{ // this matches a (#) pound char followed by any
// number of chars that aren't the (\n) newline!
s := yylex.Text()
lval.str = s[1:len(s)] // remove the leading #
log.Printf("lang: lexer: comment: `%s`", lval.str)
//return COMMENT // skip return to avoid parsing
}
The trick was the ^
negates the newline char, which matches all the junk you want, up until the newline. In the source this is the negate
property.
Sadly it seems as if all the regexp code was copy+pasted incompletely from the golang lib, and as a result, a lot of standard regexp things are missing such as :alpha:
for example.
Hope this hidden tutorial helped you!
hi,
could the following hack be added by nex?
printf '/NEX_END_OF_LEXER_STRUCT/i\np *Tacky\n.\nw\nq\n' | ed -s tacky.nn.go
also the nex file expects a newline at the end of the file... why?
I have the following regex:
/[a-z]([a-z0-9:]*[a-z0-9]+)?/
This is intended to match:
g
or
hey42
or
hey:there
but NOT:
blah:
trailing colons should not be allowed. Yet it does.
I need more extensive testing, but AFAIK this is pretty serious matching bug!
I also tested
/[a-z]+([a-z0-9:]*[a-z0-9]+)?/
Which has the same problem :/
Digging deeper... Help welcome =D
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.