rwxrob / scan Goto Github PK

This was preventing the idiomatic common usage of any scanner in a loop:

s := scan.New(in)
for s.Scan() {
  // check
}

This scanner is already heavily influenced by the infinite-memory (PEG)
approach, so might as well move it into pegn where that expectation will
be more welcome and understood.

Allowing scan.R users to have write access to the internal cursor is a
bad idea because they will attempt to manipulate it breaking any
automatic line detection and other magic that can happen in the Scan
method. The cost in performance to force a call to Scan for every rune
is minimal in comparison.

Now that we have parsing in the mix we need to be able to revert both
the buffer cursor position and the current state of the node tree that
has been parsed.

z.P parsing expressions are not allowed within either lookahead
expression, need to be sure people don't try it, at the slight cost of
performance

Structs for "is" package give much more specific error messages when
used incorrectly.

Parsing is suffering from the lack of any sort of transactional commit
method to the s.Tree.Root. While it is possible to jump back to a
position in the buffer during recursive calls to scan.X it is not
currently possible to revert the state of the s.Tree.Root to that moment
in time which produces additions to the tree that do not belong that
were successful during other scans. z.X currently has a broken fix for
this by taking a reference to s.Tree.Root and restoring it, but that
solution will not work going forward for everything.

It's quite possible there isn't a simple solution for this problem given
the nature of the Node tree since once the internal references of any
node are updated from Add or Append, removing them is difficult even
with Cut available because we don't know which Node to CutParsing is
suffering from the lack of any sort of transactional commit method to
the s.Tree.Root. While it is possible to jump back to a position in the
buffer during recursive calls to scan.X it is not currently possible to
revert the state of the s.Tree.Root to that moment in time which
produces additions to the tree that do not belong that were successful
during other scans. z.X currently has a broken fix for this by taking a
reference to s.Tree.Root and restoring it, but that solution will not
work going forward for everything.

It's quite possible there isn't a simple solution for this problem given
the nature of the Node tree since once the internal references of any
node are updated from Add or Append, removing them is difficult even
with Cut available because we don't know which Node to Cut.

The inclusion of the last scanned rune is only convenient when working
inside for loops, but it wrong and error-prone in other situations
such as from within a functional parser where each function is
responsible to advance itself, if and only if, it has a peek ahead
match.

Really we need something like the following:

• Peek - only stuff ahead of the last rune scanned (s.P)
• PeekMatch - regular expression against stuff ahead (s.P)
• Is - string including the last rune scanned (s.LP)
• Match - regular expression against last scanned (s.LP)