The apg-py from ldthomas

apg-py's People

Contributors

Stargazers

Watchers

apg-py's Issues

Issues validating UTF-8

I'd like to use this project to validate some input using our ABNF, but it's having issues with any UTF-8 that's greater than a single byte. It looks like it's has something to do with the way you handle Unicode. Maybe you have a different approach than what I'm doing.

Part of the ABNF ensures that UTF-8 text is properly XML escaped.

xml-text        = 1*(unencoded / encoded)
unencoded       = %x09 /
                  %x0A /
                  %x0D /
                  %x20-21 /
                     ; omit 0x22 double quote
                  %x23-25 /
                     ; omit 0x26 ampersand
                     ; omit 0x27 single quote
                  %x28-3B /
                     ; omit 0x3C less than
                  %x3D /
                     ; omit 0x3E greater than
                  %x3F-7E /
                  UTF8-2 /
                  UTF8-3 /
                  UTF8-4
encoded         = %x26 (%s"quot" / %s"amp" / %s"apos" / %s"lt" / %s"gt") %x3B
UTF8-2          = %xC2-DF UTF8-tail
UTF8-3          = %xE0 %xA0-BF UTF8-tail / %xE1-EC 2(UTF8-tail) /
                  %xED %x80-9F UTF8-tail / %xEE-EF 2(UTF8-tail)
UTF8-4          = %xF0 %x90-BF 2(UTF8-tail) / %xF1-F3 3(UTF8-tail) /
                  %xF4 %x80-8F 2(UTF8-tail)
UTF8-tail       = %x80-BF

Validation script:

#!/usr/bin/env python3
from apg_py.lib import utilities as utils
from apg_py.lib.parser import Parser
from apg_py.api.api import Api
import sys
with open(sys.argv[1]) as f:
    grammar = f.read()
with open(sys.argv[2]) as f:
    input = f.read()
result = Parser(Api().generate(grammar)).parse(utils.string_to_tuple(input))
print(result.success)