With ONIG_SYNTAX_DEFAULT pattern '(?s)ABC' fails with 'undefined group option'. about oniguruma HOT 12 CLOSED

@spershin

Is the syntax you used "Syntax Java" in JONI ?
Please make sure that the syntax is not something other than "Syntax Java".

JONI : options of "Syntax Java"
https://github.com/jruby/joni/blob/3750dcbf58cb5cfad76c1a3ae11cc2be65ba7fe1/src/org/joni/Syntax.java#L624-L652

I did a test to add "ONIG_SYN_OP2_QMARK_LT_NAMED_GROUP" to oniguruma's "ONIG_SYNTAX_JAVA".
The test was successful.

I think it would be good to make the syntax options of JONI and oniguruma similar.

Would it be possible to also support ?s along with ?m?

This is a question that should be answered by the Owner so I cannot answer it, sorry.

scan and rewrite the pattern before using it with Oni

This is very difficult and not recommended.

Please try "ONIG_SYNTAX_PERL_NG" of oniguruma.
(?s) and "naming capture groups" are available.

from oniguruma.

Please use (?m) .

• ONIG_SYNTAX_DEFAULT == ONIG_SYNTAX_ONIGURUMA

• (?s) and (?m)

• ONIG_SYNTAX_ONIGURUMA : " ^ matches after newline, $ matches before newline " is valid.

from oniguruma.

@tonco-miyazawa

Thank you for replying.

I would gladly follow your advice if it was possible.
Our situation is the following:

We are migrating from Presto Java to Presto Native (C++) with the latter using Velox library.
Presto Java uses JONI (Java port of Oni) for regex implementation.
Velox uses RE2, which is missing a lot of crucial functionality such as backtracking.

So, we are looking/investigating at adopting Oni instead.
Why?
Because we have hundreds of customers with established workloads and there is no way we can change their queries - too many queries, too many customers, too much negative impact if we break them.
So, ideally we would like to have functionality and features in Oni to be as close to JONI as possible.
This gives us a lot of headache, honestly. :)

Would it be possible to also support ?s along with ?m?
I don't know the code and never did anything around regex much up until now, so cannot really tell how hard it can be.

We could, potentially, scan and rewrite the pattern before using it with Oni, however in such cases how can we sure that we can safely rewrite any ?s sequence into ?m? Also, what it is a combined sequence, like ?isq or something like that? Such rewrite becomes complicated and error-prone without deep knowledge of the patterns and syntax.

Thank you!

from oniguruma.

Sorry, I'm an end user so I don't know the details.

I think you can add an option 'ONIG_SYN_OP2_QMARK_LT_NAMED_GROUP' to ONIG_SYNTAX_JAVA.

I think #296 is very helpful.
Please see fd33056. This is the fix for #296.

This adds ONIG_SYN_OP2_QMARK_GROUP_EFFECT to OnigSyntaxEmacs.

These options are documented in SYNTAX.md .
https://github.com/kkos/oniguruma/blob/master/doc/SYNTAX.md

Or do you want to use ONIG_SYNTAX_DEFAULT?
In this case, I think a fix to the regular expression parser is needed.

from oniguruma.

@tonco-miyazawa

This seems useful. Thank you.
I haven't read it though thoroughly yet as I'm continuing to investigate other possible issues of using Oni in Velox before spending considerable effort at fixing any of them.

To confirm (or disprove) that I understand it right:

Does it mean that we can build our own OnigSyntaxType structure that can encompass multiple features that we need?
For example, in this case we could take ONIG_SYNTAX_DEFAULT as the base and then add a bit so ?s is activated?

Or, as you suggested, if ONIG_SYNTAX_JAVA is good for us as well and is only missing ?<digit> support, then we can add ONIG_SYN_OP2_QMARK_LT_NAMED_GROUP to it or to our custom syntax based on the ONIG_SYNTAX_JAVA.

Thank you!

from oniguruma.

@tonco-miyazawa

We use Java syntax, it seems:

regex = new Regex(pattern.getBytes(), 0, pattern.length(), Option.DEFAULT, NonStrictUTF8Encoding.INSTANCE, Syntax.Java);

However, we are most likely be using a forked repo and not the master JONI.
Trying to confirm this now.
Update: Presto Java uses https://github.com/airlift/joni fork of JONI.

The path to the JONI jar is like that:

.../.m2/repository/io/airlift/joni/2.1.5.3/joni-2.1.5.3.jar!/io/airlift/joni/Regex.class

JONI repo does not have such version in the list of tags.

I agree the syntax should be the same, if possible, however, already there are some serious differences, like the named capture group.

I'll try ONIG_SYNTAX_PERL_NG syntax too, thank you!

from oniguruma.

I ported the options from JONI-2.1.5.3 's "Syntax.Java" to oniguruma's "OnigSyntaxJava" .

/*  JONI-2.1.5.3 's options for oniguruma-6.9.9 */
OnigSyntaxType OnigSyntaxJava = {
  (( SYN_GNU_REGEX_OP | ONIG_SYN_OP_QMARK_NON_GREEDY |
     ONIG_SYN_OP_ESC_CONTROL_CHARS | ONIG_SYN_OP_ESC_C_CONTROL |
     ONIG_SYN_OP_ESC_OCTAL3 | ONIG_SYN_OP_ESC_X_HEX2 |
     ONIG_SYN_OP_ESC_X_BRACE_HEX8 )
   & ~ONIG_SYN_OP_ESC_LTGT_WORD_BEGIN_END )
  , ( ONIG_SYN_OP2_ESC_CAPITAL_Q_QUOTE | ONIG_SYN_OP2_QMARK_GROUP_EFFECT |
      ONIG_SYN_OP2_OPTION_PERL | ONIG_SYN_OP2_PLUS_POSSESSIVE_REPEAT |
      ONIG_SYN_OP2_PLUS_POSSESSIVE_INTERVAL | ONIG_SYN_OP2_CCLASS_SET_OP |
      ONIG_SYN_OP2_QMARK_LT_NAMED_GROUP | ONIG_SYN_OP2_ESC_K_NAMED_BACKREF |
      ONIG_SYN_OP2_ESC_V_VTAB | ONIG_SYN_OP2_ESC_U_HEX4 |
      ONIG_SYN_OP2_ESC_P_BRACE_CHAR_PROPERTY )
  , ( SYN_GNU_REGEX_BV | ONIG_SYN_ISOLATED_OPTION_CONTINUE_BRANCH |
      ONIG_SYN_DIFFERENT_LEN_ALT_LOOK_BEHIND )
  , ONIG_OPTION_SINGLELINE
  ,
  {
      (OnigCodePoint )'\\'                       /* esc */
    , (OnigCodePoint )ONIG_INEFFECTIVE_META_CHAR /* anychar '.'  */
    , (OnigCodePoint )ONIG_INEFFECTIVE_META_CHAR /* anytime '*'  */
    , (OnigCodePoint )ONIG_INEFFECTIVE_META_CHAR /* zero or one time '?' */
    , (OnigCodePoint )ONIG_INEFFECTIVE_META_CHAR /* one or more time '+' */
    , (OnigCodePoint )ONIG_INEFFECTIVE_META_CHAR /* anychar anytime */
  }
};

[added option]
ONIG_SYN_OP_ESC_X_BRACE_HEX8                /* \x{7HHHHHHH} */
ONIG_SYN_OP2_QMARK_LT_NAMED_GROUP     /* (?<name>...) */
ONIG_SYN_OP2_ESC_K_NAMED_BACKREF        /* \k<name> */

[removed opiton]
ONIG_SYN_VARIABLE_LEN_LOOK_BEHIND        /* (?<=a+|..) */

[ This does not exist in oniguruma ]
OP2_ESC_P_CHAR_CHAR_PROPERTY              /* \pX, \PX */

[This does not exist in JONI-2.1.5.3]
ONIG_SYN_ISOLATED_OPTION_CONTINUE_BRANCH      /* ..(?i)...|... */

SYNTAX.md
https://github.com/kkos/oniguruma/blob/master/doc/SYNTAX.md

I tested using this syntax and it was successful.
So there is no need to try "ONIG_SYNTAX_PERL_NG" syntax.

Does it mean that we can build our own OnigSyntaxType structure that can encompass multiple features that we need?

For example, in this case we could take ONIG_SYNTAX_DEFAULT as the base and then add a bit so ?s is activated?

Or, as you suggested, if ONIG_SYNTAX_JAVA is good for us as well and is only missing ?<digit> support, then we can add ONIG_SYN_OP2_QMARK_LT_NAMED_GROUP to it or to our custom syntax based on the ONIG_SYNTAX_JAVA.

Sorry, I don't have the knowledge to answer these questions.
I will let you know if I find out anything.

from oniguruma.

@spershin

Sorry for keeping you waiting.

Does it mean that we can build our own OnigSyntaxType structure that can encompass multiple features that we need?

For example, in this case we could take ONIG_SYNTAX_DEFAULT as the base and then add a bit so ?s is activated?

Or, as you suggested, if ONIG_SYNTAX_JAVA is good for us as well and is only missing ?<digit> support, then we can add ONIG_SYN_OP2_QMARK_LT_NAMED_GROUP to it or to our custom syntax based on the ONIG_SYNTAX_JAVA.

The answer to all of these questions is "yes".
I have actually done this and confirmed that it works.

If you want to write code based on "ONIG_SYNTAX_JAVA", I recommend using the JONI-2.1.5.3 syntax above as a starting point. And please use it as your own syntax.

If you want to use (?s) in "ONIG_SYNTAX_DEFAULT" , please replace the following option:

Please exchange "ONIG_SYN_OP2_OPTION_ONIGURUMA" for "ONIG_SYN_OP2_OPTION_PERL" .
Below are descriptions of these options:

SYNTAX.md : ONIG_SYN_OP2_OPTION_ONIGURUMA
https://github.com/kkos/oniguruma/blob/master/doc/SYNTAX.md#30-onig_syn_op2_option_oniguruma-enable-options-imxwsdpy-and--imxwdsp

SYNTAX.md : ONIG_SYN_OP2_OPTION_PERL
https://github.com/kkos/oniguruma/blob/master/doc/SYNTAX.md#2-onig_syn_op2_option_perl-enable-options-imsx-and--imsx

However, this swap makes (?W) , (?D) , (?S) , (?P) invalid.

If you want to change only (?s) and (?m) , please change the code below.

'm' : (?m) , (?-m)
's' : (?s) , (?-s)
'-' negates the effect.

Below is an explanation of "ONIG_OPTION_SINGLELINE" and "ONIG_OPTION_MULTILINE" .

But such a change may be confusing to users. If I were a user, I would like to be able to choose between the JONI and oniguruma syntax. And oniguruma can do that.

If you want to replace the default syntax, please change the following:

If you want to create your own syntax, please refer to the following.
"OnigSyntaxJava" is registered like this, you can register your own syntax in the same way.

oniguruma.h

onigposix.h

regsyntax.c

from oniguruma.

@tonco-miyazawa
Thank you very much for so much detailed information!
This is VERY helpful.
I am going to try out the options you suggested (especially the one you took from JONI) and see if we are lucky enough to have most of the options working like they do in JONI.

I'm sorry that I wasn't able to reply here earlier - was busy implementing the remaining regex functions based on Oni and testing them, collecting issues I can find.

from oniguruma.

@spershin

I'm glad I could be of help to you. Don't worry about the reply.

I think the next problem will be that \pX and \PX cannot be used in oniguruma.
This is too difficult for me.

I will close this thread as it has served its purpose. Thank you.

EDIT: I can't seem to close this thread.

from oniguruma.

@tonco-miyazawa
Using your pointers on how the syntax structures work I managed to make (?s) work in ONIG_SYNTAX_DEFAULT.
Now, I'm yet to do more testing to see if there are possible side effects of this, but unit tests pass.

The code:

static OnigSyntaxType myOnigSyntax = *ONIG_SYNTAX_DEFAULT;
myOnigSyntax.op2 |= ONIG_SYN_OP2_OPTION_PERL;
gOnigSyntax = &myOnigSyntax;

where gOnigSyntax is
void* gOnigSyntax = ONIG_SYNTAX_DEFAULT;
and is used in onig_new().

Somewhat relevant:
I'm also adding

  myOnigSyntax.options |= ONIG_OPTION_SINGLELINE;

To treat string as a single line, which ONIG_SYNTAX_DEFAULT does not do and it does not match JONI in this behavior.

from oniguruma.

@spershin

unit tests pass.

Congratulations! Good job!

myOnigSyntax.options |= ONIG_OPTION_SINGLELINE;
To treat string as a single line, which ONIG_SYNTAX_DEFAULT does not do
and it does not match JONI in this behavior.

I have tested this and found that this option works correctly.
This is a very confusing subject and it's easy to get confused.
Because we are confused by settings like:

"ONIG_OPTION_SINGLELINE" and "ONIG_OPTION_MULTILINE",

"ONIG_SYN_OP2_OPTION_ONIGURUMA" and "ONIG_SYN_OP2_OPTION_PERL"

"(?s)" and "(?m)"

So I created a test to check if the option is valid or Invalid.
Below is the test for "/test/test_syntax.c".
If the option is Invalid, these tests will return "FAIL" .

  ////////////  Test to see if an option is valid /////////////
  Syntax = ONIG_SYNTAX_DEFAULT;

  x2("$", "123\n456", 7, 7);           // 7-7 == ONIG_OPTION_SINGLELINE is valid.
  
  x2("\\A.*\\z", "123\n456", 0, 7);    // 0-7 == ONIG_OPTION_MULTILINE is valid.
  ///////////////////////////////////////////////////////////////

Below is the command for testing above.
Please copy "/test/test_syntax.c" to "Documents" folder. ( This is the same as issue290. )

$ gcc ~/Documents/test_syntax.c  -L/usr/local/lib -lonig
$ ./a.out

Also, please check your onig_new() setting.

Also, see below for the values ​​of "ONIG_OPTION_DEFAULT" .

You can also check "ONIG_OPTION_SINGLELINE" by using "ONIG_OPTION_NEGATE_SINGLELINE" in onig_new() .
Try using this option to see if it changes your results.

I hope your work goes well. Thank you.

from oniguruma.