Comments (12)
Is the syntax you used "Syntax Java" in JONI ?
Please make sure that the syntax is not something other than "Syntax Java".
JONI : options of "Syntax Java"
https://github.com/jruby/joni/blob/3750dcbf58cb5cfad76c1a3ae11cc2be65ba7fe1/src/org/joni/Syntax.java#L624-L652
I did a test to add "ONIG_SYN_OP2_QMARK_LT_NAMED_GROUP" to oniguruma's "ONIG_SYNTAX_JAVA".
The test was successful.
I think it would be good to make the syntax options of JONI and oniguruma similar.
Would it be possible to also support ?s along with ?m?
This is a question that should be answered by the Owner so I cannot answer it, sorry.
scan and rewrite the pattern before using it with Oni
This is very difficult and not recommended.
Please try "ONIG_SYNTAX_PERL_NG" of oniguruma.
(?s) and "naming capture groups" are available.
from oniguruma.
Please use (?m)
.
- ONIG_SYNTAX_DEFAULT == ONIG_SYNTAX_ONIGURUMA
Line 155 in 1b37592
(?s)
and(?m)
Lines 520 to 528 in 1b37592
- ONIG_SYNTAX_ONIGURUMA : " ^ matches after newline, $ matches before newline " is valid.
Lines 172 to 183 in 1b37592
from oniguruma.
Thank you for replying.
I would gladly follow your advice if it was possible.
Our situation is the following:
We are migrating from Presto Java to Presto Native (C++) with the latter using Velox library.
Presto Java uses JONI (Java port of Oni) for regex implementation.
Velox uses RE2, which is missing a lot of crucial functionality such as backtracking.
So, we are looking/investigating at adopting Oni instead.
Why?
Because we have hundreds of customers with established workloads and there is no way we can change their queries - too many queries, too many customers, too much negative impact if we break them.
So, ideally we would like to have functionality and features in Oni to be as close to JONI as possible.
This gives us a lot of headache, honestly. :)
Would it be possible to also support ?s
along with ?m
?
I don't know the code and never did anything around regex much up until now, so cannot really tell how hard it can be.
We could, potentially, scan and rewrite the pattern before using it with Oni, however in such cases how can we sure that we can safely rewrite any ?s
sequence into ?m
? Also, what it is a combined sequence, like ?isq
or something like that? Such rewrite becomes complicated and error-prone without deep knowledge of the patterns and syntax.
Thank you!
from oniguruma.
Sorry, I'm an end user so I don't know the details.
I think you can add an option 'ONIG_SYN_OP2_QMARK_LT_NAMED_GROUP
' to ONIG_SYNTAX_JAVA
.
Lines 146 to 169 in fd33056
I think #296 is very helpful.
Please see fd33056. This is the fix for #296.
This adds ONIG_SYN_OP2_QMARK_GROUP_EFFECT
to OnigSyntaxEmacs
.
These options are documented in SYNTAX.md .
https://github.com/kkos/oniguruma/blob/master/doc/SYNTAX.md
Or do you want to use ONIG_SYNTAX_DEFAULT?
In this case, I think a fix to the regular expression parser is needed.
from oniguruma.
This seems useful. Thank you.
I haven't read it though thoroughly yet as I'm continuing to investigate other possible issues of using Oni in Velox before spending considerable effort at fixing any of them.
To confirm (or disprove) that I understand it right:
Does it mean that we can build our own OnigSyntaxType
structure that can encompass multiple features that we need?
For example, in this case we could take ONIG_SYNTAX_DEFAULT
as the base and then add a bit so ?s
is activated?
Or, as you suggested, if ONIG_SYNTAX_JAVA
is good for us as well and is only missing ?<digit>
support, then we can add ONIG_SYN_OP2_QMARK_LT_NAMED_GROUP
to it or to our custom syntax based on the ONIG_SYNTAX_JAVA
.
Thank you!
from oniguruma.
We use Java syntax, it seems:
regex = new Regex(pattern.getBytes(), 0, pattern.length(), Option.DEFAULT, NonStrictUTF8Encoding.INSTANCE, Syntax.Java);
However, we are most likely be using a forked repo and not the master JONI.
Trying to confirm this now.
Update: Presto Java uses https://github.com/airlift/joni fork of JONI.
The path to the JONI jar is like that:
.../.m2/repository/io/airlift/joni/2.1.5.3/joni-2.1.5.3.jar!/io/airlift/joni/Regex.class
JONI repo does not have such version in the list of tags.
I agree the syntax should be the same, if possible, however, already there are some serious differences, like the named capture group.
I'll try ONIG_SYNTAX_PERL_NG syntax too, thank you!
from oniguruma.
I ported the options from JONI-2.1.5.3 's "Syntax.Java
" to oniguruma's "OnigSyntaxJava
" .
/* JONI-2.1.5.3 's options for oniguruma-6.9.9 */
OnigSyntaxType OnigSyntaxJava = {
(( SYN_GNU_REGEX_OP | ONIG_SYN_OP_QMARK_NON_GREEDY |
ONIG_SYN_OP_ESC_CONTROL_CHARS | ONIG_SYN_OP_ESC_C_CONTROL |
ONIG_SYN_OP_ESC_OCTAL3 | ONIG_SYN_OP_ESC_X_HEX2 |
ONIG_SYN_OP_ESC_X_BRACE_HEX8 )
& ~ONIG_SYN_OP_ESC_LTGT_WORD_BEGIN_END )
, ( ONIG_SYN_OP2_ESC_CAPITAL_Q_QUOTE | ONIG_SYN_OP2_QMARK_GROUP_EFFECT |
ONIG_SYN_OP2_OPTION_PERL | ONIG_SYN_OP2_PLUS_POSSESSIVE_REPEAT |
ONIG_SYN_OP2_PLUS_POSSESSIVE_INTERVAL | ONIG_SYN_OP2_CCLASS_SET_OP |
ONIG_SYN_OP2_QMARK_LT_NAMED_GROUP | ONIG_SYN_OP2_ESC_K_NAMED_BACKREF |
ONIG_SYN_OP2_ESC_V_VTAB | ONIG_SYN_OP2_ESC_U_HEX4 |
ONIG_SYN_OP2_ESC_P_BRACE_CHAR_PROPERTY )
, ( SYN_GNU_REGEX_BV | ONIG_SYN_ISOLATED_OPTION_CONTINUE_BRANCH |
ONIG_SYN_DIFFERENT_LEN_ALT_LOOK_BEHIND )
, ONIG_OPTION_SINGLELINE
,
{
(OnigCodePoint )'\\' /* esc */
, (OnigCodePoint )ONIG_INEFFECTIVE_META_CHAR /* anychar '.' */
, (OnigCodePoint )ONIG_INEFFECTIVE_META_CHAR /* anytime '*' */
, (OnigCodePoint )ONIG_INEFFECTIVE_META_CHAR /* zero or one time '?' */
, (OnigCodePoint )ONIG_INEFFECTIVE_META_CHAR /* one or more time '+' */
, (OnigCodePoint )ONIG_INEFFECTIVE_META_CHAR /* anychar anytime */
}
};
[added option]
ONIG_SYN_OP_ESC_X_BRACE_HEX8 /* \x{7HHHHHHH} */
ONIG_SYN_OP2_QMARK_LT_NAMED_GROUP /* (?<name>...) */
ONIG_SYN_OP2_ESC_K_NAMED_BACKREF /* \k<name> */
[removed opiton]
ONIG_SYN_VARIABLE_LEN_LOOK_BEHIND /* (?<=a+|..) */
[ This does not exist in oniguruma ]
OP2_ESC_P_CHAR_CHAR_PROPERTY /* \pX, \PX */
[This does not exist in JONI-2.1.5.3]
ONIG_SYN_ISOLATED_OPTION_CONTINUE_BRANCH /* ..(?i)...|... */
SYNTAX.md
https://github.com/kkos/oniguruma/blob/master/doc/SYNTAX.md
I tested using this syntax and it was successful.
So there is no need to try "ONIG_SYNTAX_PERL_NG
" syntax.
Does it mean that we can build our own
OnigSyntaxType
structure that can encompass multiple features that we need?
For example, in this case we could take
ONIG_SYNTAX_DEFAULT
as the base and then add a bit so?s
is activated?
Or, as you suggested, if
ONIG_SYNTAX_JAVA
is good for us as well and is only missing?<digit>
support, then we can addONIG_SYN_OP2_QMARK_LT_NAMED_GROUP
to it or to our custom syntax based on theONIG_SYNTAX_JAVA
.
Sorry, I don't have the knowledge to answer these questions.
I will let you know if I find out anything.
from oniguruma.
Sorry for keeping you waiting.
Does it mean that we can build our own
OnigSyntaxType
structure that can encompass multiple features that we need?
For example, in this case we could take
ONIG_SYNTAX_DEFAULT
as the base and then add a bit so ?s is activated?
Or, as you suggested, if
ONIG_SYNTAX_JAVA
is good for us as well and is only missing?<digit>
support, then we can addONIG_SYN_OP2_QMARK_LT_NAMED_GROUP
to it or to our custom syntax based on theONIG_SYNTAX_JAVA
.
The answer to all of these questions is "yes".
I have actually done this and confirmed that it works.
If you want to write code based on "ONIG_SYNTAX_JAVA
", I recommend using the JONI-2.1.5.3 syntax above as a starting point. And please use it as your own syntax.
If you want to use (?s)
in "ONIG_SYNTAX_DEFAULT
" , please replace the following option:
Line 79 in f2f9c69
Please exchange "ONIG_SYN_OP2_OPTION_ONIGURUMA
" for "ONIG_SYN_OP2_OPTION_PERL
" .
Below are descriptions of these options:
SYNTAX.md : ONIG_SYN_OP2_OPTION_ONIGURUMA
https://github.com/kkos/oniguruma/blob/master/doc/SYNTAX.md#30-onig_syn_op2_option_oniguruma-enable-options-imxwsdpy-and--imxwdsp
SYNTAX.md : ONIG_SYN_OP2_OPTION_PERL
https://github.com/kkos/oniguruma/blob/master/doc/SYNTAX.md#2-onig_syn_op2_option_perl-enable-options-imsx-and--imsx
However, this swap makes (?W)
, (?D)
, (?S)
, (?P)
invalid.
If you want to change only (?s)
and (?m)
, please change the code below.
Lines 8281 to 8302 in f2f9c69
'm'
: (?m)
, (?-m)
's'
: (?s)
, (?-s)
'-
' negates the effect.
Below is an explanation of "ONIG_OPTION_SINGLELINE
" and "ONIG_OPTION_MULTILINE
" .
Lines 79 to 80 in f2f9c69
But such a change may be confusing to users. If I were a user, I would like to be able to choose between the JONI and oniguruma syntax. And oniguruma can do that.
If you want to replace the default syntax, please change the following:
Line 164 in f2f9c69
Line 281 in f2f9c69
If you want to create your own syntax, please refer to the following.
"OnigSyntaxJava
" is registered like this, you can register your own syntax in the same way.
oniguruma.h
Line 427 in f2f9c69
Line 441 in f2f9c69
onigposix.h
Line 134 in f2f9c69
Line 145 in f2f9c69
regsyntax.c
Line 146 in f2f9c69
from oniguruma.
@tonco-miyazawa
Thank you very much for so much detailed information!
This is VERY helpful.
I am going to try out the options you suggested (especially the one you took from JONI) and see if we are lucky enough to have most of the options working like they do in JONI.
I'm sorry that I wasn't able to reply here earlier - was busy implementing the remaining regex functions based on Oni and testing them, collecting issues I can find.
from oniguruma.
I'm glad I could be of help to you. Don't worry about the reply.
I think the next problem will be that \pX
and \PX
cannot be used in oniguruma.
This is too difficult for me.
I will close this thread as it has served its purpose. Thank you.
EDIT: I can't seem to close this thread.
from oniguruma.
@tonco-miyazawa
Using your pointers on how the syntax structures work I managed to make (?s)
work in ONIG_SYNTAX_DEFAULT
.
Now, I'm yet to do more testing to see if there are possible side effects of this, but unit tests pass.
The code:
static OnigSyntaxType myOnigSyntax = *ONIG_SYNTAX_DEFAULT;
myOnigSyntax.op2 |= ONIG_SYN_OP2_OPTION_PERL;
gOnigSyntax = &myOnigSyntax;
where gOnigSyntax
is
void* gOnigSyntax = ONIG_SYNTAX_DEFAULT;
and is used in onig_new()
.
Somewhat relevant:
I'm also adding
myOnigSyntax.options |= ONIG_OPTION_SINGLELINE;
To treat string as a single line, which ONIG_SYNTAX_DEFAULT
does not do and it does not match JONI in this behavior.
from oniguruma.
unit tests pass.
Congratulations! Good job!
myOnigSyntax.options |= ONIG_OPTION_SINGLELINE;
To treat string as a single line, whichONIG_SYNTAX_DEFAULT
does not do
and it does not match JONI in this behavior.
I have tested this and found that this option works correctly.
This is a very confusing subject and it's easy to get confused.
Because we are confused by settings like:
"ONIG_OPTION_SINGLELINE
" and "ONIG_OPTION_MULTILINE
",
"ONIG_SYN_OP2_OPTION_ONIGURUMA
" and "ONIG_SYN_OP2_OPTION_PERL
"
"(?s)
" and "(?m)
"
So I created a test to check if the option is valid or Invalid.
Below is the test for "/test/test_syntax.c
".
If the option is Invalid, these tests will return "FAIL
" .
//////////// Test to see if an option is valid /////////////
Syntax = ONIG_SYNTAX_DEFAULT;
x2("$", "123\n456", 7, 7); // 7-7 == ONIG_OPTION_SINGLELINE is valid.
x2("\\A.*\\z", "123\n456", 0, 7); // 0-7 == ONIG_OPTION_MULTILINE is valid.
///////////////////////////////////////////////////////////////
Below is the command for testing above.
Please copy "/test/test_syntax.c
" to "Documents
" folder. ( This is the same as issue290. )
$ gcc ~/Documents/test_syntax.c -L/usr/local/lib -lonig
$ ./a.out
Also, please check your onig_new()
setting.
Lines 76 to 104 in bcfbab2
Also, see below for the values โโof "ONIG_OPTION_DEFAULT
" .
Line 372 in bcfbab2
You can also check "ONIG_OPTION_SINGLELINE
" by using "ONIG_OPTION_NEGATE_SINGLELINE
" in onig_new()
.
Try using this option to see if it changes your results.
Line 85 in bcfbab2
I hope your work goes well. Thank you.
from oniguruma.
Related Issues (20)
- Dynamic library generated failed to support "(?-i)" HOT 3
- oniguruma not able to installed on PPC64le architecture HOT 3
- `.{0,99}` and `.*` behave differently on short input HOT 2
- mbc_enc_len(const UChar* p) to be improved HOT 4
- Does Oniguruma support POSIX collating symbols or equivalence classes? HOT 1
- New version?๐ค HOT 1
- Does "retry-limit-in-match over" mean NOT MATCH? HOT 23
- ONIG_SYN_CONTEXT_INDEP_REPEAT_OPS not working for ^* pattern HOT 1
- Literal escaped braces HOT 2
- what is escape code for `.` (literal dot)? HOT 1
- onig_match() can return wrong results with lookbehind at the end of the string HOT 2
- ONIG_SYNTAX_EMACS doesn't support "shy groups"
- Error 'unmatched range specifier in char-class' with hyphen in character class. HOT 1
- typos HOT 1
- :alpha: character class matches "non-letter" unicode characters. HOT 4
- Sample pattern and content to trigger error in onig_search? HOT 2
- Invalid utf8 sequence matches the pattern while it probably should not. HOT 1
- Test files ( 'e' function )
- How do I statically link the library to my project without installing it on my system
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. ๐๐๐
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google โค๏ธ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from oniguruma.