mifmif / generex Goto Github PK
View Code? Open in Web Editor NEWA Java library for generating String from a regular expression.
A Java library for generating String from a regular expression.
Hi @mifmif !
The last commit on the master was more than 2 years ago. The number of issues and PR-s in this repository is growing and it seems the project isn't actively maintained.
Is this project still alive?
I run the following test and it failed 10/10 times.
for (int i = 0; i < 100; i++) {
// given
Generex generex = new Generex("([0-9]){5}");
// when
Stream<String> ids = Stream.generate(() -> generex.random()).limit(100);
// then
Assertions.assertThat(ids.distinct().count()).isEqualTo(100);
}
After drilling down in the code I saw that each Gerenex.random() call instantiates local instance of Random object with some seed based on current time. When calling random() in parallel we may then get Random instances with similar seed thus generating similar pseudo-random sequences.
Is it possible to fix or overcome this problem? Would you like us to come up with a patch?
And thanks very much for this library - very useful!
Hello @mifmif ,
I noticed one issue when generating some RegExps. If we try this:
Generex generex = new Generex("(.{1,6})( *, *(.{1,6})){0,5}");
System.out.println(generex.random());
it takes about 1 second to finish. But, if we try this RegEx:
Generex generex = new Generex("(.{1,6})( *, *(.{1,6})){0,12}");
System.out.println(generex.random());
it takes about 40 seconds.
Finally, the code
Generex generex = new Generex("(.{1,6})( *, *(.{1,6})){0,15}");
System.out.println(generex.random());
took a whopping 3 minutes and generated one character as a result.
Now I'm perfectly aware that users can write all sorts of RegExps. It's probably impossible to make code which efficiently generates them all. Yet I hope there's a possibility that for some cases generation might be faster.
If the regex expression is a range of numbers, the random() method of Generex class does not ever return the last item in the interval.
E.g.:
new Generex("[0-1]").random(); // will never return value 1, only 0;
I have the following Generex:
return new Generex("(?=.*[a-z])(?=.*[A-Z])(?=.*[0-9$@!%*#?&])[A-Za-z0-9$@!%*#?&]{6,10}") .random();
My result is this:
?=㉲h⫗]"?�mm�fF遛A 葊3ꦠu!o?쇄�x8rp㥌䜂廘>N#溦꼲NZcM�x熚�d?'੫
jWu㞠�䰙â[w? 鰭;f?騐灝峱^栀㡓곖pY��hPx
趪�^쌒U�deOb7xl>f&# ⪫p풵vRw�/s?6<b?焹蠢Mg?乭s?=A:�F@�좽?8NR@FV&2U5ꇖ⧉柭垚6+펊/N?=(;!*嗼!V:!^"�!ߚ"'4=";$!1b*筍/�'�0D*&v"@c
%[�?<,!BCy@#e닳:??";뽴.'5?k-%C("!""!"$"!t@"坓+:,匓!;"8㡡�:&"25#!"%Z�$(<$礯(!�긿;>#"7��7>&!;"&,+<"@cq%�)"!$#)+ᗗ'+$8@Z)"�%'""<!�#)@./%_�6,*^?)3"@=(""�!N$$�1,僰)@&!!a&
Is looks like Positive Lookahead isn't supported. Can this be implemented?
like "color=(?:red|green|blue)"
you end up generating ?:red instead of just red
@ val test = new Generex("color=(?:red|green|blue)")
test: Generex = com.mifmif.common.regex.Generex@459d81f1
@ test.random
res36: String = "color=green"
@ test.random
res37: String = "color=?:red" <<<---- BUG (should be color=red)
@ test.random
res38: String = "color=blue"
It seems that ^
and $
are used as normal characters but actually they assert position at start or end of the string.
As a result the following code
new Generex("^A$").random()
will generate ^A$
, instead of just A
.
Hello,
When I try the generex with the regex "[a-z0-9A-Z]{16}" and try to call getFirstMatch it takes too long time.
Can you advise on that.
Any chance of uploading this to Maven Central?
Generex g = new Generex("(नि|😃|丈)+")
for(int i=0; i<100; i++) {
System.out.println(g.random(69, 70))
}
Output:
ि丈निनि丈नि丈निनि😃नि丈丈😃😃😃😃नि丈निनिनि😃😃丈नि丈丈丈😃नि丈😃नि丈नि丈😃नि丈😃丈
丈丈丈丈😃丈निनिनि丈丈丈नि丈नि丈丈नि😃😃😃😃😃😃नि😃丈丈😃नि😃丈निनि丈नि😃नि丈नि😃😃丈
丈丈निनि😃नि😃😃😃丈丈नि丈丈丈नि😃😃😃😃😃丈😃丈😃丈😃😃😃丈नि丈丈नि丈丈निनि丈😃😃丈😃
😃निनि丈丈😃नि丈😃丈丈丈丈😃😃निनि丈丈丈😃निनि😃😃丈丈丈निनिनि丈😃😃丈नि😃丈丈丈丈丈😃丈丈नि
丈😃नि丈नि丈丈निनि丈निनि😃😃😃😃नि丈😃丈丈丈丈😃निनि丈नि😃丈丈😃निनि😃निनिनि丈😃नि丈
丈नि😃丈丈नि😃丈丈😃😃丈निनि丈😃丈丈丈😃नि😃नि丈😃😃नि😃😃निनि😃丈नि😃निनि丈😃😃नि
नि😃丈丈😃😃नि丈नि丈丈😃丈निनिनिनिनिनि😃निनिनि丈नि丈निनिनि丈😃丈丈😃😃丈丈😃丈नि丈😃丈
😃😃😃丈丈नि丈नि😃नि丈丈丈निनि😃丈丈😃😃😃😃丈निनिनि😃丈丈😃丈丈निनि丈नि丈😃丈😃नि😃😃
java.lang.StackOverflowError
at java.base/java.util.TimSort.sort(TimSort.java:220)
at java.base/java.util.Arrays.sort(Arrays.java:1441)
at dk.brics.automaton.State.getSortedTransitionArray(State.java:137)
at dk.brics.automaton.State.getSortedTransitions(State.java:147)
at com.mifmif.common.regex.Generex.prepareRandom(Generex.java:340)
at com.mifmif.common.regex.Generex.prepareRandom(Generex.java:366)
at com.mifmif.common.regex.Generex.prepareRandom(Generex.java:366)
at com.mifmif.common.regex.Generex.prepareRandom(Generex.java:366)
Would be possible to have the tags of the (latest) releases (e.g. 1.0.1, 1.0.2) available in Maven Central?
Thanks for providing this library!
It would be nice if it was possible to pass a seed, similar to new Random(long seed)
. This would allow to repeat the exact generated results, e.g. to repeat test with the same data.
Maybe this is already possible and I just overlooked it?
Thanks,
Pascal
[INFO]
[INFO] <<< javancss-maven-plugin:2.0:check (default) < :report @ generex <<<
[INFO]
[INFO] --- javancss-maven-plugin:2.0:check (default) @ generex ---
[WARNING] Your code has 1 method(s) with a ccn greater than 10
[WARNING] com.mifmif.common.regex.Generex.prepareRandom(String,State,int,int)
If you use a regex such as
^[a-z]$
You will get results of
^a$
^b$
^c$
...
^z$
The anchors likely should be stripped before processing as you would expect to just get all the lowercase letters of the alphabet instead.
Currently it is hard to integrate Generex and https://github.com/pholser/junit-quickcheck.
Here's "example" implementation: https://github.com/pholser/junit-quickcheck/blob/61e751ceb55cb767d7ef150959df0244543fcd64/examples/src/test/java/com/pholser/junit/quickcheck/examples/text/Structured.java#L51
There are two problems there:
Generex(pattern)
initialization is performed on each generation. In other words, Generex
parses the pattern and builds regex automata for each generate call which is waste.
Cloning Random
is kind of hard. In practice, Generex#prepareRandom
does not need the full power of Random
, so it might make sense to add a thin interface RandomSource
with methods like nextInt(int)
and nextFloat()
.
Then end-users could instantiate Generex
just once, and pick random values as needed (providing it with RandomSource
instance).
Currently you need to set min/max length with Java.
Isn't it possible to use the min/max length specified by the regular expression?
Example Use-Case:
I need to generate a strong password which expects atleast 1 small-case letter, 1 Capital letter, 1 digit, 1 special character and the length should be at least 8 characters.
(?=^.{8,}$)(?=.*\d)(?=.*[a-z])(?=.*[A-Z])(?=.*[!@#$%^&*()_+}{":;'?/>.<,])(?!.*\s).*$
It is not possible to do it this way:
Generex generex = new Generex("(?=^.{8,}$)(?=.*\d)(?=.*[a-z])(?=.*[A-Z])(?=.*[!@#$%^&*()_+}{\":;'?/>.<,])(?!.*\s).*$");
String password = generex.random();
If I want to use regex as (a|)
then it doesn't work, although it is proper expression.
new Generex("(a|)").random()
generate error and expecting )
at the end.
Also (a)?
always generate a
and never empty string.
Hi. I want to generate string that matchs regex \\w
, but generated string contains obviously too many Underscores. Does java.util.Random
's unevenness cause this?
The following is sample code. (I'm Sorry, I use Kotlin, not Java)
fun main(args: Array<String>) {
val value = Generex("\\w{1000}").random()
val counter = mutableMapOf<Char, Int>()
for (v in value.toList()) {
val c = counter.get(v)
when (c) {
null -> counter.put(v, 1)
else -> counter.put(v, c + 1)
}
}
counter.toList().sortedByDescending { it.second }.take(10).forEach {
println("char: ${it.first}, count: ${it.second}")
}
}
// sample result
// char: _, count: 225
// char: 3, count: 38
// char: 8, count: 30
// char: 4, count: 29
// char: 9, count: 28
// char: 5, count: 27
// char: 1, count: 27
// char: 7, count: 24
// char: 2, count: 24
// char: 0, count: 22
Hey guys, I'm facing this issue:
Caused by: java.lang.ClassNotFoundException: dk.brics.automaton.RegExp
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:520)
... 86 more
This happen when I try to use the Genrex in simplest way possible.
I have already added automaton
dependency to my project, but this keep happening.
Any clue about this bug?
Thanks guys
Thanks again for adding the seed. 👍
It works fine for most cases, but fails for when a regular expression with a {min, max} quantifier is used.
Example:
@Test
public void testSeedWithMinMaxQuantifier() {
long seed = -5106534569952410475L;
String pattern = "[A-Z]{1,10}";
Generex firstGenerex = new Generex(pattern);
firstGenerex.setSeed(seed);
String firstValue = firstGenerex.random();
Generex secondGenerex = new Generex(pattern);
secondGenerex.setSeed(seed);
String secondValue = secondGenerex.random();
Assert.assertEquals(firstValue, secondValue);
}
It works for exact quantifiers.
Generex currently replaces predefined character classes while wrapping them in square brackets: \d
becomes [0-9]
However, if the \d
is already in a character class expression then [\d]
becomes [[0-9]]
, which is compilable correctly by java.util.regex.Pattern
but not by dk.brics.automaton.Automaton
used by Generex.
Simple regex replacement is apparently not enough, it looks like contextual replacement is needed (tracking if \d
is inside [..] char by char, tracking already escaped \
)
Input: [\d]
(Java String literal "[\\d]"
)
Expected output:
[0-9]
0
1
2
3
4
5
6
7
8
9
Actual output:
[[0-9]]
0]
1]
2]
3]
4]
5]
6]
7]
8]
9]
[]
Got java.lang.StackOverflowError
if I try to generate random string with regexp that contains unescaped "@" symbol:
Generex rnd = new Generex("[a-z]@mail.com");
System.out.println(rnd.random());
//specifying the boundaries doesn't help
Generex rnd = new Generex("[a-z]@mail.com");
System.out.println(rnd.random(20, 50));
while next line is totally ok for Java Pattern:
Pattern.compile("[a-z]@mail.com").matcher("[email protected]").matches()
Generex works fine with next patterns:"[a-z]\\@mail.com"
, "[a-z][@]mail.com"
.
Stack:
java.lang.StackOverflowError
at java.util.TimSort.countRunAndMakeAscending(TimSort.java:355)
at java.util.TimSort.sort(TimSort.java:220)
at java.util.Arrays.sort(Arrays.java:1438)
at dk.brics.automaton.State.getSortedTransitionArray(Unknown Source)
at dk.brics.automaton.State.getSortedTransitions(Unknown Source)
at com.mifmif.common.regex.Generex.prepareRandom(Generex.java:282)
at com.mifmif.common.regex.Generex.prepareRandom(Generex.java:303)
at com.mifmif.common.regex.Generex.prepareRandom(Generex.java:303)
at com.mifmif.common.regex.Generex.prepareRandom(Generex.java:303)
at com.mifmif.common.regex.Generex.prepareRandom(Generex.java:303)
.....
Generex generex = new Generex("^(.?=.{8,})(?=.[\d])(?=.[\W]).$");
String password = generex.random(20, 20);
Result: 494 chars string.
Sample regex: [-_a-zA-Z0-9]
.
random(1, 70)
produces the following strings:
CY
6--Y-
0I-_-R
f
O
--A77_
-
_-V
05F-43-4-
f-2-QF_w
d_-
_0
_P
762_
_
_i_t__
Y
-b
-1i
5_J
X_S
_t-
9
p8-_yqHp4-
-
H_u
_
HW
_J-S-
Almost all the strings are short which is not always good.
Current code picks the value with probability of 66%
That makes it quite impossible to produce 70 chars for random(minLength=1, maxLength=70)
because Generex
would try to stop at each string with probability of 66%.
What if Generex had a mode where it stops as soon as the length exceeds given minLength
?
Then random(minLength, maxLength)
would be pick a random V1 between min...max, and then generate a string and stop as soon as the length exceeds V1
.
Currently Generex uses automaton
v1.11-8 which is more than 10 years old.
The current version of automaton
is: 1.12-4 (Jan 03, 2022).
It would be nice to provide possiblity to escape some parts of regex from generation without need to escape each special character one by one.
For example I'd like to use
minion_\d{3}\[email protected]\E
instead of
minion_\d{3}\@gru\.evil
Here is a workaround code to transform the former to latter:
private static String requote(String regex) {
final Pattern PATTERN_QUOTED = Pattern.compile("\\\\Q(.*?)\\\\E");
// http://stackoverflow.com/questions/399078/what-special-characters-must-be-escaped-in-regular-expressions
// adding "@" prevents StackOverflowError inside generex: https://github.com/mifmif/Generex/issues/21
final Pattern PATTERN_SPECIALS = Pattern.compile("[.^$*+?(){|\\[\\\\@]");
StringBuilder sb = new StringBuilder(regex);
Matcher matcher = PATTERN_QUOTED.matcher(sb);
while (matcher.find()) {
sb.replace(matcher.start(), matcher.end(), PATTERN_SPECIALS.matcher(matcher.group(1)).replaceAll("\\\\$0"));
//matcher.reset();
}
return sb.toString();
}
Using java 8.
Simply call random() brings Exception in Generex line 278 with random(nextInt(0)). Since the randomTransition.getMax and randomTransition.getMin are the same (value "i")
Workarounding this. Also the rootNode is not present before asking a value so asking the size before asking the value leads to NPE
static{
try {
buildMethod = Generex.class.getDeclaredMethod("buildRootNode");
buildMethod.setAccessible(true);
} catch (NoSuchMethodException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (SecurityException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
Generex generex = new Generex(regex);
try {
buildMethod.invoke(generex);
} catch (IllegalAccessException | IllegalArgumentException | InvocationTargetException e) {
e.printStackTrace();
}
int index = new Random(generex.matchedStringsSize()).nextInt();
String generatedValue = generex.getMatchedString(index);
Thank you so much for making this tool!
When testing it, I ran into this case that causes a stack overflow:
String pattern = "^[a-zA-Z\s]*$";
Generex generex = new Generex(pattern);
String firstMatch = generex.getFirstMatch();
And when that code is run I get this exception:
Exception in thread "main" java.lang.StackOverflowError
at java.base/java.util.HashMap$HashIterator.<init>(HashMap.java:1475)
at java.base/java.util.HashMap$KeyIterator.<init>(HashMap.java:1514)
at java.base/java.util.HashMap$KeySet.iterator(HashMap.java:912)
at java.base/java.util.HashSet.iterator(HashSet.java:173)
at java.base/java.util.AbstractCollection.toArray(AbstractCollection.java:184)
at dk.brics.automaton.State.getSortedTransitionArray(Unknown Source)
at dk.brics.automaton.State.getSortedTransitions(Unknown Source)
at com.mifmif.common.regex.Generex.prepareTransactionNodes(Generex.java:265)
Could this be fixed?
new Generex("[A]").random(10, 10);
produce : "A"
new Generex("[A]{10}").random(5, 5);
produces : "AAAAAAAAAA"
The random(int)
does not seams to work as weel.
Tested version 1.0.2
and 1.0.1
I'm interested in using this lib but it has no licence information.
What am I allowed to to with?
Used version: 1.0.2
Example code to reproduce:
SystemPreferenceList out = new SystemPreferenceList();
Generex generex = new Generex("(^$)|(^[a-zA-Z0-9_-]+(\\.[a-zA-Z0-9_-]+)*$)");
System.out.println(generex.random());
Generated string: ^_XC-5o-K-.X.6-0$
This string doesnt match the regex
Regex online to validate:
https://regex101.com/r/XelPbS/1
In Your example we have:
// it print 0a 0b 0c 0e 0ee 0e 0e 0f 0fe 0f 0f 0g 0ge 0g 0g 1a 1b 1c 1e
// 1ee 1e 1e 1f 1fe 1f 1f 1g 1ge 1g 1g 2a 2b 2c 2e 2ee 2e 2e 2f 2fe 2f 2f 2g
// 2ge 2g 2g 3a 3b 3c 3e 3ee 3e 3e 3f 3fe 3f 3f 3g 3ge 3g 3g 1ee
bold - duplicates;
When I run sample code from how-to I get:
0a 0b 0c 0e 0ee 0e 0e 0f 0fe 0f 0f 0g 0ge 0g 0g 1a 1b 1c 1e
1ee 1e 1e 1f 1fe 1f 1f 1g 1ge 1g 1g 2a 2b 2c 2e 2ee 2e 2e 2f 2fe 2f 2f 2g
2ge 2g 2g 3a 3b 3c 3e 3ee 3e 3e 3f 3fe 3f 3f 3g 3ge 3g 3g
but when You loop over list from getAllMatchedStrings()
// Generate all String that matches the given Regex.
List<String> matchedStrs = generex.getAllMatchedStrings();
for (String string : matchedStrs) {
System.out.print(string + " ");
}
You will see a little bit different result
0a 0b 0c 0e 0ee 0ef 0eg 0f 0fe 0ff 0fg 0g 0ge 0gf 0gg 1a 1b 1c 1e
1ee 1ef 1eg 1f 1fe 1ff 1fg 1g 1ge 1gf 1gg 2a 2b 2c 2e 2ee 2ef 2eg 2f 2fe 2ff 2fg 2g
2ge 2gf 2gg 3a 3b 3c 3e 3ee 3ef 3eg 3f 3fe 3ff 3fg 3g 3ge 3gf 3gg
bold - correct values
I've run into a problem deploying an app (built with Maven) using generex to Tomcat. Tomcat spits out the following -
INFO: Deploying web application archive /var/lib/tomcat/webapps/ROOT.war
Nov 04, 2014 1:44:50 PM org.apache.catalina.util.ExtensionValidator validateManifestResources
INFO: ExtensionValidator[][generex-0.0.1.jar]: Required extension [automaton] not found.
Nov 04, 2014 1:44:50 PM org.apache.catalina.util.ExtensionValidator validateManifestResources
I think it's down to these lines in MANIFEST.MF
Extension-List: automaton
automaton-Extension-Name: automaton
automaton-Implementation-Version: 1.11-8
Class-Path: dk/brics/automaton/automaton/1.11-8/automaton-1.11-8.jar
which is not where Tomcat is going to find automaton (in this case it's in ROOT/WEB-INF/lib/automaton-1.11-8.jar)
Sometimes I'm getting a recursion stackoverflow exception when I generate strings for:
new Generex("\\S+_a+/").random()
You can reproduce this by running the code above 100k times or so.
I tried playing around with this:
\\S
with [^ \t\n\f\r]
or even [^ \n]
- FAIL\\S
with \\W
or \\D
- OK/
at the end of the expression - OK_
- OKI can't make sense of it. I'm guessing it's something to do with some UTF-8 character (though why would it work well with \\W
?
For complex patterns like:
((?!000)(?!666)(?:[0-6]\\d{2}|7[0-2][0-9]|73[0-3]|7[5-6][0-9]|77[0-2]))-((?!00)\\d{2})-((?!0000)\\d{4})
the ?!
character seem to be interpreted literally, it would be nice to have support for all of the supported characters that Java support (https://docs.oracle.com/javase/7/docs/api/java/util/regex/Pattern.html)
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.