Compile new binaries for hyperscan v4.6.0 about hyperscan-java HOT 10 CLOSED

I attempted to build 4.6.0 on MacOS but was unsuccessful. Failed at the link stage. Opened intel/hyperscan#69.

from hyperscan-java.

A friend suggested to try it with those cmake options (haven't had the opportunity to try those yet - will do that later and report back):

SET(CMAKE_XCODE_ATTRIBUTE_CLANG_CXX_LANGUAGE_STANDARD "c++11")
SET(CMAKE_XCODE_ATTRIBUTE_CLANG_CXX_LIBRARY "libc++")

from hyperscan-java.

The libraries are now automatically built using travis ci. A built can be triggered if a contributor changes the file https://github.com/cerebuild/hyperscan-java/blob/master/.hyperscan-version directly on the master branch (not via pull request).

from hyperscan-java.

@gliwka can we push out a new release with the new 4.6.0 libs?

from hyperscan-java.

@eliaslevy Done, v0.4.10 has been published.

I'll try to incorporate the changes in hyper scan (i.e. the newly introduced pattern matching in combination with Levenshtein distance) in the upcoming weeks. It mostly hinges on #27.

from hyperscan-java.

Danke.

from hyperscan-java.

Looks lik the v0.4.10 release breaks my job when executing in EC2, but not locally on my Mac. Reverting to v0.4.9 fixes the problem.

It is dying in Scanner. allocScratch with an java.lang.OutOfMemoryError, but that is probably wrong. Scanner.allocScratch doesn't check the return value of hs_check_alloc for a specific error, so the problem may be something else.

I don't have time to debug it at the moment. I'll try to find some time tomorrow.

from hyperscan-java.

@eliaslevy That would be much appreciated.

from hyperscan-java.

A bit more on the error I was seeing. I've reproduced it on my MBP and narrowed it down to a call I make to Expression.validate on start up to check the expressions. It only seems to trigger if there are sufficient concurrent threads, each of which run the same code. In my case I need at least four threads for it to trigger, and it won't trigger if I run the code in the debugger. The error goes away if I synchronize the call to Expression.validate.

The error report generated by the JVM shows two threads calling into hs_free_compile_error from validate with the top of the stack being:

Stack: [0x00007000055c4000,0x00007000056c4000],  sp=0x00007000056c2440,  free space=1017k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [libsystem_c.dylib+0x1b52]  strlen+0x12
C  [jna4845073464406839331.tmp+0x4073]  Java_com_sun_jna_Native_getStringBytes+0x23
j  com.sun.jna.Native.getStringBytes(Lcom/sun/jna/Pointer;JJ)[B+0
j  com.sun.jna.Native.getString(Lcom/sun/jna/Pointer;JLjava/lang/String;)Ljava/lang/String;+6
j  com.sun.jna.Pointer.getString(JLjava/lang/String;)Ljava/lang/String;+3
j  com.sun.jna.Structure.readField(Lcom/sun/jna/Structure$StructField;)Ljava/lang/Object;+134
j  com.sun.jna.Structure.read()V+100
j  com.sun.jna.Structure.autoRead()V+8
j  com.sun.jna.Function.invoke(Ljava/lang/reflect/Method;[Ljava/lang/Class;Ljava/lang/Class;[Ljava/lang/Object;Ljava/util/Map;)Ljava/lang/Object;+395
j  com.sun.jna.Library$Handler.invoke(Ljava/lang/Object;Ljava/lang/reflect/Method;[Ljava/lang/Object;)Ljava/lang/Object;+344
j  com.sun.proxy.$Proxy3.hs_free_compile_error(Lcom/gliwka/hyperscan/jna/CompileErrorStruct;)I+24
j  com.gliwka.hyperscan.wrapper.Expression.validate()Lcom/gliwka/hyperscan/wrapper/Expression$ValidationResult;+77

Looks like it is dying in the call to strlen from JNA's Java_com_sun_jna_Native_getStringBytes.

I am not familiar with JNA, but I believe the issue is that CompileErrorStruct is created by passing it a native struct that has been allocated by hyperscan. The call to hs_free_compile_error deallocates the the struct. But JNA's Structure has a feature enabled by default that will read the contents of the native struct into the Java Structure after every native call. So after calling hs_free_compile_error, JNA is trying to read the fields of the struct that hyperscan just freed.

In the single threaded case this is usually not a problem, as nothing will have modified that memory. But in a multi-threaded environment, the memory could be allocated and modified by another thread, leading to an error when the original thread tried to read the fields. Something changed between hyperscan 4.5.2 and 4.6.0 to make this more likely, but the problem was present before we upgraded to 4.6.0.

If I modify hyperscan-java to call errorStruct.setAutoRead(false); before every call to hs_free_compile_error, thus disabling read of the struct after the call to hs_free_compile_error, the error goes away.

from hyperscan-java.

Thank you very much for looking into this.

I'll open a issue for this and try to reproduce this using a simple test case.

from hyperscan-java.