Recursive callbacks cause a deadlock about pemja HOT 12 CLOSED

Hi @ljeub-pometry , do you plan to take use of pemja in Raphtory?

from pemja.

@ljeub-pometry Thanks for your usage. We don't have step 3 such use cases before, I will try your case to see where the problem is.

from pemja.

@ljeub-pometry I have figured out the cause of the error. The reason is that the PythonInterperter you use in the main thread will hold the GIL of Python during the calling process. When the calling process has not finished, you try to call to create a new PythonInterperter. During the creation process, will go to lock GIL, which leads to deadlock.

from pemja.

I'll do some experimenting and see how to meet this need

from pemja.

Thank you for looking into it. Ideally, this situation shouldn't require more than one python thread at all, i.e., when calling a jvm method, it probably should release the GIL while python is waiting for the result, such that the interpreter is free to execute the step 3 callback above.

from pemja.

@ljeub-pometry Yes. You are right. Before calling Java Function, we need to release GIL just like numpy do. I will fix this issue asap.

from pemja.

I have created PR #15 to fix it

from pemja.

I have merged the fix into master branch and release-0.2 branch

from pemja.

Thank you for the quick fix! I just tested it and this does indeed fixes the deadlock!

from pemja.

@ljeub-pometry Thanks for your verification.

from pemja.

We are still in the early stages of adding python support to Raphtory but it does look like pemja is the most viable option for executing python code in the backend in terms of performance.

from pemja.

Ok. If you have any needs and ideas for pemja, welcome to exchange. According to the plan, in the next version, pemja will provide support on numpy and windows

from pemja.