Unable to build UDFs about nebula HOT 8 CLOSED

@jjsimps Have you tried pulling the vesoft/nebula-dev:centos7 image again? Locally I get the following output from your above command:

$ docker run --rm -ti vesoft/nebula-dev:centos7 bash                                                                                                                                                       130 ↵
[root@d0f41c1786bc ~]# nm -D /lib64/libstdc++.so.6 | grep CXXABI
0000000000000000 A CXXABI_1.3
0000000000000000 A CXXABI_1.3.1
0000000000000000 A CXXABI_1.3.10
0000000000000000 A CXXABI_1.3.11
0000000000000000 A CXXABI_1.3.12
0000000000000000 A CXXABI_1.3.2
0000000000000000 A CXXABI_1.3.3
0000000000000000 A CXXABI_1.3.4
0000000000000000 A CXXABI_1.3.5
0000000000000000 A CXXABI_1.3.6
0000000000000000 A CXXABI_1.3.7
0000000000000000 A CXXABI_1.3.8
0000000000000000 A CXXABI_1.3.9
0000000000000000 A CXXABI_FLOAT128
0000000000000000 A CXXABI_TM_1
[root@d0f41c1786bc ~]#

I guess @jjsimps is about to run graphd with udf folder attached in production, thus leveraging our graphd docker image rather than the dev one.

I checked that the implementation of the code does not execute dlopen every time. All udfs are loaded only at the beginning and then saved in the function manager, so the problem mentioned above should not exist.

from nebula.

This is exactly the situation of how UDF in C++ works, i.e. the ABI between the UDF and the host program must be compatible. ABI might be broken between different GCC versions, especially the ones before and after GCC 5.

There are indeed some techniques to address such issues like symbol versioning, but for the old ABI before C++11 it's too tedious, difficult and unreliable to maintain. So we will not support such cases.

As for the compatibility for the newer versions, a serious UDF implementation maybe should try best to handle this. But it is still a tough work to do. Keeping the building environments exactly the same is the easy and reliable way.

from nebula.

@jjsimps

The basic idea is that you build the UDF itself with toolchain of the same version as graphd. Besides, since graphd links libstdc++ statically(not proper for the UDF scenario) but the dependency on libstdc++ by UDF cannot be eliminated, you have to ship the UDF library with libstdc++.so(maybe with a RPATH set to locate it).

Personally I never reviewed this UDF framework implementation. But it smells bad to me after a fast browsing. It does invoke dlopen on each user function call. Maybe you could try to refactor the code to do symbol lookup in the init function and release the function reference in another destroy function.

As for the serious one, we don't have a plan on this so far. Actually we have been working on a huge refactoring which includes user defined function and procedure. But the roadmap to release is not determined yet.

from nebula.

Awesome, thank you.

I tried the rpath method + shipping libstdc++.6.so and that seems to work.

I'll see about refactoring some of the code to bring it back up to a bit of a better level.

from nebula.

Thanks for the input everyone!

@wey-gu is exactly correct -- the UDFs are built using the nebula-dev image which has the builtools and CXXABI_1.3.9. But the prod image is the regular centos7 image which does not have it.

I also tried your suggestion of building using ubuntu2004 image and that seems to work. But that does require me to build my own graphd image using ubuntu2004 (not the worst, but not ideal of course). I expect the same would be fine if I build the graphd image using the nebula-dev:centos7 as the base image instead of just centos7.

As for calling dlopen, please see here. As you can see, the lambda, when called, will call dlopen(), create the UDF, call it, destroy, then call dlclose().

I do realize that this feature was kinda put in as a stealth feature (not fully documented, but referenced in release notes). I was looking forward to it as it allows me to run a bit more complex logic that would either otherwise manifest as a very complex query (with branch, bitwise operators, etc), or have to be taken back to the application level.

What would you say is the current outlook/roadmap for the serious UDF support? Is there anywhere I can find out this info? In the meantime, I think that the current implementation is 85% there for most of my use-cases. I would like to know the proper solution for dealing with the ABI (hopefully by changing the prod base image to something that has it), and also a fix for the performance issue when calling the UDF (I noticed query times jump from 10s of ms to 100s of ms when calling a cusom UDF which seems absurd).

Thank you :)

from nebula.

• missing ABI dep, I used to do so with the binary package being installed from vanilla Ubuntu docker image like: https://github.com/wey-gu/nebulagraph-ai/blob/main/docker/Dockerfile-ngai-graphd , could you go with this way first?

• I also noticed that It calls dlopen() every time the function is called, which seems like it would cause a huge performance overhead. Why not cache the function pointer?

  @yixinglu could we decouple the dlopen() per non-first-time UD function call with some cache mechanism?

from nebula.

@jjsimps Have you tried pulling the vesoft/nebula-dev:centos7 image again? Locally I get the following output from your above command:

$ docker run --rm -ti vesoft/nebula-dev:centos7 bash                                                                                                                                                       130 ↵
[root@d0f41c1786bc ~]# nm -D /lib64/libstdc++.so.6 | grep CXXABI
0000000000000000 A CXXABI_1.3
0000000000000000 A CXXABI_1.3.1
0000000000000000 A CXXABI_1.3.10
0000000000000000 A CXXABI_1.3.11
0000000000000000 A CXXABI_1.3.12
0000000000000000 A CXXABI_1.3.2
0000000000000000 A CXXABI_1.3.3
0000000000000000 A CXXABI_1.3.4
0000000000000000 A CXXABI_1.3.5
0000000000000000 A CXXABI_1.3.6
0000000000000000 A CXXABI_1.3.7
0000000000000000 A CXXABI_1.3.8
0000000000000000 A CXXABI_1.3.9
0000000000000000 A CXXABI_FLOAT128
0000000000000000 A CXXABI_TM_1
[root@d0f41c1786bc ~]#

from nebula.

@jjsimps Have you tried pulling the vesoft/nebula-dev:centos7 image again? Locally I get the following output from your above command:

$ docker run --rm -ti vesoft/nebula-dev:centos7 bash                                                                                                                                                       130 ↵
[root@d0f41c1786bc ~]# nm -D /lib64/libstdc++.so.6 | grep CXXABI
0000000000000000 A CXXABI_1.3
0000000000000000 A CXXABI_1.3.1
0000000000000000 A CXXABI_1.3.10
0000000000000000 A CXXABI_1.3.11
0000000000000000 A CXXABI_1.3.12
0000000000000000 A CXXABI_1.3.2
0000000000000000 A CXXABI_1.3.3
0000000000000000 A CXXABI_1.3.4
0000000000000000 A CXXABI_1.3.5
0000000000000000 A CXXABI_1.3.6
0000000000000000 A CXXABI_1.3.7
0000000000000000 A CXXABI_1.3.8
0000000000000000 A CXXABI_1.3.9
0000000000000000 A CXXABI_FLOAT128
0000000000000000 A CXXABI_TM_1
[root@d0f41c1786bc ~]#

I guess @jjsimps is about to run graphd with udf folder attached in production, thus leveraging our graphd docker image rather than the dev one.

from nebula.