the problem with compiling tensorflow about nuitka HOT 12 OPEN

I also tried to repeat this on another PC with a similar system + tried to use the dev version - it gives this error everywhere

from nuitka.

I suspect this is tensorflow / protobuf in a newer version, I am trying to reproduce it now, but generally tensorflow was working previously, so this is a regression. Also I wonder what tensorflow/core/protobuf/replay_log.proto is, maybe it's a new data file, and that is causing this.

from nuitka.

Good afternoon, thanks for the reply!

Could you tell me the versions of python/nuitka/tensorflow on which everything worked?

The strangest thing is that I cannot find this file on the PC, I tried searching through
sudo find / -name replay_log.proto

from nuitka.

I don't keep track of it like that. I saw the proto files in grep when using --list-package-data=tensorflow, I didn't check the precise file, but it seems there were really many of these.

from nuitka.

also, i can compile without some dependencies
python -m nuitka --standalone --nofollow-import-to=tensorflow --nofollow-import-to=keras --nofollow-import-to=google main.py

but if i run ./main.bin
it's show me error ModuleNotFoundError: No module named 'tensorflow'
How can I make the binary have a dependency on tensorflow from my environment?

from nuitka.

You cannot add to standalone @mookor for that you would have to use accelerated mode, which has of course portability (to other systems) problems.

from nuitka.

@kayhayen I think nuitka may add a new compile option to skip compile 3rd None Pure-Python packages automatically. For the libs such as tensorflow, the lib owner have compiled them to pyd/so files. Nuitka should focus on the Pure-Python packages.

from nuitka.

In the plugins, you can decide the compilation mode and, if at all, on a per-module basis; however, if you put "bytecode" there, it will not necessarily work at all. One typical upcoming example will be that the source code will be needed for JITs or whatever, and I don't really want to find out myself why then things don't work and support that. By the time the bytecode is created in a good way and is re-used cached, then it can also just generate the C code and be good too, since that is not really the problem usually.

The real solution is improved scalability of Nuitka, which will be my focus in the summer. With a 10x improvement in Python compile time, and maybe caching of its result one day, we will be good. Also with 3.13, we might try and compile with the no-GIL variant eventually in parallel or we use multiprocessing, but that will require caching to work, because it's going to be how the result is communicated back.

Anyway, it's a nice idea, and if you can write a plugin that makes it work for you, and you feel that is time spent well, go on. For me, it's a diversion from my main goals.

from nuitka.

So, I reproduced it and found that the string in question lives in MiniTensorflowOpt.dist/tensorflow/python/_pywrap_tensorflow_internal.so. Of course, the question now is why, if at all, it is loaded twice, this may just misdiagnosis of that code. My googling found only cases of unsupported platforms.

from nuitka.

So, this is probably the issue:

grep: MiniTensorflowOpt.dist/tensorflow/python/_pywrap_tensorflow_internal.so: binary file matches
grep: MiniTensorflowOpt.dist/_pywrap_tensorflow_internal.so: binary file matches

This is because of those:

<included_extension name="_pywrap_tensorflow_internal.so" dest_path="tensorflow/python/_pywrap_tensorflow_internal.so" source_path="~/.local/lib/python3.9/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so" package="tensorflow.python" ignored="no" reason="used extension module" />
<included_dll name="_pywrap_tensorflow_internal.so" dest_path="_pywrap_tensorflow_internal.so" source_path="~/.local/lib/python3.9/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so" package="tensorflow.compiler.mlir.quantization.tensorflow.calibrator" ignored="no" reason="Used by 'tensorflow/compiler/mlir/quantization/tensorflow/calibrator/pywrap_calibration.so'" />

So, it's found as both an extension module and a DLL, which is very strange.

from nuitka.

One thing I noticed and dislike, is this:

        libtensorflow_framework.so.2 => /home/hayen/repos/Py2C/MiniTensorflowOpt.dist/tensorflow/compiler/mlir/quantization/tensorflow/calibrator/../../../../../../libtensorflow_framework.so.2 (0x00007f0d950df000)

It seems, that it's carried over from the original PyPI package

        libtensorflow_framework.so.2 => /home/hayen/.local/lib/python3.9/site-packages/tensorflow/compiler/mlir/quantization/tensorflow/calibrator/../../../../../libtensorflow_framework.so.2 

Of course, that is very ugly and may, in fact, cause issues if those directories do not all exist, although I think that at least Linux is forgiving there, so that's not our issue.

What happens, though, it seems, is that there is a dependency as a DLL on an extension module.

        _pywrap_tensorflow_internal.so => /home/hayen/.local/lib/python3.9/site-packages/tensorflow/compiler/mlir/quantization/tensorflow/calibrator/../../../../../python/_pywrap_tensorflow_internal.so 

Nuitka places DLLs near the main binary by default to make it most likely to be found by everything.

from nuitka.

So, I guess the solution is to remove DLLs from detection, which are just extension modules and hope for the best, namely that they are found.

from nuitka.