Comments (19)
Ok, could reproduce it now, issue is with the compilation linker step which for some reason cannot link to the libmkl_rt.so
file even though it exists, and the problem is not limited to python.
Will investigate about it to see how to correct it.
from poismf.
Thanks for the bug report. Could you post the full output of:
python setup.py build_ext --inplace --force
python setup.py install --force
from poismf.
For sudo python3 setup.py build_ext --inplace --force
running build_ext
Installation: Using BLAS library found in:
/home/sarah/.local/lib/libmkl_rt.so
cythoning poismf/poismf_c_wrapper.pyx to poismf/poismf_c_wrapper.c
/home/sarah/.local/lib/python3.6/site-packages/Cython/Compiler/Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /home/sarah/poismf/poismf/poismf_c_wrapper.pyx
tree = Parsing.p_module(s, pxd, full_module_name)
building 'poismf.poismf_c_wrapper' extension
C compiler: x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC
creating build/temp.linux-x86_64-3.6
creating build/temp.linux-x86_64-3.6/poismf
creating build/temp.linux-x86_64-3.6/src
compile options: '-D_FOR_PYTHON -DHAS_MKL -I/home/sarah/.local/lib/python3.6/site-packages/numpy/core/include -I/home/sarah/.local/include -I/home/sarah/.local/lib/python3.6/site-packages/findblas/ -I/usr/include/python3.6m -c'
extra options: '-O2 -fopenmp -march=native -std=c99'
x86_64-linux-gnu-gcc: poismf/poismf_c_wrapper.c
x86_64-linux-gnu-gcc: src/nonnegcg.c
src/nonnegcg.c: In function ‘minimize_nonneg_cg’:
src/nonnegcg.c:271:72: warning: ‘grad_prev’ may be used uninitialized in this function [-Wmaybe-uninitialized]
beta += ( x[i] <= 0 )? 0 : grad_curr[i] * (grad_curr[i] - grad_prev[i]);
~~~~~~~~~^~~
src/nonnegcg.c:270:62: warning: ‘direction_prev’ may be used uninitialized in this function [-Wmaybe-uninitialized]
theta += ( x[i] <= 0 )? 0 : grad_curr[i] * direction_prev[i];
~~~~~~~~~~~~~~^~~
src/nonnegcg.c:274:9: warning: ‘grad_prev_norm_sq’ may be used uninitialized in this function [-Wmaybe-uninitialized]
beta /= grad_prev_norm_sq;
~~~~~^~~~~~~~~~~~~~~~~~~~
In file included from /home/sarah/.local/lib/python3.6/site-packages/numpy/core/include/numpy/ndarraytypes.h:1830:0,
from /home/sarah/.local/lib/python3.6/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,
from /home/sarah/.local/lib/python3.6/site-packages/numpy/core/include/numpy/arrayobject.h:4,
from poismf/poismf_c_wrapper.c:598:
/home/sarah/.local/lib/python3.6/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
#warning "Using deprecated NumPy API, disable it with " \
^~~~~~~
x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-Bsymbolic-functions -Wl,-z,relro -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.6/poismf/poismf_c_wrapper.o build/temp.linux-x86_64-3.6/src/nonnegcg.o -o /home/sarah/poismf/poismf/poismf_c_wrapper.cpython-36m-x86_64-linux-gnu.so -fopenmp -L/home/sarah/.local/lib -l:libmkl_rt.so
For sudo python3 setup.py install --force
running install
running bdist_egg
running egg_info
writing poismf.egg-info/PKG-INFO
writing dependency_links to poismf.egg-info/dependency_links.txt
writing requirements to poismf.egg-info/requires.txt
writing top-level names to poismf.egg-info/top_level.txt
reading manifest file 'poismf.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'poismf.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
creating build/lib.linux-x86_64-3.6
creating build/lib.linux-x86_64-3.6/poismf
copying poismf/__init__.py -> build/lib.linux-x86_64-3.6/poismf
running build_ext
Installation: Using BLAS library found in:
/home/sarah/.local/lib/libmkl_rt.so
skipping 'poismf/poismf_c_wrapper.c' Cython extension (up-to-date)
building 'poismf.poismf_c_wrapper' extension
C compiler: x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC
compile options: '-D_FOR_PYTHON -DHAS_MKL -I/home/sarah/.local/lib/python3.6/site-packages/numpy/core/include -I/home/sarah/.local/include -I/home/sarah/.local/lib/python3.6/site-packages/findblas/ -I/usr/include/python3.6m -c'
extra options: '-O2 -fopenmp -march=native -std=c99'
x86_64-linux-gnu-gcc: poismf/poismf_c_wrapper.c
x86_64-linux-gnu-gcc: src/nonnegcg.c
src/nonnegcg.c: In function ‘minimize_nonneg_cg’:
src/nonnegcg.c:271:72: warning: ‘grad_prev’ may be used uninitialized in this function [-Wmaybe-uninitialized]
beta += ( x[i] <= 0 )? 0 : grad_curr[i] * (grad_curr[i] - grad_prev[i]);
~~~~~~~~~^~~
src/nonnegcg.c:270:62: warning: ‘direction_prev’ may be used uninitialized in this function [-Wmaybe-uninitialized]
theta += ( x[i] <= 0 )? 0 : grad_curr[i] * direction_prev[i];
~~~~~~~~~~~~~~^~~
src/nonnegcg.c:274:9: warning: ‘grad_prev_norm_sq’ may be used uninitialized in this function [-Wmaybe-uninitialized]
beta /= grad_prev_norm_sq;
~~~~~^~~~~~~~~~~~~~~~~~~~
In file included from /home/sarah/.local/lib/python3.6/site-packages/numpy/core/include/numpy/ndarraytypes.h:1830:0,
from /home/sarah/.local/lib/python3.6/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,
from /home/sarah/.local/lib/python3.6/site-packages/numpy/core/include/numpy/arrayobject.h:4,
from poismf/poismf_c_wrapper.c:598:
/home/sarah/.local/lib/python3.6/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
#warning "Using deprecated NumPy API, disable it with " \
^~~~~~~
x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-Bsymbolic-functions -Wl,-z,relro -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.6/poismf/poismf_c_wrapper.o build/temp.linux-x86_64-3.6/src/nonnegcg.o -o build/lib.linux-x86_64-3.6/poismf/poismf_c_wrapper.cpython-36m-x86_64-linux-gnu.so -fopenmp -L/home/sarah/.local/lib -l:libmkl_rt.so
creating build/bdist.linux-x86_64/egg
creating build/bdist.linux-x86_64/egg/poismf
copying build/lib.linux-x86_64-3.6/poismf/__init__.py -> build/bdist.linux-x86_64/egg/poismf
copying build/lib.linux-x86_64-3.6/poismf/poismf_c_wrapper.cpython-36m-x86_64-linux-gnu.so -> build/bdist.linux-x86_64/egg/poismf
byte-compiling build/bdist.linux-x86_64/egg/poismf/__init__.py to __init__.cpython-36.pyc
creating stub loader for poismf/poismf_c_wrapper.cpython-36m-x86_64-linux-gnu.so
byte-compiling build/bdist.linux-x86_64/egg/poismf/poismf_c_wrapper.py to poismf_c_wrapper.cpython-36.pyc
creating build/bdist.linux-x86_64/egg/EGG-INFO
copying poismf.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO
copying poismf.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying poismf.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying poismf.egg-info/requires.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying poismf.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
writing build/bdist.linux-x86_64/egg/EGG-INFO/native_libs.txt
zip_safe flag not set; analyzing archive contents...
poismf.__pycache__.poismf_c_wrapper.cpython-36: module references __file__
creating 'dist/poismf-0.1.4-py3.6-linux-x86_64.egg' and adding 'build/bdist.linux-x86_64/egg' to it
removing 'build/bdist.linux-x86_64/egg' (and everything under it)
Processing poismf-0.1.4-py3.6-linux-x86_64.egg
removing '/usr/local/lib/python3.6/dist-packages/poismf-0.1.4-py3.6-linux-x86_64.egg' (and everything under it)
creating /usr/local/lib/python3.6/dist-packages/poismf-0.1.4-py3.6-linux-x86_64.egg
Extracting poismf-0.1.4-py3.6-linux-x86_64.egg to /usr/local/lib/python3.6/dist-packages
poismf 0.1.4 is already the active version in easy-install.pth
Installed /usr/local/lib/python3.6/dist-packages/poismf-0.1.4-py3.6-linux-x86_64.egg
Processing dependencies for poismf==0.1.4
Searching for findblas==0.1.3.4
Best match: findblas 0.1.3.4
Adding findblas 0.1.3.4 to easy-install.pth file
Using /home/sarah/.local/lib/python3.6/site-packages
Searching for Cython==0.29.14
Best match: Cython 0.29.14
Adding Cython 0.29.14 to easy-install.pth file
Installing cygdb script to /usr/local/bin
Installing cython script to /usr/local/bin
Installing cythonize script to /usr/local/bin
Using /home/sarah/.local/lib/python3.6/site-packages
Searching for pandas==0.25.3
Best match: pandas 0.25.3
Adding pandas 0.25.3 to easy-install.pth file
Using /home/sarah/.local/lib/python3.6/site-packages
Searching for numpy==1.17.4
Best match: numpy 1.17.4
Adding numpy 1.17.4 to easy-install.pth file
Installing f2py script to /usr/local/bin
Installing f2py3 script to /usr/local/bin
Installing f2py3.6 script to /usr/local/bin
Using /home/sarah/.local/lib/python3.6/site-packages
Searching for pytz==2019.3
Best match: pytz 2019.3
Adding pytz 2019.3 to easy-install.pth file
Using /home/sarah/.local/lib/python3.6/site-packages
Searching for python-dateutil==2.8.1
Best match: python-dateutil 2.8.1
Adding python-dateutil 2.8.1 to easy-install.pth file
Using /home/sarah/.local/lib/python3.6/site-packages
Searching for six==1.13.0
Best match: six 1.13.0
Adding six 1.13.0 to easy-install.pth file
Using /home/sarah/.local/lib/python3.6/site-packages
Finished processing dependencies for poismf==0.1.4
from poismf.
Strange case. Looks from the output that it compiled correctly and generated the following file in the first case (build_ext
):
/home/sarah/poismf/poismf/poismf_c_wrapper.cpython-36m-x86_64-linux-gnu.so
If you do indeed see that file, then it should be possible to import it by running python
in this folder:
/home/sarah/poismf
(e.g. cd /home/sarah/poismf && python
, then inside python from poismf import PoisMF
)
If it still complains that the file is not there, only reason I can think of is that somehow you have different python versions under python
, python3
, sudo python
, in which case you'd have to use the same command for installing and running it.
From the second one (install
), issue is that it gets installed in a system folder, but then you run python locally in that same folder, and it sees that there's a local module poismf
, so it imports that module instead of the system one, but the compiled file is not there as it was generated in the installation folder.
Solution in the 2nd case is to launch python in a different folder, or use build_ext
if you want to launch it from the same folder. Also I don't think you should be using sudo
for that, as otherwise if you then run python
as a user it will again try to import packages from a different folder.
from poismf.
Hey so for the cd /home/sarah/poismf && python
command upon importing I am getting another error
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/sarah/poismf/poismf/__init__.py", line 4, in <module>
from .poismf_c_wrapper import run_pgd, _predict_multiple, _predict_factors
ImportError: libmkl_rt.so: cannot open shared object file: No such file or directory
I used python3
in all of my commands for installing and running, so I hope that should not be a problem.
from poismf.
For the second case, I opened a new terminal and tried the following, still getting the same error as in the above case
python3
Python 3.6.8 (default, Oct 7 2019, 12:59:55)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from poismf import PoisMF
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python3.6/dist-packages/poismf-0.1.4-py3.6-linux-x86_64.egg/poismf/__init__.py", line 4, in <module>
from .poismf_c_wrapper import run_pgd, _predict_multiple, _predict_factors
ImportError: libmkl_rt.so: cannot open shared object file: No such file or directory
>>>
The reason I had to use sudo was the commnd python setup.py install
was giving an permission denied
error.
from poismf.
For sanity testing, I tried running it on R, and it seems to be working without any error.
head(predict(model, 1)) ## predict the whole row 1
[1] 3.248912e-08 2.351517e-08 2.005383e-08 3.683958e-08 2.652832e-08
[6] 3.473318e-08
>
> #all predictions for new row/user/doc
> head(predict(model, data.frame(col_ix = c(1,2,3), count = c(4,5,6)) ))
[1] 8.896338e-06 6.439046e-06 5.491243e-06 1.008760e-05 7.264121e-06
[6] 9.510819e-06
from poismf.
I see now: thing is, this package depends on some BLAS library as backend for linear algebra operations. In your case, you have a local install of MKL which gets chosen as the library to link to. In ubuntu I think R will by default install OpenBLAS as backend.
Now, the BLAS library to link to is determined through package findblas
, which will prefer MKL over OpenBLAS if it can find it. But for some reason, after it links to the file /home/sarah/.local/lib/libmkl_rt.so
, it then cannot find it (does that file exist in your computer?).
I have no idea how that happened (gcc
shows the correct arguments: -L/home/sarah/.local/lib -l:libmkl_rt.so
), and no idea how to solve it (this is managed by setuptools
). Only possible solutions I can think of are:
- Pre-load the library:
LD_PRELOAD=/home/sarah/.local/lib/libmkl_rt.so python
, maybeLD_LIBRARY_PATH=/home/sarah/.local/lib python
, or both at the same time. - Force a relinkage to the
libmkl_rt.so
in your local folder (complicated), or to the samelibblas.so
used by R (probably something like/usr/lib/x86_64-linux-gnu/openblas/libblas.so
). - Install openblas (
sudo aptitude install libopenblas-dev
) if not already installed, then insetup.py
change this line:
from findblas.distutils import build_ext_with_blas
to this:
from Cython.Distutils import build_ext as build_ext_with_blas
then in the same file
e.extra_link_args += ['-fopenmp']
to
e.extra_link_args += ['-fopenmp', '-lcblas']
EDIT: still wrong for the last one, would then need editing the file pgd.c
by changing #include "findblas.h"
to #include "cblas.h"
, and add to the setup file the path in which the headers are, which is yet another source of issues.
Also just tested it in a default conda install in debian 10, seemed to work fine there.
from poismf.
BTW: if you see that this file does not exist: /home/sarah/.local/lib/libmkl_rt.so
Then it's an issue with this other package: https://www.github.com/david-cortes/findblas
If the file is not there, would appreciate it if you could open an issue in that other package describing how you installed python and numpy/scipy.
from poismf.
I am having the same error:
from poismf import PoisMF File "/home/patrick/.local/lib/python3.6/site-packages/poismf/__init__.py", line 4, in <module> from .poismf_c_wrapper import run_pgd, _predict_multiple, _predict_factors ImportError: libmkl_rt.so: cannot open shared object file: No such file or directory
I'm using Linux Mint 18.3. Installed mkl-devel using:
python3 -m pip install mkl-devel --user
then poismf similarly. libmkl_rt.so exists in the ./local/lib folder, and the LD_PRELOAD/LD_LIBRARY_PATH suggestion above didn't change the error message.
from poismf.
@patrickfletcher : Thanks for the report. Will better try it under Ubuntu now.
In the meantime, could you please find the generated .so
file (something like poismf_c_wrapper.cpython-37m-x86_64-linux-gnu.so
, under the same path as __init__.py
(/home/patrick/.local/lib/python3.6/site-packages/poismf/
)), run ldd
on it (linux command) – e.g.:
ldd /home/patrick/.local/lib/python3.6/site-packages/poismf/poismf_c_wrapper*.so
and paste the full output in here.
from poismf.
ldd /home/patrick/.local/lib/python3.6/site-packages/poismf/poismf_c_wrapper*.so linux-vdso.so.1 (0x00007fff981a2000) libmkl_rt.so => not found libgomp.so.1 => /usr/lib/x86_64-linux-gnu/libgomp.so.1 (0x00007fd342b66000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fd342775000) /lib64/ld-linux-x86-64.so.2 (0x00007fd342fb0000) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fd342571000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fd342352000)
from poismf.
@patrickfletcher : Another thing: how did you install NumPy? E.g.: conda
, pip
, apt-get
/aptitude
, compiled from source, etc.
from poismf.
Yes, I have numpy installed: numpy==1.18.1 - I installed via
python3 -m pip -U numpy --user
from poismf.
Thanks for your prompt response! :)
from poismf.
I wasn’t able to figure out why it fails to install. Only solution I found is to either use a Python install from conda
, or to uninstall the mkl-devel
and numpy
from pip
and install instead the ones from system packages, or to use openblas
.
The problem nevertheless also seems to affect NumPy somehow (installs but is not able to find MKL), so I opened an issue there: numpy/numpy#15372
from poismf.
OK, interesting. I'll try the other solutions when I get a chance. Thanks!
from poismf.
@patrickfletcher : I've pushed an update to package findblas
(dependency of this) which should solve the issue. Could you please update to the latest version (python3 -m pip install findblas==0.1.4
) and then confirm if this package installs and loads correctly? (You'll need to run pip3 uninstall poismf
first if you installed)
from poismf.
It works now! I did as you said - upgraded findblas, then I rebuilt poismf (not use the cached whl).
Thanks!
from poismf.
Related Issues (2)
- HPF implementation HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from poismf.