Comments (15)
Works, fantastic! Thanks for the hint!
I noticed that the makefile does not reliably re-create the .cpp
files from the .pyx
files. In Spack, I had to remove them beforehand and disable make
parallelism (-j
) to get it to build: spack/spack#28223
from adios.
Ping @jychoi-hpc @pnorbert @dmitry-ganyushin @lwan86: do you think this can be easily fixed? :)
from adios.
@ax3l sorry for my late reply. Can you help me to understand the error? Is this because of using new version of numpy? which python version did you use? I tried to build on Cori with python 3.9.7 (numpy 1.20.3, cython 0.29.24) and didn't get an error.
from adios.
@jychoi-hpc no worries. I think this is due to a new version of CPython. tp_print
is dropped from Python 3.9 onwards.
https://docs.python.org/3/c-api/typeobj.html#c.PyTypeObject.tp_vectorcall_offset
Maybe Cython of some of the checked-in ADIOS Cython calls/file generations need to be updated?
from adios.
I see the problem on conda-forge:
https://docs.python.org/3/c-api/typeobj.html#c.PyTypeObject.tp_vectorcall_offset
and build like this:
https://github.com/conda-forge/adios-python-feedstock/blob/master/recipe/build.sh
Should I run Cython or something beforehand? :)
from adios.
I just tried again by doing exactly the same thing in build.sh
on Cori without any issue. I used Python 3.9.7. It looks like condo-forge is using more recent python. Do you know which version of python on conda-forge is used?
Alternatively, you can try the following command to invoke 'cython' beforehand:
make CYTHON=y python
instead of
make python
from adios.
Thanks a lot, let's try this in Spack (spack/spack#28003 / spack/spack#28223) and Conda-Forge (conda-forge/adios-python-feedstock#11).
X-ref: #202
from adios.
Looks like we still hit this error with suggested commands:
[ 214s] + make CYTHON=y MPI=y python
[ 214s] python setup.py build_ext -lrt
[ 214s] running build_ext
[ 214s] building 'adios.adios' extension
[ 214s] creating build
[ 214s] creating build/temp.linux-aarch64-3.9
[ 214s] gcc -Wno-unused-result -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -D_GNU_SOURCE -fPIC -fwrapv -D_GNU_SOURCE -fPIC -fwrapv -fstack-protector-strong -D_FORTIFY_SOURCE=2 -O2 -funsigned-char -D_GNU_SOURCE -fPIC -fwrapv -fstack-protector-strong -D_FORTIFY_SOURCE=2 -O2 -funsigned-char -D_GNU_SOURCE -fPIC -fwrapv -fstack-protector-strong -D_FORTIFY_SOURCE=2 -O2 -funsigned-char -I/opt/ohpc/pub/libs/gnu12/numpy/1.19.5/lib64/python3.9/site-packages/numpy/core/include -I/home/abuild/rpmbuild/BUILD/adios-1.13.1/src/public -L/home/abuild/rpmbuild/BUILD/adios-1.13.1/src -I/opt/ohpc/pub/mpi/openmpi4-gnu12/4.1.4/include -fPIC -D_NOMPI -I/opt/ohpc/pub/libs/gnu12/numpy/1.19.5/lib64/python3.9/site-packages/numpy/core/include -I/opt/ohpc/pub/libs/gnu12/openmpi4/adios/1.13.1/include -I/opt/ohpc/pub/libs/gnu12/openmpi4/netcdf/4.9.0/include -I/opt/ohpc/pub/libs/gnu12/openmpi4/hdf5/1.10.8/include -I/usr/include -I/usr/include/python3.9 -c adios.cpp -o build/temp.linux-aarch64-3.9/adios.o -Wno-uninitialized -Wno-unused-function
[ 214s] In file included from /opt/ohpc/pub/libs/gnu12/numpy/1.19.5/lib64/python3.9/site-packages/numpy/core/include/numpy/ndarraytypes.h:1822,
[ 214s] from /opt/ohpc/pub/libs/gnu12/numpy/1.19.5/lib64/python3.9/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,
[ 214s] from /opt/ohpc/pub/libs/gnu12/numpy/1.19.5/lib64/python3.9/site-packages/numpy/core/include/numpy/arrayobject.h:4,
[ 214s] from adios.cpp:540:
[ 214s] /opt/ohpc/pub/libs/gnu12/numpy/1.19.5/lib64/python3.9/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
[ 214s] 17 | #warning "Using deprecated NumPy API, disable it with " \
[ 214s] | ^~~~~~~
[ 215s] adios.cpp: In function ‘PyObject* PyInit_adios()’:
[ 215s] adios.cpp:52250:30: error: ‘PyTypeObject’ {aka ‘struct _typeobject’} has no member named ‘tp_print’
[ 215s] 52250 | __pyx_type_5adios_softdict.tp_print = 0;
[ 215s] | ^~~~~~~~
[ 215s] adios.cpp:52261:26: error: ‘PyTypeObject’ {aka ‘struct _typeobject’} has no member named ‘tp_print’
[ 215s] 52261 | __pyx_type_5adios_file.tp_print = 0;
[ 215s] | ^~~~~~~~
[ 215s] adios.cpp:52287:31: error: ‘PyTypeObject’ {aka ‘struct _typeobject’} has no member named ‘tp_print’
[ 215s] 52287 | __pyx_type_5adios_blockinfo.tp_print = 0;
[ 215s] | ^~~~~~~~
[ 215s] adios.cpp:52300:25: error: ‘PyTypeObject’ {aka ‘struct _typeobject’} has no member named ‘tp_print’
[ 215s] 52300 | __pyx_type_5adios_var.tp_print = 0;
[ 215s] | ^~~~~~~~
[ 215s] adios.cpp:52306:26: error: ‘PyTypeObject’ {aka ‘struct _typeobject’} has no member named ‘tp_print’
[ 215s] 52306 | __pyx_type_5adios_attr.tp_print = 0;
[ 215s] | ^~~~~~~~
[ 215s] adios.cpp:52312:27: error: ‘PyTypeObject’ {aka ‘struct _typeobject’} has no member named ‘tp_print’
[ 215s] 52312 | __pyx_type_5adios_group.tp_print = 0;
[ 215s] | ^~~~~~~~
[ 215s] adios.cpp:52338:31: error: ‘PyTypeObject’ {aka ‘struct _typeobject’} has no member named ‘tp_print’
[ 215s] 52338 | __pyx_type_5adios_smartdict.tp_print = 0;
[ 215s] | ^~~~~~~~
[ 215s] adios.cpp:52343:28: error: ‘PyTypeObject’ {aka ‘struct _typeobject’} has no member named ‘tp_print’
[ 215s] 52343 | __pyx_type_5adios_writer.tp_print = 0;
[ 215s] | ^~~~~~~~
[ 215s] adios.cpp:52348:30: error: ‘PyTypeObject’ {aka ‘struct _typeobject’} has no member named ‘tp_print’
[ 215s] 52348 | __pyx_type_5adios_attrinfo.tp_print = 0;
[ 215s] | ^~~~~~~~
[ 215s] adios.cpp:52353:29: error: ‘PyTypeObject’ {aka ‘struct _typeobject’} has no member named ‘tp_print’
[ 215s] 52353 | __pyx_type_5adios_varinfo.tp_print = 0;
[ 215s] | ^~~~~~~~
Any suggestion?
from adios.
Trying to build ADIOS 1.13.1 (latest release, from Apr 18 2018) fails with the above problem.
adios.cpp and adios_mpi.cpp are regenerated with Cython 0.29 at Oct 2 2019 with 327e159
I think it is time for a new release!
from adios.
It looks like the cython version 0.29 at Oct 2 2019 is old. I just tried adios 1.13.1, cython 0.29.32, and python 3.9.13 and it works for me:
jyc@login4 numpy ((v1.13.1)) $ cython --version
Cython version 0.29.32
jyc@login4 numpy ((v1.13.1)) $ python --version
Python 3.9.13
jyc@login4 numpy ((v1.13.1)) $ make CYTHON=y MPI=y python
cython --cplus adios_mpi.pyx
/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib/python3.9/site-packages/Cython/Compiler/Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /gpfs/alpine/csc143/world-shared/jyc/summit/src/adios_develop/wrappers/numpy/adios_mpi.pyx
tree = Parsing.p_module(s, pxd, full_module_name)
cython --cplus adios.pyx
/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib/python3.9/site-packages/Cython/Compiler/Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /gpfs/alpine/csc143/world-shared/jyc/summit/src/adios_develop/wrappers/numpy/adios.pyx
tree = Parsing.p_module(s, pxd, full_module_name)
python setup.py build_ext -lrt
/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib/python3.9/site-packages/setuptools/dist.py:771: UserWarning: Usage of dash-separated 'description-file' will not be supported in future versions. Please use the underscore name 'description_file' instead
warnings.warn(
running build_ext
building 'adios.adios' extension
gcc -pthread -B /gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -mcpu=power8 -fPIC -O2 -isystem /gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/include -I/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/include -mcpu=power8 -fPIC -O2 -isystem /gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/include -fPIC -D_NOMPI -I/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib/python3.9/site-packages/numpy/core/include -I/gpfs/alpine/world-shared/csc143/jyc/summit/sw/adios/v1.13.1/gcc9.1.0/include -I/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/include/python3.9 -c adios.cpp -o build/temp.linux-ppc64le-cpython-39/adios.o -Wno-uninitialized -Wno-unused-function
In file included from /gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib/python3.9/site-packages/numpy/core/include/numpy/ndarraytypes.h:1969,
from /gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib/python3.9/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,
from /gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib/python3.9/site-packages/numpy/core/include/numpy/arrayobject.h:4,
from adios.cpp:761:
/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib/python3.9/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
17 | #warning "Using deprecated NumPy API, disable it with " \
| ^~~~~~~
g++ -pthread -B /gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/compiler_compat -shared -Wl,-rpath,/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib -Wl,-rpath-link,/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib -L/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib -L/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib -Wl,-rpath,/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib -Wl,-rpath-link,/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib -L/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib build/temp.linux-ppc64le-cpython-39/adios.o -L/gpfs/alpine/world-shared/csc143/jyc/summit/sw/adios/v1.13.1/gcc9.1.0/lib -ladios_nompi -lrt -lm -lrt -o build/lib.linux-ppc64le-cpython-39/adios/adios.cpython-39-powerpc64le-linux-gnu.so
python setup_mpi.py build
/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib/python3.9/site-packages/setuptools/dist.py:771: UserWarning: Usage of dash-separated 'description-file' will not be supported in future versions. Please use the underscore name 'description_file' instead
warnings.warn(
running build
running build_src
running build_py
copying src_mpi/__init__.py -> build/lib.linux-ppc64le-cpython-39/adios_mpi
running build_ext
MPI configuration: [mpi] from 'mpi.cfg'
MPI C compiler: /sw/summit/spack-envs/base/opt/linux-rhel8-ppc64le/gcc-9.1.0/spectrum-mpi-10.4.0.3-20210112-6jbupg3thjwhsabgevk6xmwhd2bbyxdc/bin/mpicc
MPI C++ compiler: /sw/summit/spack-envs/base/opt/linux-rhel8-ppc64le/gcc-9.1.0/spectrum-mpi-10.4.0.3-20210112-6jbupg3thjwhsabgevk6xmwhd2bbyxdc/bin/mpicxx
MPI F77 compiler: /sw/summit/spack-envs/base/opt/linux-rhel8-ppc64le/gcc-9.1.0/spectrum-mpi-10.4.0.3-20210112-6jbupg3thjwhsabgevk6xmwhd2bbyxdc/bin/mpif77
MPI F90 compiler: /sw/summit/spack-envs/base/opt/linux-rhel8-ppc64le/gcc-9.1.0/spectrum-mpi-10.4.0.3-20210112-6jbupg3thjwhsabgevk6xmwhd2bbyxdc/bin/mpif90
building 'adios_mpi.adios_mpi' extension
/sw/summit/spack-envs/base/opt/linux-rhel8-ppc64le/gcc-9.1.0/spectrum-mpi-10.4.0.3-20210112-6jbupg3thjwhsabgevk6xmwhd2bbyxdc/bin/mpicc -B /gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/compiler_compat -fPIC -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -I/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib/python3.9/site-packages/numpy/core/include -I/gpfs/alpine/world-shared/csc143/jyc/summit/sw/adios/v1.13.1/gcc9.1.0/include -I/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/include/python3.9 -c adios_mpi.cpp -o build/temp.linux-ppc64le-cpython-39/adios_mpi.o -Wno-unused-function -Wno-uninitialized
In file included from /gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib/python3.9/site-packages/numpy/core/include/numpy/ndarraytypes.h:1969,
from /gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib/python3.9/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,
from /gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib/python3.9/site-packages/numpy/core/include/numpy/arrayobject.h:4,
from adios_mpi.cpp:762:
/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib/python3.9/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
17 | #warning "Using deprecated NumPy API, disable it with " \
| ^~~~~~~
/sw/summit/spack-envs/base/opt/linux-rhel8-ppc64le/gcc-9.1.0/spectrum-mpi-10.4.0.3-20210112-6jbupg3thjwhsabgevk6xmwhd2bbyxdc/bin/mpicxx -B /gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/compiler_compat -fPIC -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -B /gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/compiler_compat -shared -Wl,-rpath,/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib -Wl,-rpath-link,/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib -L/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib -L/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib -Wl,-rpath,/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib -Wl,-rpath-link,/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib -L/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib build/temp.linux-ppc64le-cpython-39/adios_mpi.o -L/gpfs/alpine/world-shared/csc143/jyc/summit/sw/adios/v1.13.1/gcc9.1.0/lib -ladios -lrt -lm -o build/lib.linux-ppc64le-cpython-39/adios_mpi/adios_mpi.cpython-39-powerpc64le-linux-gnu.so
(removed some warnings)
I don't know which version of cython starts supproting python 3.9. But, can you try with different cython versions?
from adios.
I wonder how this works for you with v1.13.1!
https://raw.githubusercontent.com/ornladios/ADIOS/v1.13.1/wrappers/numpy/adios.cpp does not use checks like #if PY_VERSION_HEX < 0x030800B1
before trying to access .tp_print
as it does in master
I'll try to investigate some more tomorrow!
from adios.
@martin-g @Yikun you are probably aware (and don't let me break our enthusiasm on getting old builds to work), but ADIOS1 is now deprecated and ADIOS2 is the new hot project: https://github.com/ornladios/ADIOS2
from adios.
I used the following command to re-generate adios.cpp
and adios_mpi.cpp
:
make CYTHON=y MPI=y python
As @ax3l said, I also recommend to use Adios2 if possible.
from adios.
Thank you for your help, @jychoi-hpc and @ax3l !
I didn't know it is possible to re-generate these files on the fly!
This is what we did at https://github.com/openhpc/ohpc/pull/1635/files
If we face other issues most probably we will move to ADIOS 2!
from adios.
If we face other issues most probably we will move to ADIOS 2!
ADIOS2 also has more features that have been maintained for a few years.
from adios.
Related Issues (20)
- ADIOS_READ_METHOD_BP_AGGREGATE not working in read API HOT 1
- bpdump crash on attribute file written by machine of different en HOT 4
- runconfig fail on theta@alcf HOT 3
- Numpy: Slicing is missing the stepping HOT 4
- python 3 support broken by print statements HOT 1
- SZ transformation build error HOT 2
- adios_mpi.cpp doesn't work with Python 3.7 HOT 2
- Support ZFP 0.5.1+ HOT 2
- Possible bug in bp2ncd utility HOT 1
- GFortran 10 Argument Mismatch HOT 2
- Question about timers HOT 4
- ADIOS Spack package broken on Summit HOT 5
- ZFP_INCLUDE_DIR not set HOT 1
- Compilation error HOT 6
- Attribute of Array of Complex
- Python DeprecationWarning: np.asscalar(a) is deprecated since NumPy v1.16, use a.item() instead HOT 1
- Python: wrapper of adios.File close errors
- Bug writing empty data sets? HOT 13
- Subsetting Data for in situ visualization HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from adios.