Git Product home page Git Product logo

Comments (15)

ax3l avatar ax3l commented on July 30, 2024 1

Works, fantastic! Thanks for the hint!

I noticed that the makefile does not reliably re-create the .cpp files from the .pyx files. In Spack, I had to remove them beforehand and disable make parallelism (-j) to get it to build: spack/spack#28223

from adios.

ax3l avatar ax3l commented on July 30, 2024

Ping @jychoi-hpc @pnorbert @dmitry-ganyushin @lwan86: do you think this can be easily fixed? :)

from adios.

jychoi-hpc avatar jychoi-hpc commented on July 30, 2024

@ax3l sorry for my late reply. Can you help me to understand the error? Is this because of using new version of numpy? which python version did you use? I tried to build on Cori with python 3.9.7 (numpy 1.20.3, cython 0.29.24) and didn't get an error.

from adios.

ax3l avatar ax3l commented on July 30, 2024

@jychoi-hpc no worries. I think this is due to a new version of CPython. tp_print is dropped from Python 3.9 onwards.
https://docs.python.org/3/c-api/typeobj.html#c.PyTypeObject.tp_vectorcall_offset

Maybe Cython of some of the checked-in ADIOS Cython calls/file generations need to be updated?

from adios.

ax3l avatar ax3l commented on July 30, 2024

I see the problem on conda-forge:
https://docs.python.org/3/c-api/typeobj.html#c.PyTypeObject.tp_vectorcall_offset
and build like this:
https://github.com/conda-forge/adios-python-feedstock/blob/master/recipe/build.sh

Should I run Cython or something beforehand? :)

from adios.

jychoi-hpc avatar jychoi-hpc commented on July 30, 2024

I just tried again by doing exactly the same thing in build.sh on Cori without any issue. I used Python 3.9.7. It looks like condo-forge is using more recent python. Do you know which version of python on conda-forge is used?

Alternatively, you can try the following command to invoke 'cython' beforehand:

make CYTHON=y python

instead of

make python

from adios.

ax3l avatar ax3l commented on July 30, 2024

Thanks a lot, let's try this in Spack (spack/spack#28003 / spack/spack#28223) and Conda-Forge (conda-forge/adios-python-feedstock#11).

X-ref: #202

from adios.

Yikun avatar Yikun commented on July 30, 2024

Looks like we still hit this error with suggested commands:

[  214s] + make CYTHON=y MPI=y python
[  214s] python setup.py build_ext -lrt
[  214s] running build_ext
[  214s] building 'adios.adios' extension
[  214s] creating build
[  214s] creating build/temp.linux-aarch64-3.9
[  214s] gcc -Wno-unused-result -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -D_GNU_SOURCE -fPIC -fwrapv -D_GNU_SOURCE -fPIC -fwrapv -fstack-protector-strong -D_FORTIFY_SOURCE=2 -O2 -funsigned-char -D_GNU_SOURCE -fPIC -fwrapv -fstack-protector-strong -D_FORTIFY_SOURCE=2 -O2 -funsigned-char -D_GNU_SOURCE -fPIC -fwrapv -fstack-protector-strong -D_FORTIFY_SOURCE=2 -O2 -funsigned-char -I/opt/ohpc/pub/libs/gnu12/numpy/1.19.5/lib64/python3.9/site-packages/numpy/core/include -I/home/abuild/rpmbuild/BUILD/adios-1.13.1/src/public -L/home/abuild/rpmbuild/BUILD/adios-1.13.1/src -I/opt/ohpc/pub/mpi/openmpi4-gnu12/4.1.4/include -fPIC -D_NOMPI -I/opt/ohpc/pub/libs/gnu12/numpy/1.19.5/lib64/python3.9/site-packages/numpy/core/include -I/opt/ohpc/pub/libs/gnu12/openmpi4/adios/1.13.1/include -I/opt/ohpc/pub/libs/gnu12/openmpi4/netcdf/4.9.0/include -I/opt/ohpc/pub/libs/gnu12/openmpi4/hdf5/1.10.8/include -I/usr/include -I/usr/include/python3.9 -c adios.cpp -o build/temp.linux-aarch64-3.9/adios.o -Wno-uninitialized -Wno-unused-function
[  214s] In file included from /opt/ohpc/pub/libs/gnu12/numpy/1.19.5/lib64/python3.9/site-packages/numpy/core/include/numpy/ndarraytypes.h:1822,
[  214s]                  from /opt/ohpc/pub/libs/gnu12/numpy/1.19.5/lib64/python3.9/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,
[  214s]                  from /opt/ohpc/pub/libs/gnu12/numpy/1.19.5/lib64/python3.9/site-packages/numpy/core/include/numpy/arrayobject.h:4,
[  214s]                  from adios.cpp:540:
[  214s] /opt/ohpc/pub/libs/gnu12/numpy/1.19.5/lib64/python3.9/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
[  214s]    17 | #warning "Using deprecated NumPy API, disable it with " \
[  214s]       |  ^~~~~~~
[  215s] adios.cpp: In function ‘PyObject* PyInit_adios()’:
[  215s] adios.cpp:52250:30: error: ‘PyTypeObject’ {aka ‘struct _typeobject’} has no member named ‘tp_print’
[  215s] 52250 |   __pyx_type_5adios_softdict.tp_print = 0;
[  215s]       |                              ^~~~~~~~
[  215s] adios.cpp:52261:26: error: ‘PyTypeObject’ {aka ‘struct _typeobject’} has no member named ‘tp_print’
[  215s] 52261 |   __pyx_type_5adios_file.tp_print = 0;
[  215s]       |                          ^~~~~~~~
[  215s] adios.cpp:52287:31: error: ‘PyTypeObject’ {aka ‘struct _typeobject’} has no member named ‘tp_print’
[  215s] 52287 |   __pyx_type_5adios_blockinfo.tp_print = 0;
[  215s]       |                               ^~~~~~~~
[  215s] adios.cpp:52300:25: error: ‘PyTypeObject’ {aka ‘struct _typeobject’} has no member named ‘tp_print’
[  215s] 52300 |   __pyx_type_5adios_var.tp_print = 0;
[  215s]       |                         ^~~~~~~~
[  215s] adios.cpp:52306:26: error: ‘PyTypeObject’ {aka ‘struct _typeobject’} has no member named ‘tp_print’
[  215s] 52306 |   __pyx_type_5adios_attr.tp_print = 0;
[  215s]       |                          ^~~~~~~~
[  215s] adios.cpp:52312:27: error: ‘PyTypeObject’ {aka ‘struct _typeobject’} has no member named ‘tp_print’
[  215s] 52312 |   __pyx_type_5adios_group.tp_print = 0;
[  215s]       |                           ^~~~~~~~
[  215s] adios.cpp:52338:31: error: ‘PyTypeObject’ {aka ‘struct _typeobject’} has no member named ‘tp_print’
[  215s] 52338 |   __pyx_type_5adios_smartdict.tp_print = 0;
[  215s]       |                               ^~~~~~~~
[  215s] adios.cpp:52343:28: error: ‘PyTypeObject’ {aka ‘struct _typeobject’} has no member named ‘tp_print’
[  215s] 52343 |   __pyx_type_5adios_writer.tp_print = 0;
[  215s]       |                            ^~~~~~~~
[  215s] adios.cpp:52348:30: error: ‘PyTypeObject’ {aka ‘struct _typeobject’} has no member named ‘tp_print’
[  215s] 52348 |   __pyx_type_5adios_attrinfo.tp_print = 0;
[  215s]       |                              ^~~~~~~~
[  215s] adios.cpp:52353:29: error: ‘PyTypeObject’ {aka ‘struct _typeobject’} has no member named ‘tp_print’
[  215s] 52353 |   __pyx_type_5adios_varinfo.tp_print = 0;
[  215s]       |                             ^~~~~~~~

Any suggestion?

[1] https://build.openeuler.org/package/live_build_log/home:huangtianhua:ohpc/adios-gnu12-openmpi4/standard_aarch64/aarch64

from adios.

martin-g avatar martin-g commented on July 30, 2024

Trying to build ADIOS 1.13.1 (latest release, from Apr 18 2018) fails with the above problem.
adios.cpp and adios_mpi.cpp are regenerated with Cython 0.29 at Oct 2 2019 with 327e159
I think it is time for a new release!

from adios.

jychoi-hpc avatar jychoi-hpc commented on July 30, 2024

It looks like the cython version 0.29 at Oct 2 2019 is old. I just tried adios 1.13.1, cython 0.29.32, and python 3.9.13 and it works for me:

jyc@login4 numpy ((v1.13.1)) $ cython --version
Cython version 0.29.32
jyc@login4 numpy ((v1.13.1)) $ python --version
Python 3.9.13
jyc@login4 numpy ((v1.13.1)) $ make CYTHON=y MPI=y python
cython --cplus adios_mpi.pyx
/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib/python3.9/site-packages/Cython/Compiler/Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /gpfs/alpine/csc143/world-shared/jyc/summit/src/adios_develop/wrappers/numpy/adios_mpi.pyx
  tree = Parsing.p_module(s, pxd, full_module_name)
cython --cplus adios.pyx
/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib/python3.9/site-packages/Cython/Compiler/Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /gpfs/alpine/csc143/world-shared/jyc/summit/src/adios_develop/wrappers/numpy/adios.pyx
  tree = Parsing.p_module(s, pxd, full_module_name)
python setup.py build_ext -lrt
/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib/python3.9/site-packages/setuptools/dist.py:771: UserWarning: Usage of dash-separated 'description-file' will not be supported in future versions. Please use the underscore name 'description_file' instead
  warnings.warn(
running build_ext
building 'adios.adios' extension
gcc -pthread -B /gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -mcpu=power8 -fPIC -O2 -isystem /gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/include -I/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/include -mcpu=power8 -fPIC -O2 -isystem /gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/include -fPIC -D_NOMPI -I/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib/python3.9/site-packages/numpy/core/include -I/gpfs/alpine/world-shared/csc143/jyc/summit/sw/adios/v1.13.1/gcc9.1.0/include -I/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/include/python3.9 -c adios.cpp -o build/temp.linux-ppc64le-cpython-39/adios.o -Wno-uninitialized -Wno-unused-function
In file included from /gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib/python3.9/site-packages/numpy/core/include/numpy/ndarraytypes.h:1969,
                 from /gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib/python3.9/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,
                 from /gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib/python3.9/site-packages/numpy/core/include/numpy/arrayobject.h:4,
                 from adios.cpp:761:
/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib/python3.9/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
   17 | #warning "Using deprecated NumPy API, disable it with " \
      |  ^~~~~~~
g++ -pthread -B /gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/compiler_compat -shared -Wl,-rpath,/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib -Wl,-rpath-link,/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib -L/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib -L/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib -Wl,-rpath,/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib -Wl,-rpath-link,/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib -L/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib build/temp.linux-ppc64le-cpython-39/adios.o -L/gpfs/alpine/world-shared/csc143/jyc/summit/sw/adios/v1.13.1/gcc9.1.0/lib -ladios_nompi -lrt -lm -lrt -o build/lib.linux-ppc64le-cpython-39/adios/adios.cpython-39-powerpc64le-linux-gnu.so
python setup_mpi.py build
/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib/python3.9/site-packages/setuptools/dist.py:771: UserWarning: Usage of dash-separated 'description-file' will not be supported in future versions. Please use the underscore name 'description_file' instead
  warnings.warn(
running build
running build_src
running build_py
copying src_mpi/__init__.py -> build/lib.linux-ppc64le-cpython-39/adios_mpi
running build_ext
MPI configuration: [mpi] from 'mpi.cfg'
MPI C compiler:    /sw/summit/spack-envs/base/opt/linux-rhel8-ppc64le/gcc-9.1.0/spectrum-mpi-10.4.0.3-20210112-6jbupg3thjwhsabgevk6xmwhd2bbyxdc/bin/mpicc
MPI C++ compiler:  /sw/summit/spack-envs/base/opt/linux-rhel8-ppc64le/gcc-9.1.0/spectrum-mpi-10.4.0.3-20210112-6jbupg3thjwhsabgevk6xmwhd2bbyxdc/bin/mpicxx
MPI F77 compiler:  /sw/summit/spack-envs/base/opt/linux-rhel8-ppc64le/gcc-9.1.0/spectrum-mpi-10.4.0.3-20210112-6jbupg3thjwhsabgevk6xmwhd2bbyxdc/bin/mpif77
MPI F90 compiler:  /sw/summit/spack-envs/base/opt/linux-rhel8-ppc64le/gcc-9.1.0/spectrum-mpi-10.4.0.3-20210112-6jbupg3thjwhsabgevk6xmwhd2bbyxdc/bin/mpif90
building 'adios_mpi.adios_mpi' extension
/sw/summit/spack-envs/base/opt/linux-rhel8-ppc64le/gcc-9.1.0/spectrum-mpi-10.4.0.3-20210112-6jbupg3thjwhsabgevk6xmwhd2bbyxdc/bin/mpicc -B /gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/compiler_compat -fPIC -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -I/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib/python3.9/site-packages/numpy/core/include -I/gpfs/alpine/world-shared/csc143/jyc/summit/sw/adios/v1.13.1/gcc9.1.0/include -I/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/include/python3.9 -c adios_mpi.cpp -o build/temp.linux-ppc64le-cpython-39/adios_mpi.o -Wno-unused-function -Wno-uninitialized
In file included from /gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib/python3.9/site-packages/numpy/core/include/numpy/ndarraytypes.h:1969,
                 from /gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib/python3.9/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,
                 from /gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib/python3.9/site-packages/numpy/core/include/numpy/arrayobject.h:4,
                 from adios_mpi.cpp:762:
/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib/python3.9/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
   17 | #warning "Using deprecated NumPy API, disable it with " \
      |  ^~~~~~~
/sw/summit/spack-envs/base/opt/linux-rhel8-ppc64le/gcc-9.1.0/spectrum-mpi-10.4.0.3-20210112-6jbupg3thjwhsabgevk6xmwhd2bbyxdc/bin/mpicxx -B /gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/compiler_compat -fPIC -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -B /gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/compiler_compat -shared -Wl,-rpath,/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib -Wl,-rpath-link,/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib -L/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib -L/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib -Wl,-rpath,/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib -Wl,-rpath-link,/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib -L/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib build/temp.linux-ppc64le-cpython-39/adios_mpi.o -L/gpfs/alpine/world-shared/csc143/jyc/summit/sw/adios/v1.13.1/gcc9.1.0/lib -ladios -lrt -lm -o build/lib.linux-ppc64le-cpython-39/adios_mpi/adios_mpi.cpython-39-powerpc64le-linux-gnu.so

(removed some warnings)

I don't know which version of cython starts supproting python 3.9. But, can you try with different cython versions?

from adios.

martin-g avatar martin-g commented on July 30, 2024

I wonder how this works for you with v1.13.1!
https://raw.githubusercontent.com/ornladios/ADIOS/v1.13.1/wrappers/numpy/adios.cpp does not use checks like #if PY_VERSION_HEX < 0x030800B1 before trying to access .tp_print as it does in master

I'll try to investigate some more tomorrow!

from adios.

ax3l avatar ax3l commented on July 30, 2024

@martin-g @Yikun you are probably aware (and don't let me break our enthusiasm on getting old builds to work), but ADIOS1 is now deprecated and ADIOS2 is the new hot project: https://github.com/ornladios/ADIOS2

from adios.

jychoi-hpc avatar jychoi-hpc commented on July 30, 2024

I used the following command to re-generate adios.cpp and adios_mpi.cpp:

make CYTHON=y MPI=y python

As @ax3l said, I also recommend to use Adios2 if possible.

from adios.

martin-g avatar martin-g commented on July 30, 2024

Thank you for your help, @jychoi-hpc and @ax3l !

I didn't know it is possible to re-generate these files on the fly!
This is what we did at https://github.com/openhpc/ohpc/pull/1635/files
If we face other issues most probably we will move to ADIOS 2!

from adios.

williamfgc avatar williamfgc commented on July 30, 2024

If we face other issues most probably we will move to ADIOS 2!

ADIOS2 also has more features that have been maintained for a few years.

from adios.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.