llnl / macsio Goto Github PK

View Code? Open in Web Editor NEW

33.0 8.0 16.0 11.18 MB

A Multi-purpose, Application-Centric, Scalable I/O Proxy Application

Home Page: https://computing.llnl.gov/projects/co-design/macsio

License: Other

Makefile 1.73% C 94.48% C++ 1.33% CMake 2.19% Shell 0.28%

proxy-application hpc scientific-computing performance-testing io-benchmark parallel-io io-library

macsio's Issues

Fix all JSON-C symbols to avoid collisions with real JSON-C library

INSTALL.md not corect

The instructions should say to use hostname -f since that is what is done in the Makefile.

Eliminate HAVE_MPI conditional compilation

We made the MPI code conditionally compiled originally on the idea that maybe multiple forms of parallelism would be supported here. But, do we expect to be doing anything useful at large scale without MPI? If not, having MPI conditionally compiled only complicates the code without much benefit and it should be removed.

Add CL args to control file system dir naming

Macsio generates a large number of log files. Is it possible to redirect those files to another directory in the command line?

login-1:macsio_darshan$ ls
darshan_runtime_logs	    macsio_silo_00002_005.silo	macsio_silo_00005_004.silo
macsio			    macsio_silo_00002_006.silo	macsio_silo_00005_005.silo
macsio-log.log		    macsio_silo_00002_007.silo	macsio_silo_00005_006.silo
macsio-timings.log	    macsio_silo_00002_008.silo	macsio_silo_00005_007.silo
macsio_silo_00000_000.silo  macsio_silo_00002_009.silo	macsio_silo_00005_008.silo
macsio_silo_00000_001.silo  macsio_silo_00003_000.silo	macsio_silo_00005_009.silo
macsio_silo_00000_002.silo  macsio_silo_00003_001.silo	macsio_silo_00006_000.silo
macsio_silo_00000_003.silo  macsio_silo_00003_002.silo	macsio_silo_00006_001.silo
macsio_silo_00000_004.silo  macsio_silo_00003_003.silo	macsio_silo_00006_002.silo
macsio_silo_00000_005.silo  macsio_silo_00003_004.silo	macsio_silo_00006_003.silo
macsio_silo_00000_006.silo  macsio_silo_00003_005.silo	macsio_silo_00006_004.silo
macsio_silo_00000_007.silo  macsio_silo_00003_006.silo	macsio_silo_00006_005.silo
macsio_silo_00000_008.silo  macsio_silo_00003_007.silo	macsio_silo_00006_006.silo
macsio_silo_00000_009.silo  macsio_silo_00003_008.silo	macsio_silo_00006_007.silo
macsio_silo_00001_000.silo  macsio_silo_00003_009.silo	macsio_silo_00006_008.silo
macsio_silo_00001_001.silo  macsio_silo_00004_000.silo	macsio_silo_00006_009.silo

Building Error. /bin/sh: line 0: cd: ../json-cwx: No such file or directory

Hi, I followed the steps in the file 'INSTALL' to try building the proxy app.
But I got the following error. It seems there is no 'json-cwx' directory anywhere in the repository.

zinechant@MiBookChant:~/ECP/MACSio/MACSio/macsio$ make all
cd ../json-cwx; \
mkdir build install; \
cd json-cwx; \
./autogen.sh; \
cd ..; \
cd build; \
../json-cwx/configure CC="mpicc" --prefix=`pwd`/../install
/bin/sh: line 0: cd: ../json-cwx: No such file or directory
mkdir: cannot create directory ‘build’: File exists
mkdir: cannot create directory ‘install’: File exists
/bin/sh: line 2: cd: json-cwx: No such file or directory
/bin/sh: line 3: ./autogen.sh: No such file or directory
/bin/sh: line 5: cd: build: No such file or directory
/bin/sh: line 6: ../json-cwx/configure: No such file or directory
Makefile:128: recipe for target '../json-cwx/build' failed
make: *** [../json-cwx/build] Error 127

Thanks in advance.

Segfault and core dump on Summit

I need some guidance on how to run MACSio on Summit.
To reproduce: I was able to build successfully the MACSio binary with the following dependencies:

ldd ~/opt/macsio/macsio 
	linux-vdso64.so.1 =>  (0x00007fffb6120000)
	libjson-cwx.so.2 => /ccs/home/wgodoy/opt/json-cwx/lib/libjson-cwx.so.2 (0x00007fffb60e0000)
	libmpiprofilesupport.so.3 => /autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/spectrum-mpi-10.3.1.2-20200121-awz2q5brde7wgdqqw4ugalrkukeub4eb/lib/libmpiprofilesupport.so.3 (0x00007fffb60b0000)
	libmpi_ibm.so.3 => /autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/spectrum-mpi-10.3.1.2-20200121-awz2q5brde7wgdqqw4ugalrkukeub4eb/lib/libmpi_ibm.so.3 (0x00007fffb5f30000)
	libstdc++.so.6 => /sw/summit/gcc/6.4.0/lib64/libstdc++.so.6 (0x00007fffb5d20000)
	libm.so.6 => /lib64/libm.so.6 (0x00007fffb5c10000)
	libgcc_s.so.1 => /sw/summit/gcc/6.4.0/lib64/libgcc_s.so.1 (0x00007fffb5bd0000)
	libc.so.6 => /lib64/libc.so.6 (0x00007fffb59e0000)
	librt.so.1 => /lib64/librt.so.1 (0x00007fffb59b0000)
	libutil.so.1 => /lib64/libutil.so.1 (0x00007fffb5980000)
	libhwloc_ompi.so.15 => /autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/spectrum-mpi-10.3.1.2-20200121-awz2q5brde7wgdqqw4ugalrkukeub4eb/lib/libhwloc_ompi.so.15 (0x00007fffb5910000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007fffb58e0000)
	libevent-2.1.so.6 => /autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/spectrum-mpi-10.3.1.2-20200121-awz2q5brde7wgdqqw4ugalrkukeub4eb/lib/libevent-2.1.so.6 (0x00007fffb5860000)
	libevent_pthreads-2.1.so.6 => /autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/spectrum-mpi-10.3.1.2-20200121-awz2q5brde7wgdqqw4ugalrkukeub4eb/lib/libevent_pthreads-2.1.so.6 (0x00007fffb5830000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fffb57f0000)
	libopen-rte.so.3 => /autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/spectrum-mpi-10.3.1.2-20200121-awz2q5brde7wgdqqw4ugalrkukeub4eb/lib/libopen-rte.so.3 (0x00007fffb56e0000)
	libopen-pal.so.3 => /autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/spectrum-mpi-10.3.1.2-20200121-awz2q5brde7wgdqqw4ugalrkukeub4eb/lib/libopen-pal.so.3 (0x00007fffb55f0000)
	/lib64/ld64.so.2 (0x00007fffb6140000)

Unfortunately, any combination of input parameters result in a core dump being emitted and a seg fault.

jsrun -n 2 ${MACSIO_EXEC} --interface hdf5 --parallel_file_mode MIF 2 --part_size 1M

Results:

cat output.490227 
[b28n03:147697] *** Process received signal ***
[b28n03:147697] Signal: Segmentation fault (11)
[b28n03:147697] Signal code: Address not mapped (1)
[b28n03:147697] Failing at address: 0x40
[b28n03:147697] [ 0] [0x2000000504d8]
[b28n03:147697] [ 1] [0x20000004d6b0]
[b28n03:147697] [ 2] /ccs/home/wgodoy/opt/json-cwx/lib/libjson-cwx.so.2(json_object_set_string+0x58)[0x2000000f9838]
[b28n03:147697] [ 3] /ccs/home/wgodoy/opt/json-cwx/lib/libjson-cwx.so.2(json_object_path_set_string+0x34)[0x2000000fa764]
[b28n03:147697] [ 4] /ccs/home/wgodoy/opt/macsio/macsio[0x10018cc0]
[b28n03:147697] [ 5] /ccs/home/wgodoy/opt/macsio/macsio(main+0x900)[0x10005980]
[b28n03:147697] [ 6] /lib64/libc.so.6(+0x25200)[0x200000655200]
[b28n03:147697] [ 7] /lib64/libc.so.6(__libc_start_main+0xc4)[0x2000006553f4]
[b28n03:147697] *** End of error message ***
[b28n03:147696] *** Process received signal ***
[b28n03:147696] Signal: Segmentation fault (11)
[b28n03:147696] Signal code: Address not mapped (1)
[b28n03:147696] Failing at address: 0x40
[b28n03:147696] [ 0] [0x2000000504d8]
[b28n03:147696] [ 1] [0x20000004d6b0]
[b28n03:147696] [ 2] /ccs/home/wgodoy/opt/json-cwx/lib/libjson-cwx.so.2(json_object_set_string+0x58)[0x2000000f9838]
[b28n03:147696] [ 3] /ccs/home/wgodoy/opt/json-cwx/lib/libjson-cwx.so.2(json_object_path_set_string+0x34)[0x2000000fa764]
[b28n03:147696] [ 4] /ccs/home/wgodoy/opt/macsio/macsio[0x10018cc0]
[b28n03:147696] [ 5] /ccs/home/wgodoy/opt/macsio/macsio(main+0x900)[0x10005980]
[b28n03:147696] [ 6] /lib64/libc.so.6(+0x25200)[0x200000655200]
[b28n03:147696] [ 7] /lib64/libc.so.6(__libc_start_main+0xc4)[0x2000006553f4]
[b28n03:147696] *** End of error message ***
ERROR:  One or more process (first noticed rank 0) terminated with signal 11 (core dumped)

macsio-log.log:

--------------------------------------------------------Processor 000000-------------------------------------------------------

Any help would be appreciated!

Restrict use of C++ language features to bare minimum

MACSio is almost entirely C. But, there are a few places where the keyword mutable is used for members of a struct that is used in a mostly const setting. Also, plugins are loaded during executable load time via the use of non-constant, static initializers which are not supported in C but are in C++.

We should find a way to restrict all this to a single C++ source file having all other files be C code and compiled with a C compiler. Currently, we are using C++ compiler everywhere.

Alternatively, we can keep use of C++ compiler everywhere but we should then change the source file suffixes to be either .C or .cxx.

json-cwx build error

It looks like the json-cwx package is missing the INSTALL file from auto tools, preventing MACSio from building out-of-the-box.

Solution 1:
Re-generate auto tools for the json-cwx package by adding cd json-cwx; ./autogen.sh; cd .. to macsio/Makefile:125

Solution 2:
Include the generated file at json-cwx/INSTALL in this repository.

Possible variable eval tool

http://www.partow.net/programming/exprtk/index.html

open-simplex-noise* are all HTML rather than source

The files were not inserted correctly. Also, there should be some good reference back to the original for attribution.

Note about shared library requirements

I configured to just use HDF5 and it needed the HDF5 shared libraries as well as the json-c/install/lib. This is not listed in the INSTALL.md

H5pubconf.h not in my configuration

I have a H5pubconf-64.h. I think there is a difference in how it gets configured for 64-bit setups or something? There are some problems with my setup. Please close this issue. Will make a new one if it is still a problem.

Rename default branch from master to develop

I would like to rename the default branch from master to develop.

I notice there are 11 forks.

Would anyone be unduly negatively effected by this change?

segfault with intel compiler

MACSio/macsio/macsio_clargs.c

Line 234 in e8bece9

MACSIO_KnownArgInfo_t *newArg, *oldArg;

The way this code is written triggers an segfault (null pointer dereference later in the loop) with intel compiler due to undefined behavior of the language. gnu does not reinitialize recurring variable declaration, however intel does, so the linked line will reset both pointers to NULL in every iteration!

Best,
Jens

llnl / macsio Goto Github PK

macsio's Issues

Fix all JSON-C symbols to avoid collisions with real JSON-C library

INSTALL.md not corect

Eliminate HAVE_MPI conditional compilation

Add CL args to control file system dir naming

Building Error. /bin/sh: line 0: cd: ../json-cwx: No such file or directory

Segfault and core dump on Summit

Restrict use of C++ language features to bare minimum

json-cwx build error

Possible variable eval tool

open-simplex-noise* are all HTML rather than source

Note about shared library requirements

H5pubconf.h not in my configuration

Rename default branch from master to develop

segfault with intel compiler

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent