openpmd / openpmd-api Goto Github PK
View Code? Open in Web Editor NEW:floppy_disk: C++ & Python API for Scientific I/O
Home Page: https://openpmd-api.readthedocs.io
License: GNU Lesser General Public License v3.0
openpmd-api's People
Contributors
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot]")
![pre-commit-ci[bot] avatar](https://avatars.githubusercontent.com/in/68672?v=4 "pre-commit-ci[bot]")
Stargazers
Watchers
Forkers
ax3l c0nsultant psychocoderhpc amundson eucall-software cfgrote happy-forks stanczakdominik lge0303 franzpoeschel panosc-vinyl sbastrakov andrewlongman guj lucafedeli88 maxthevenet gtrichardson arghyac007 ldamorim revathijambunathan verolero86 qjohn2017 carolsand jianghongping nicolasaunai prometheuspi christophermayes sayerhs anton-le externalrepositories eschnett samcom12 jeanbez researchapps stillerpatrick wkliao parikshitbajpai dernils-git pinkdiamond1 bernhardmgruber lgtm-migrator skalarproduktraum change72 dpgrote pordyna spencerx iliancs jjwillsc titoirideopenpmd-api's Issues
Handle possible combinations of backends and MPI gracefully
This will result in a more general solution of #108.
Currently, all backends that are MPI-aware only provide their MPI-aware interfaces if both MPI and the required libraries are available. As such, these combinations are handled well:
- noMPI+noBACKEND
- noMPI+BACKEND
- MPI+BACKEND
If MPI is available, but the backend is inactive, there is no constructor with an interface matching the calls in the MPI-aware AbstractIOHandler::createIOHandler.
Possible solutions include explicitly treating the combination MPI+noBACKEND in AbstractIOHandler::createIOHandler (not preferred) or in the backend itself (preferred).
Sphinx: C++/Py Side-By-Side
We should add a little CSS hack in docs/source/_static/ that layouts directly following C++ and Python code blocks side-by-side.
Depends on
JSON
A (serial-only) JSON IO-backend would be cool.
python API: Dask Compatible
CI: Appveyor
Add appveyor to CI for testing Windows platforms.
Serial testing and python support is probably sufficient.
Namespace Per Dir
Follow-up to #81.
Each directory inside include/openPMD/ should define everything in a namespace with the same name as the dir.
backend/(openPMD::backend)IO/(openPMD::IO)ADIOS/(openPMD::IO::ADIOS)HDF5/(openPMD::IO::HDF5)
CMake: Component NOMPI
We should add a CMake component NOMPI to our configuration package to allow explicitly searching for no-MPI installs in find_package.
Namespace for Tests
Tests should also be placed in C++ namespaces.
This way we can keep them included in doxygen builds but they do not pollute the root scope.
Please see http://www.openpmd.org/openPMD-api/annotated.html for reference :)
Python MPI Support
Python (Numpy) API
This issue is about providing a general Python API on top of C++ that can be used in various projects, both parallel and serial.
Possible libraries for bindings:
Ideally, it would be compatible to libs such as Dask #12 and shall very numpy-like for low-level data-access.
Docs: Contribution Guide
Add a contribution guide explaining the dev-master development model, issues & PRs, reviews, etc.
Can in broad parts be taken from e.g. PIConGPU's guide
MeshRecordComponent: Frontend
The MeshRecordComponent is currently located in the backend but used in the user-frontend.
It should therefore be moved accordingly.
uint64_t should be used for all extents
I am not sure if 32bit systems should be supported but if than uint64_t should be used for all extents instead of size_t. size_t is 32bit on 32bit systems and can result in issues with offsets or extents over 4GiB.
I saw in the code that offset and extent is defined as uint64_t but on many other places size_t is used.
bool: h5py compatible
Representing bool types in HDF5 is not pre-defined in HDF5 and can e.g. be implemented via enums.
Those are a little tricky and even case sensitive but can be implemented in a widely accepted way, that e.g. h5py understands. See the implementation in libSplash for reference:
Versioning Scheme
Versioning of the lib can be tricky since the openPMD base standard also has a version and we want to avoid confusion.
The idea I came up with yet was to use always the same major version of openPMD-api as the base standard and try to support all minor versions of this major (but not previous major versions; see updater).
The only problem comes in when we want to do incompatible API changes within a major version, which would break the usual semantic versioning of openPMD-api.
In the README (badges & test) and in openPMD/version.hpp defines we then define additional macros (C++: done already; Py: Needs Exposure) that expose which min and max of the openPMD standard are supported.
Of course we could also just do a simple, independent semantic versioning of openPMD-api (with the docs as mentioned above) but I would bet this will confuse people a lot...
Structure: openPMD/ dir and namespace
All includes should be inside a openPMD/ directory for grouping and additionally in an openPMD:: namespace. A collective openPMD/openPMD.hpp include file would be nice as well.
Remove Boost Dependency
Please replace boost MPL lists with mp11 lists.
Requires Boost 1.66.0+
Update: oh wait, that is only boost variant you are controlling, right?
Anyway, careful with global BOOST_MPL_ defines in-code! Better set them in CMake as PUBLIC target properties since downstream code might use BOOST_MPL as well! (and the compile-time impact of BOOST_MPL_CFG_NO_PREPROCESSED_HEADERS might be big).
There are also other C++11 libs of variant that we could consider to use in order to avoid Boost::MPL modifications.
Possible candidates for C++11 variants:
Possible replacements for C++11 filesystem (we require directory traversal, listing and creation):
- https://github.com/cginternals/cppfs
- https://github.com/wjakob/filesystem
- http://www.martinbroadhurst.com/list-the-files-in-a-directory-in-c.html
- https://github.com/cxong/tinydir
- https://github.com/LLNL/conduit/blob/master/src/libs/conduit/conduit_utils.cpp
- https://github.com/PollardBanknote/cppbackport
Travis: OSX
Add a single OSX build to the travis build matrix.
poc_HDF5writer: Segfault / Uninit Values
The HDF5 writer POC binary segfaults on some machines. Checking it with valgrind reviles some jumps/moves depending on uninitialized values:
Software
- gcc 4.9.2
- HDF5 1.10.0.1
- OpenMPI 2.0.2 (MPI: 3.1)
- valgrind 3.12.0.SVN
on Debian 9.3 (64bit)
Valgrind
valgrind ./bin/poc_HDF5writer
==14663== Memcheck, a memory error detector
==14663== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==14663== Using Valgrind-3.12.0.SVN and LibVEX; rerun with -h for copyright info
==14663== Command: ./bin/poc_HDF5writer
==14663==
Unknown compression format zlib. This might mean that compression will not be enabled.
==14663== Conditional jump or move depends on uninitialised value(s)
==14663== at 0x510D43A: boost::filesystem::path::filename() const (in /usr/lib/x86_64-linux-gnu/libboost_filesystem.so.1.62.0)
==14663== by 0x5106787: boost::filesystem::detail::create_directories(boost::filesystem::path const&, boost::system::error_code*) (in /usr/lib/x86_64-linux-gnu/libboost_filesystem.so.1.62.0)
==14663== by 0x4E69CD: boost::filesystem::create_directories(boost::filesystem::path const&) (operations.hpp:566)
==14663== by 0x4D8AFA: HDF5IOHandlerImpl::createFile(Writable*, std::map<std::string, Variadic<ArgumentDatatype<std::string, std::vector<unsigned long, std::allocator<unsigned long> >, void*, std::shared_ptr<void>, Datatype, boost::variant<char, unsigned char, short, int, long, unsigned short, unsigned int, unsigned long, float, double, long double, std::string, std::vector<char, std::allocator<char> >, std::vector<short, std::allocator<short> >, std::vector<int, std::allocator<int> >, std::vector<long, std::allocator<long> >, std::vector<unsigned char, std::allocator<unsigned char> >, std::vector<unsigned short, std::allocator<unsigned short> >, std::vector<unsigned int, std::allocator<unsigned int> >, std::vector<unsigned long, std::allocator<unsigned long> >, std::vector<float, std::allocator<float> >, std::vector<double, std::allocator<double> >, std::vector<long double, std::allocator<long double> >, std::vector<std::string, std::allocator<std::string> >, std::array<double, 7ul>, bool>, std::shared_ptr<std::vector<unsigned long, std::allocator<unsigned long> > >, std::shared_ptr<Datatype>, std::shared_ptr<boost::variant<char, unsigned char, short, int, long, unsigned short, unsigned int, unsigned long, float, double, long double, std::string, std::vector<char, std::allocator<char> >, std::vector<short, std::allocator<short> >, std::vector<int, std::allocator<int> >, std::vector<long, std::allocator<long> >, std::vector<unsigned char, std::allocator<unsigned char> >, std::vector<unsigned short, std::allocator<unsigned short> >, std::vector<unsigned int, std::allocator<unsigned int> >, std::vector<unsigned long, std::allocator<unsigned long> >, std::vector<float, std::allocator<float> >, std::vector<double, std::allocator<double> >, std::vector<long double, std::allocator<long double> >, std::vector<std::string, std::allocator<std::string> >, std::array<double, 7ul>, bool> >, std::shared_ptr<std::vector<std::string, std::allocator<std::string> > > > >, std::less<std::string>, std::allocator<std::pair<std::string const, ArgumentDatatype<std::string, std::vector<unsigned long, std::allocator<unsigned long> >, void*, std::shared_ptr<void>, Datatype, boost::variant<char, unsigned char, short, int, long, unsigned short, unsigned int, unsigned long, float, double, long double, std::string, std::vector<char, std::allocator<char> >, std::vector<short, std::allocator<short> >, std::vector<int, std::allocator<int> >, std::vector<long, std::allocator<long> >, std::vector<unsigned char, std::allocator<unsigned char> >, std::vector<unsigned short, std::allocator<unsigned short> >, std::vector<unsigned int, std::allocator<unsigned int> >, std::vector<unsigned long, std::allocator<unsigned long> >, std::vector<float, std::allocator<float> >, std::vector<double, std::allocator<double> >, std::vector<long double, std::allocator<long double> >, std::vector<std::string, std::allocator<std::string> >, std::array<double, 7ul>, bool>, std::shared_ptr<std::vector<unsigned long, std::allocator<unsigned long> > >, std::shared_ptr<Datatype>, std::shared_ptr<boost::variant<char, unsigned char, short, int, long, unsigned short, unsigned int, unsigned long, float, double, long double, std::string, std::vector<char, std::allocator<char> >, std::vector<short, std::allocator<short> >, std::vector<int, std::allocator<int> >, std::vector<long, std::allocator<long> >, std::vector<unsigned char, std::allocator<unsigned char> >, std::vector<unsigned short, std::allocator<unsigned short> >, std::vector<unsigned int, std::allocator<unsigned int> >, std::vector<unsigned long, std::allocator<unsigned long> >, std::vector<float, std::allocator<float> >, std::vector<double, std::allocator<double> >, std::vector<long double, std::allocator<long double> >, std::vector<std::string, std::allocator<std::string> >, std::array<double, 7ul>, bool> >, std::shared_ptr<std::vector<std::string, std::allocator<std::string> > > > > > > const&) (HDF5IOHandler.cpp:167)
==14663== by 0x4D8697: HDF5IOHandlerImpl::flush() (HDF5IOHandler.cpp:94)
==14663== by 0x4E681B: HDF5IOHandler::flush() (HDF5IOHandler.cpp:1434)
==14663== by 0x48824D: Series::flushGroupBased() (Series.cpp:496)
==14663== by 0x487E80: Series::flush() (Series.cpp:454)
==14663== by 0x410C56: write2() (writer.cpp:150)
==14663== by 0x412694: main (writer.cpp:206)
==14663==
==14663== Conditional jump or move depends on uninitialised value(s)
==14663== at 0x510D44A: boost::filesystem::path::filename() const (in /usr/lib/x86_64-linux-gnu/libboost_filesystem.so.1.62.0)
==14663== by 0x5106787: boost::filesystem::detail::create_directories(boost::filesystem::path const&, boost::system::error_code*) (in /usr/lib/x86_64-linux-gnu/libboost_filesystem.so.1.62.0)
==14663== by 0x4E69CD: boost::filesystem::create_directories(boost::filesystem::path const&) (operations.hpp:566)
==14663== by 0x4D8AFA: HDF5IOHandlerImpl::createFile(Writable*, std::map<std::string, Variadic<ArgumentDatatype<std::string, std::vector<unsigned long, std::allocator<unsigned long> >, void*, std::shared_ptr<void>, Datatype, boost::variant<char, unsigned char, short, int, long, unsigned short, unsigned int, unsigned long, float, double, long double, std::string, std::vector<char, std::allocator<char> >, std::vector<short, std::allocator<short> >, std::vector<int, std::allocator<int> >, std::vector<long, std::allocator<long> >, std::vector<unsigned char, std::allocator<unsigned char> >, std::vector<unsigned short, std::allocator<unsigned short> >, std::vector<unsigned int, std::allocator<unsigned int> >, std::vector<unsigned long, std::allocator<unsigned long> >, std::vector<float, std::allocator<float> >, std::vector<double, std::allocator<double> >, std::vector<long double, std::allocator<long double> >, std::vector<std::string, std::allocator<std::string> >, std::array<double, 7ul>, bool>, std::shared_ptr<std::vector<unsigned long, std::allocator<unsigned long> > >, std::shared_ptr<Datatype>, std::shared_ptr<boost::variant<char, unsigned char, short, int, long, unsigned short, unsigned int, unsigned long, float, double, long double, std::string, std::vector<char, std::allocator<char> >, std::vector<short, std::allocator<short> >, std::vector<int, std::allocator<int> >, std::vector<long, std::allocator<long> >, std::vector<unsigned char, std::allocator<unsigned char> >, std::vector<unsigned short, std::allocator<unsigned short> >, std::vector<unsigned int, std::allocator<unsigned int> >, std::vector<unsigned long, std::allocator<unsigned long> >, std::vector<float, std::allocator<float> >, std::vector<double, std::allocator<double> >, std::vector<long double, std::allocator<long double> >, std::vector<std::string, std::allocator<std::string> >, std::array<double, 7ul>, bool> >, std::shared_ptr<std::vector<std::string, std::allocator<std::string> > > > >, std::less<std::string>, std::allocator<std::pair<std::string const, ArgumentDatatype<std::string, std::vector<unsigned long, std::allocator<unsigned long> >, void*, std::shared_ptr<void>, Datatype, boost::variant<char, unsigned char, short, int, long, unsigned short, unsigned int, unsigned long, float, double, long double, std::string, std::vector<char, std::allocator<char> >, std::vector<short, std::allocator<short> >, std::vector<int, std::allocator<int> >, std::vector<long, std::allocator<long> >, std::vector<unsigned char, std::allocator<unsigned char> >, std::vector<unsigned short, std::allocator<unsigned short> >, std::vector<unsigned int, std::allocator<unsigned int> >, std::vector<unsigned long, std::allocator<unsigned long> >, std::vector<float, std::allocator<float> >, std::vector<double, std::allocator<double> >, std::vector<long double, std::allocator<long double> >, std::vector<std::string, std::allocator<std::string> >, std::array<double, 7ul>, bool>, std::shared_ptr<std::vector<unsigned long, std::allocator<unsigned long> > >, std::shared_ptr<Datatype>, std::shared_ptr<boost::variant<char, unsigned char, short, int, long, unsigned short, unsigned int, unsigned long, float, double, long double, std::string, std::vector<char, std::allocator<char> >, std::vector<short, std::allocator<short> >, std::vector<int, std::allocator<int> >, std::vector<long, std::allocator<long> >, std::vector<unsigned char, std::allocator<unsigned char> >, std::vector<unsigned short, std::allocator<unsigned short> >, std::vector<unsigned int, std::allocator<unsigned int> >, std::vector<unsigned long, std::allocator<unsigned long> >, std::vector<float, std::allocator<float> >, std::vector<double, std::allocator<double> >, std::vector<long double, std::allocator<long double> >, std::vector<std::string, std::allocator<std::string> >, std::array<double, 7ul>, bool> >, std::shared_ptr<std::vector<std::string, std::allocator<std::string> > > > > > > const&) (HDF5IOHandler.cpp:167)
==14663== by 0x4D8697: HDF5IOHandlerImpl::flush() (HDF5IOHandler.cpp:94)
==14663== by 0x4E681B: HDF5IOHandler::flush() (HDF5IOHandler.cpp:1434)
==14663== by 0x48824D: Series::flushGroupBased() (Series.cpp:496)
==14663== by 0x487E80: Series::flush() (Series.cpp:454)
==14663== by 0x410C56: write2() (writer.cpp:150)
==14663== by 0x412694: main (writer.cpp:206)
==14663==
==14663== Use of uninitialised value of size 8
==14663== at 0x510D4AA: boost::filesystem::path::filename() const (in /usr/lib/x86_64-linux-gnu/libboost_filesystem.so.1.62.0)
==14663== by 0x5106787: boost::filesystem::detail::create_directories(boost::filesystem::path const&, boost::system::error_code*) (in /usr/lib/x86_64-linux-gnu/libboost_filesystem.so.1.62.0)
==14663== by 0x4E69CD: boost::filesystem::create_directories(boost::filesystem::path const&) (operations.hpp:566)
==14663== by 0x4D8AFA: HDF5IOHandlerImpl::createFile(Writable*, std::map<std::string, Variadic<ArgumentDatatype<std::string, std::vector<unsigned long, std::allocator<unsigned long> >, void*, std::shared_ptr<void>, Datatype, boost::variant<char, unsigned char, short, int, long, unsigned short, unsigned int, unsigned long, float, double, long double, std::string, std::vector<char, std::allocator<char> >, std::vector<short, std::allocator<short> >, std::vector<int, std::allocator<int> >, std::vector<long, std::allocator<long> >, std::vector<unsigned char, std::allocator<unsigned char> >, std::vector<unsigned short, std::allocator<unsigned short> >, std::vector<unsigned int, std::allocator<unsigned int> >, std::vector<unsigned long, std::allocator<unsigned long> >, std::vector<float, std::allocator<float> >, std::vector<double, std::allocator<double> >, std::vector<long double, std::allocator<long double> >, std::vector<std::string, std::allocator<std::string> >, std::array<double, 7ul>, bool>, std::shared_ptr<std::vector<unsigned long, std::allocator<unsigned long> > >, std::shared_ptr<Datatype>, std::shared_ptr<boost::variant<char, unsigned char, short, int, long, unsigned short, unsigned int, unsigned long, float, double, long double, std::string, std::vector<char, std::allocator<char> >, std::vector<short, std::allocator<short> >, std::vector<int, std::allocator<int> >, std::vector<long, std::allocator<long> >, std::vector<unsigned char, std::allocator<unsigned char> >, std::vector<unsigned short, std::allocator<unsigned short> >, std::vector<unsigned int, std::allocator<unsigned int> >, std::vector<unsigned long, std::allocator<unsigned long> >, std::vector<float, std::allocator<float> >, std::vector<double, std::allocator<double> >, std::vector<long double, std::allocator<long double> >, std::vector<std::string, std::allocator<std::string> >, std::array<double, 7ul>, bool> >, std::shared_ptr<std::vector<std::string, std::allocator<std::string> > > > >, std::less<std::string>, std::allocator<std::pair<std::string const, ArgumentDatatype<std::string, std::vector<unsigned long, std::allocator<unsigned long> >, void*, std::shared_ptr<void>, Datatype, boost::variant<char, unsigned char, short, int, long, unsigned short, unsigned int, unsigned long, float, double, long double, std::string, std::vector<char, std::allocator<char> >, std::vector<short, std::allocator<short> >, std::vector<int, std::allocator<int> >, std::vector<long, std::allocator<long> >, std::vector<unsigned char, std::allocator<unsigned char> >, std::vector<unsigned short, std::allocator<unsigned short> >, std::vector<unsigned int, std::allocator<unsigned int> >, std::vector<unsigned long, std::allocator<unsigned long> >, std::vector<float, std::allocator<float> >, std::vector<double, std::allocator<double> >, std::vector<long double, std::allocator<long double> >, std::vector<std::string, std::allocator<std::string> >, std::array<double, 7ul>, bool>, std::shared_ptr<std::vector<unsigned long, std::allocator<unsigned long> > >, std::shared_ptr<Datatype>, std::shared_ptr<boost::variant<char, unsigned char, short, int, long, unsigned short, unsigned int, unsigned long, float, double, long double, std::string, std::vector<char, std::allocator<char> >, std::vector<short, std::allocator<short> >, std::vector<int, std::allocator<int> >, std::vector<long, std::allocator<long> >, std::vector<unsigned char, std::allocator<unsigned char> >, std::vector<unsigned short, std::allocator<unsigned short> >, std::vector<unsigned int, std::allocator<unsigned int> >, std::vector<unsigned long, std::allocator<unsigned long> >, std::vector<float, std::allocator<float> >, std::vector<double, std::allocator<double> >, std::vector<long double, std::allocator<long double> >, std::vector<std::string, std::allocator<std::string> >, std::array<double, 7ul>, bool> >, std::shared_ptr<std::vector<std::string, std::allocator<std::string> > > > > > > const&) (HDF5IOHandler.cpp:167)
==14663== by 0x4D8697: HDF5IOHandlerImpl::flush() (HDF5IOHandler.cpp:94)
==14663== by 0x4E681B: HDF5IOHandler::flush() (HDF5IOHandler.cpp:1434)
==14663== by 0x48824D: Series::flushGroupBased() (Series.cpp:496)
==14663== by 0x487E80: Series::flush() (Series.cpp:454)
==14663== by 0x410C56: write2() (writer.cpp:150)
==14663== by 0x412694: main (writer.cpp:206)
==14663==
==14663== Invalid read of size 1
==14663== at 0x510D4AA: boost::filesystem::path::filename() const (in /usr/lib/x86_64-linux-gnu/libboost_filesystem.so.1.62.0)
==14663== by 0x5106787: boost::filesystem::detail::create_directories(boost::filesystem::path const&, boost::system::error_code*) (in /usr/lib/x86_64-linux-gnu/libboost_filesystem.so.1.62.0)
==14663== by 0x4E69CD: boost::filesystem::create_directories(boost::filesystem::path const&) (operations.hpp:566)
==14663== by 0x4D8AFA: HDF5IOHandlerImpl::createFile(Writable*, std::map<std::string, Variadic<ArgumentDatatype<std::string, std::vector<unsigned long, std::allocator<unsigned long> >, void*, std::shared_ptr<void>, Datatype, boost::variant<char, unsigned char, short, int, long, unsigned short, unsigned int, unsigned long, float, double, long double, std::string, std::vector<char, std::allocator<char> >, std::vector<short, std::allocator<short> >, std::vector<int, std::allocator<int> >, std::vector<long, std::allocator<long> >, std::vector<unsigned char, std::allocator<unsigned char> >, std::vector<unsigned short, std::allocator<unsigned short> >, std::vector<unsigned int, std::allocator<unsigned int> >, std::vector<unsigned long, std::allocator<unsigned long> >, std::vector<float, std::allocator<float> >, std::vector<double, std::allocator<double> >, std::vector<long double, std::allocator<long double> >, std::vector<std::string, std::allocator<std::string> >, std::array<double, 7ul>, bool>, std::shared_ptr<std::vector<unsigned long, std::allocator<unsigned long> > >, std::shared_ptr<Datatype>, std::shared_ptr<boost::variant<char, unsigned char, short, int, long, unsigned short, unsigned int, unsigned long, float, double, long double, std::string, std::vector<char, std::allocator<char> >, std::vector<short, std::allocator<short> >, std::vector<int, std::allocator<int> >, std::vector<long, std::allocator<long> >, std::vector<unsigned char, std::allocator<unsigned char> >, std::vector<unsigned short, std::allocator<unsigned short> >, std::vector<unsigned int, std::allocator<unsigned int> >, std::vector<unsigned long, std::allocator<unsigned long> >, std::vector<float, std::allocator<float> >, std::vector<double, std::allocator<double> >, std::vector<long double, std::allocator<long double> >, std::vector<std::string, std::allocator<std::string> >, std::array<double, 7ul>, bool> >, std::shared_ptr<std::vector<std::string, std::allocator<std::string> > > > >, std::less<std::string>, std::allocator<std::pair<std::string const, ArgumentDatatype<std::string, std::vector<unsigned long, std::allocator<unsigned long> >, void*, std::shared_ptr<void>, Datatype, boost::variant<char, unsigned char, short, int, long, unsigned short, unsigned int, unsigned long, float, double, long double, std::string, std::vector<char, std::allocator<char> >, std::vector<short, std::allocator<short> >, std::vector<int, std::allocator<int> >, std::vector<long, std::allocator<long> >, std::vector<unsigned char, std::allocator<unsigned char> >, std::vector<unsigned short, std::allocator<unsigned short> >, std::vector<unsigned int, std::allocator<unsigned int> >, std::vector<unsigned long, std::allocator<unsigned long> >, std::vector<float, std::allocator<float> >, std::vector<double, std::allocator<double> >, std::vector<long double, std::allocator<long double> >, std::vector<std::string, std::allocator<std::string> >, std::array<double, 7ul>, bool>, std::shared_ptr<std::vector<unsigned long, std::allocator<unsigned long> > >, std::shared_ptr<Datatype>, std::shared_ptr<boost::variant<char, unsigned char, short, int, long, unsigned short, unsigned int, unsigned long, float, double, long double, std::string, std::vector<char, std::allocator<char> >, std::vector<short, std::allocator<short> >, std::vector<int, std::allocator<int> >, std::vector<long, std::allocator<long> >, std::vector<unsigned char, std::allocator<unsigned char> >, std::vector<unsigned short, std::allocator<unsigned short> >, std::vector<unsigned int, std::allocator<unsigned int> >, std::vector<unsigned long, std::allocator<unsigned long> >, std::vector<float, std::allocator<float> >, std::vector<double, std::allocator<double> >, std::vector<long double, std::allocator<long double> >, std::vector<std::string, std::allocator<std::string> >, std::array<double, 7ul>, bool> >, std::shared_ptr<std::vector<std::string, std::allocator<std::string> > > > > > > const&) (HDF5IOHandler.cpp:167)
==14663== by 0x4D8697: HDF5IOHandlerImpl::flush() (HDF5IOHandler.cpp:94)
==14663== by 0x4E681B: HDF5IOHandler::flush() (HDF5IOHandler.cpp:1434)
==14663== by 0x48824D: Series::flushGroupBased() (Series.cpp:496)
==14663== by 0x487E80: Series::flush() (Series.cpp:454)
==14663== by 0x410C56: write2() (writer.cpp:150)
==14663== by 0x412694: main (writer.cpp:206)
==14663== Address 0x10079f34a7 is not stack'd, malloc'd or (recently) free'd
==14663==
==14663==
==14663== Process terminating with default action of signal 11 (SIGSEGV)
==14663== Access not within mapped region at address 0x10079F34A7
==14663== at 0x510D4AA: boost::filesystem::path::filename() const (in /usr/lib/x86_64-linux-gnu/libboost_filesystem.so.1.62.0)
==14663== by 0x5106787: boost::filesystem::detail::create_directories(boost::filesystem::path const&, boost::system::error_code*) (in /usr/lib/x86_64-linux-gnu/libboost_filesystem.so.1.62.0)
==14663== by 0x4E69CD: boost::filesystem::create_directories(boost::filesystem::path const&) (operations.hpp:566)
==14663== by 0x4D8AFA: HDF5IOHandlerImpl::createFile(Writable*, std::map<std::string, Variadic<ArgumentDatatype<std::string, std::vector<unsigned long, std::allocator<unsigned long> >, void*, std::shared_ptr<void>, Datatype, boost::variant<char, unsigned char, short, int, long, unsigned short, unsigned int, unsigned long, float, double, long double, std::string, std::vector<char, std::allocator<char> >, std::vector<short, std::allocator<short> >, std::vector<int, std::allocator<int> >, std::vector<long, std::allocator<long> >, std::vector<unsigned char, std::allocator<unsigned char> >, std::vector<unsigned short, std::allocator<unsigned short> >, std::vector<unsigned int, std::allocator<unsigned int> >, std::vector<unsigned long, std::allocator<unsigned long> >, std::vector<float, std::allocator<float> >, std::vector<double, std::allocator<double> >, std::vector<long double, std::allocator<long double> >, std::vector<std::string, std::allocator<std::string> >, std::array<double, 7ul>, bool>, std::shared_ptr<std::vector<unsigned long, std::allocator<unsigned long> > >, std::shared_ptr<Datatype>, std::shared_ptr<boost::variant<char, unsigned char, short, int, long, unsigned short, unsigned int, unsigned long, float, double, long double, std::string, std::vector<char, std::allocator<char> >, std::vector<short, std::allocator<short> >, std::vector<int, std::allocator<int> >, std::vector<long, std::allocator<long> >, std::vector<unsigned char, std::allocator<unsigned char> >, std::vector<unsigned short, std::allocator<unsigned short> >, std::vector<unsigned int, std::allocator<unsigned int> >, std::vector<unsigned long, std::allocator<unsigned long> >, std::vector<float, std::allocator<float> >, std::vector<double, std::allocator<double> >, std::vector<long double, std::allocator<long double> >, std::vector<std::string, std::allocator<std::string> >, std::array<double, 7ul>, bool> >, std::shared_ptr<std::vector<std::string, std::allocator<std::string> > > > >, std::less<std::string>, std::allocator<std::pair<std::string const, ArgumentDatatype<std::string, std::vector<unsigned long, std::allocator<unsigned long> >, void*, std::shared_ptr<void>, Datatype, boost::variant<char, unsigned char, short, int, long, unsigned short, unsigned int, unsigned long, float, double, long double, std::string, std::vector<char, std::allocator<char> >, std::vector<short, std::allocator<short> >, std::vector<int, std::allocator<int> >, std::vector<long, std::allocator<long> >, std::vector<unsigned char, std::allocator<unsigned char> >, std::vector<unsigned short, std::allocator<unsigned short> >, std::vector<unsigned int, std::allocator<unsigned int> >, std::vector<unsigned long, std::allocator<unsigned long> >, std::vector<float, std::allocator<float> >, std::vector<double, std::allocator<double> >, std::vector<long double, std::allocator<long double> >, std::vector<std::string, std::allocator<std::string> >, std::array<double, 7ul>, bool>, std::shared_ptr<std::vector<unsigned long, std::allocator<unsigned long> > >, std::shared_ptr<Datatype>, std::shared_ptr<boost::variant<char, unsigned char, short, int, long, unsigned short, unsigned int, unsigned long, float, double, long double, std::string, std::vector<char, std::allocator<char> >, std::vector<short, std::allocator<short> >, std::vector<int, std::allocator<int> >, std::vector<long, std::allocator<long> >, std::vector<unsigned char, std::allocator<unsigned char> >, std::vector<unsigned short, std::allocator<unsigned short> >, std::vector<unsigned int, std::allocator<unsigned int> >, std::vector<unsigned long, std::allocator<unsigned long> >, std::vector<float, std::allocator<float> >, std::vector<double, std::allocator<double> >, std::vector<long double, std::allocator<long double> >, std::vector<std::string, std::allocator<std::string> >, std::array<double, 7ul>, bool> >, std::shared_ptr<std::vector<std::string, std::allocator<std::string> > > > > > > const&) (HDF5IOHandler.cpp:167)
==14663== by 0x4D8697: HDF5IOHandlerImpl::flush() (HDF5IOHandler.cpp:94)
==14663== by 0x4E681B: HDF5IOHandler::flush() (HDF5IOHandler.cpp:1434)
==14663== by 0x48824D: Series::flushGroupBased() (Series.cpp:496)
==14663== by 0x487E80: Series::flush() (Series.cpp:454)
==14663== by 0x410C56: write2() (writer.cpp:150)
==14663== by 0x412694: main (writer.cpp:206)
==14663== If you believe this happened as a result of a stack
==14663== overflow in your program's main thread (unlikely but
==14663== possible), you can try to increase the size of the
==14663== main thread stack using the --main-stacksize= flag.
==14663== The main thread stack size used in this run was 8388608.
==14663==
==14663== HEAP SUMMARY:
==14663== in use at exit: 243,674 bytes in 2,984 blocks
==14663== total heap usage: 3,463 allocs, 479 frees, 436,960 bytes allocated
==14663==
==14663== LEAK SUMMARY:
==14663== definitely lost: 0 bytes in 0 blocks
==14663== indirectly lost: 0 bytes in 0 blocks
==14663== possibly lost: 0 bytes in 0 blocks
==14663== still reachable: 243,674 bytes in 2,984 blocks
==14663== of which reachable via heuristic:
==14663== stdstring : 2,372 bytes in 64 blocks
==14663== suppressed: 0 bytes in 0 blocks
==14663== Rerun with --leak-check=full to see details of leaked memory
==14663==
==14663== For counts of detected and suppressed errors, rerun with: -v
==14663== Use --track-origins=yes to see where uninitialised values come from
==14663== ERROR SUMMARY: 4 errors from 4 contexts (suppressed: 0 from 0)
Segmentation faultAdhere to CRP C++ Style Guide
https://github.com/ComputationalRadiationPhysics/contributing/blob/master/codingGuideLines/cpp.md
What this project does not do:
- doxygen
@file ...documentation - wrap at line 80 (we do 120-ish)
- no code alignment
- indentation inside the deepest namespace
- spaces around parameters in single-argument functions
- trailing return type declaration (for lambdas, AFAIK
autois never used for function defs) - function/method specifier on newline
- newline after and before
(/)of function signature - parent classes are indented and placed on a new line
- constructor
:on same line (we do newline)
What this project does not do consistently:
- copyright notice in every file
- namespace structure (per dir, see #82)
- lowercase foldernames (e.g.
IO,ADIOS,HDF5) - indent of 4
- camelcase (some non-exposed internal functions use underscores, on purpose)
- closing of multiline comments on newline
- spaces around binary operators
- newline on each
,in function signatures - single spaces around
&(reference) and*(pointer) (most often the left one is omitted) - initialization over assignment
- indented
casestatements - prefix
T_for template parameters - case for template names
- each template parameter is on a separate line
- spaces around parameters in template instantiation (we try hard, but might have missed a few)
Example: MPI Benchmark
We need to add a parallel read & write MPI benchmark at some point, that can be scaled with ranks e.g. for 3D meshes and large particle records.
It can then measure performance in terms of bandwidth, etc.
Move to openPMD org
When the first examples build, move the repo to github.com/openPMD and fork back
Fix Clang Warnings
We should fix all the clang (5.0) warnings that are currently raised in the code.
We should try to aim to be warning free with the following compiler flags:
export CXXFLAGS="-Wall -Wextra -Woverloaded-virtual -Wshadow"HighFive: C++ API HDF5
TIL: https://github.com/BlueBrain/HighFive
A third-party, C++(11) API for HDF5, even supporting parallel I/O :)
(Note: we stay with the official C-API for our HDF5 backend now. But if we want to try this C++ library, we would write a second HighFive backend for it.)
We currently implement serial & parallel HDF5 via the official HDF5 C-API.
Additionally, we could implement another HDF5 backend via HighFive's C++ API.
Docs: Add RTD
load/storeChunk: slice view
The A.storeChunk(data, offset, extent) and A.loadChunk(data, chunk_offset, chunk_extent) methods should for syntactic sugar provide a slice-view access of the form:
// write (copy)
A[view] = data;
// returns: std::unique_ptr< A.dtype >
auto B = A[view];
// hm, maybe that's not possible, so let's say something like
auto B = A[view].astype< some_dtype >();Add README.rst
add a readme.rst
HDF5 Closing: uninitialised byte(s)
Running valgrind on the SerialIOTests reviles uninitialised bytes in HDF5 close.
Maybe check what's given into HDF5IOHandlerImpl::~HDF5IOHandlerImpl() in file HDF5IOHandler.cpp:69?
Note: I did run the binary without downloaded sample files.
valgrind bin/SerialIOTests
# [...]
==14254== Syscall param write(buf) points to uninitialised byte(s)
==14254== at 0x6E75190: __write_nocancel (syscall-template.S:84)
==14254== by 0x5B2DA35: ??? (in /usr/lib/x86_64-linux-gnu/libhdf5_openmpi.so.100.0.1)
==14254== by 0x5B277CB: H5FD_write (in /usr/lib/x86_64-linux-gnu/libhdf5_openmpi.so.100.0.1)
==14254== by 0x5B0E94F: H5F__accum_flush (in /usr/lib/x86_64-linux-gnu/libhdf5_openmpi.so.100.0.1)
==14254== by 0x5B0A15B: H5F_flush (in /usr/lib/x86_64-linux-gnu/libhdf5_openmpi.so.100.0.1)
==14254== by 0x5B0A82A: H5F_dest (in /usr/lib/x86_64-linux-gnu/libhdf5_openmpi.so.100.0.1)
==14254== by 0x5B0AF2C: H5F_try_close (in /usr/lib/x86_64-linux-gnu/libhdf5_openmpi.so.100.0.1)
==14254== by 0x5B0B180: H5F_close (in /usr/lib/x86_64-linux-gnu/libhdf5_openmpi.so.100.0.1)
==14254== by 0x5B83BEE: H5I_dec_ref (in /usr/lib/x86_64-linux-gnu/libhdf5_openmpi.so.100.0.1)
==14254== by 0x5B83D7B: H5I_dec_app_ref (in /usr/lib/x86_64-linux-gnu/libhdf5_openmpi.so.100.0.1)
==14254== by 0x5B05898: H5Fclose (in /usr/lib/x86_64-linux-gnu/libhdf5_openmpi.so.100.0.1)
==14254== by 0x5D526A: HDF5IOHandlerImpl::~HDF5IOHandlerImpl() (HDF5IOHandler.cpp:69)
==14254== Address 0x8bb3d4a is 2,490 bytes inside a block of size 4,104 alloc'd
==14254== at 0x4C2BBAF: malloc (vg_replace_malloc.c:299)
==14254== by 0x5B3170C: H5FL_blk_malloc (in /usr/lib/x86_64-linux-gnu/libhdf5_openmpi.so.100.0.1)
==14254== by 0x5B322F3: H5FL_blk_realloc (in /usr/lib/x86_64-linux-gnu/libhdf5_openmpi.so.100.0.1)
==14254== by 0x5B0DAB4: ??? (in /usr/lib/x86_64-linux-gnu/libhdf5_openmpi.so.100.0.1)
==14254== by 0x5B0F35F: H5F__accum_write (in /usr/lib/x86_64-linux-gnu/libhdf5_openmpi.so.100.0.1)
==14254== by 0x5B12509: H5F_block_write (in /usr/lib/x86_64-linux-gnu/libhdf5_openmpi.so.100.0.1)
==14254== by 0x5AADDE7: H5C__flush_single_entry (in /usr/lib/x86_64-linux-gnu/libhdf5_openmpi.so.100.0.1)
==14254== by 0x5AAEF24: H5C_flush_cache (in /usr/lib/x86_64-linux-gnu/libhdf5_openmpi.so.100.0.1)
==14254== by 0x5A86F1C: H5AC_flush (in /usr/lib/x86_64-linux-gnu/libhdf5_openmpi.so.100.0.1)
==14254== by 0x5B0A100: H5F_flush (in /usr/lib/x86_64-linux-gnu/libhdf5_openmpi.so.100.0.1)
==14254== by 0x5B0A82A: H5F_dest (in /usr/lib/x86_64-linux-gnu/libhdf5_openmpi.so.100.0.1)
==14254== by 0x5B0AF2C: H5F_try_close (in /usr/lib/x86_64-linux-gnu/libhdf5_openmpi.so.100.0.1)
==14254==
# [...]Docs: Sphinx, Doxygen & Breathe
Add Sphinx, Doxygen & Breathe
ParalleIOTest failing
@C0nsultant currently, the ParalleIOTest.cpp does not compile.
Issues are:
- access to the
Series()object - changed API of the
loadChunk()member function.
I think the first issue is solvable via
diff --git a/test/ParallelIOTest.cpp b/test/ParallelIOTest.cpp
index cef531f..b45c9e6 100755
--- a/test/ParallelIOTest.cpp
+++ b/test/ParallelIOTest.cpp
@@ -35,9 +35,9 @@ BOOST_AUTO_TEST_CASE(git_hdf5_sample_content_test)
MPI_Comm_rank(MPI_COMM_WORLD, &mpi_rank);
/* only a 3x3x3 chunk of the actual data is hardcoded. every worker reads 1/3 */
uint64_t rank = mpi_rank % 3;
- Series o = Series("../samples/git-sample/",
- "data00000%T.h5",
- true);
+ Series o = Series::read("../samples/git-sample/",
+ "data00000%T.h5",
+ true);
{Can you please provide a PR that fixes the test?
loadChunk: std::future?
We could .loadChunk() let return a std::future which would make it on "simple use" (not using the result val -> destroyed) a blocking operation and on extended use (using the future) a nice C++11-ish object.
Any comments?
The question is then still how to bind the async functionality to Python...
Create spack build package
Pickle Support
Add pickle support to user-facing classes and states.
We have to check if that is entirely possible and if things like file-handles need to be restored/properly shut down/etc. Probably depends on #171 as well.
Writing and Reading Non-openPMD Infos
It would be necessary to allow storing non-openPMD information as well.
In the simple case, this means adding additional, user-defined attributes to records, record-components and paths.
In the more complex case, application specific states such as RNG states, unqiue-number-generator states, etc. need to be stored as records outside of the openPMD-interpreted paths.
CMake target: openPMD
We currently build two "library" targets, core and io.
We should add a general "openPMD" target (with associated libopenPMD.[a|so]) so we do not confuse people.
MPI_INFO_ENV
Currently the parallel test fails due to a missing MPI_INFO_ENV in mpi.h.
Please be aware this is an MPI 3.0+ feature:
https://www.open-mpi.org/doc/v2.0/man3/MPI_Info_env.3.php
While requesting MPI 2.3+ support is very common, MPI 3+ is not necessarily yet common (standard: 2012) but might be acceptable nowadays. This just means our OpenMPI/MPICH/... dependencies need to be "quite recent" (in HPC terms).
Affects issues in:
src/IO/HDF5/ParallelHDF5IOHandler.cpp
Docs: How to Write a Backend
Write a document documenting how to implement and what the concepts of a (new) backend are.
The according file to modify is:
docs/source/dev/backend.rst
CTest: Add targets
Add test targets to CMakeLists.txt so one can directly build & run the tests with make test
Travis-CI
Add travis CI
Test cases with 0-sized reads & writes
To verify the desired parallel behaviour, we should add tests where multiple participants from an MPI-communicator contribute 0-sized chunks to a collective IO operation.
A minimal example would consist of 3 ranks with multiple tested combinations of empty/non-empty chunks.
C++ code style & static code analysis in CI
It would be desirable to also have multiple checks for code quality, code style and static analysis of the code, independent of the compiler.
Possible code style checkers:
- cpplint (Google)
- KWStyle (KitWare)
- vera++
clang-format#291
Possible code analysis tools:
- OCLint
- Flint++
Clang-Tidy#291- clang-analyzer /
scan-build include_what_you_use#291- Cppcheck
- Codacy (post-commit hook for Github integration)
- Coverity
Test / Code coverage:
Docs: Example Codes
Add compiling, self-contained examples for
- serial read
- serial write
- parallel read
- parallel write
Add them in the manual and CI.
Please keep them as short and to the point as possible and place them in examples/ with meaningful names such as serial-reader.cpp, parallel-writer.cpp, etc.
Then soft-link and include those examples in docs/source/usage/serial.rst (parallel.rst)
Ext: Species Type
Add support to define a particle species to each record.
This includes particle and mesh records: openPMD/openPMD-standard#180
NetCDF backend
A NetCDF backend would be useful for FermiLab & SLAC codes.
Note: NetCDF4 can use HDF5 underneath.
Note: NetCDF also supports serial & parallel I/O.
Resources: how to write a backend
Parallel: Offset Namings
@C0nsultant namings parameters of offsets of MPI-ranks in parallel writes can be tricky.
Ideally, stick as close as possible to the openPMD namings in parameters.
If anything is missing, here is how we call offsets in PIConGPU to fill the rest. (The old libSplash namings are outdated and inconsistent, don't read those please.)
Series Constructor: Path & File?
Do we really need both a path and a file (prefix) attribute in the (Series) constructors?
Probably it is also possible just to pass one string and split to basename with boost, similar to yt and openPMD-viewer.
Rename Variadic to Variant
The generic storage used for e.g. Attribute values was initially designed to store arbitrary datatypes, hence the name Variadic was chosen.
During developement it became pretty obvious that a restriction to a finite, predefined set of datatypes is the more sensible way. Since boost/mpark/std variants do exactly that, they have been used all along.
I propose to rename the auxiliary class Variadic to Variant to reflect this decision.
Repo Name: openPMD-api
In consistency with the other repos in the openPMD organization we should rename this repo to, e.g. openPMD-api
openPMD 1.1.0 Support: optional paths
As a minor improvement of the 1.0.X series of openPMD, the meshesPath and particlesPath are not optional and interpreted as "no meshes/particles" if missing. We need to add this logic in both writers and readers.
Changelog of openPMD 1.1.0:
https://github.com/openPMD/openPMD-standard/releases/tag/1.1.0
Add LGPLv3+ Headers
Add LGPLv3+ Headers to files with copyright 2017 & file author
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.