openpmd / openpmd-converter Goto Github PK

Merge our old tools for splash2xdmf and pic2xdmf (usage see PIConGPU wiki) to a unified tool for openPMD files.

A specific tool for the ED-PIC extension might not be necessary since all definitions of the base standard stay valid (e.g., positions, etc.).

I recently needed a small script that converted our usual "fileBased" iterationEncoding to a single "groupBased" file (in HDF5).

Next time I need it I will rewrite it in pure python so it's more portable. (Actually, I tried pure bash but needed a small in-file manipulation which I found no CLI tool for).

Nevertheless, here it is already:

#!/usr/bin/env bash
#
# Copyright 2017 Axel Huebl
#
# Use under the ISC software license.
#

# needs input argument parsing & help

dumps=($(ls h5/*_*.h5))
first=0

for d in ${dumps[@]};
do
    if [ $first -eq 0 ]
    then
        # includes copy of / attributes from first file
        cp $d simData.h5
        first=1
    else
        step=$(echo $d | sed 's/.\+_\([0-9]\+\)\.h5/\1/')
        h5copy -i $d -s "/data/$step" \
               -o simData.h5 -d "/data/$step"
    fi
done

# that might not be necessary if your files already had compression enabled
h5repack -i simData.h5 -o simData.h5.gz -f GZIP=1 && \
    mv simData.h5.gz simData.h5

# change attributes:
#   "/iterationEncoding" -> "groupBased"
#   "/iterationFormat" -> "/data/%T/"
python -c 'import h5py; import numpy as np; f=h5py.File("simData.h5", "a"); f.attrs["iterationEncoding"] = np.string_("groupBased"); f.attrs["iterationFormat"] = np.string_("/data/%T/"); f.close()'

How do we compare to http://cfconventions.org/ ?

We are file format agnostic.
The openPMD base standard is (scientific) domain and application agnostic.
We do not yet support unstructured meshes, besides expressing them as particles (dataframes).

I think the title of the README should be changed to "openPMD tools" instead of "openPMD Validator scripts"

https://en.wikipedia.org/wiki/OpenVDB:

OpenVDB is an open source software library for working with sparse volumetric data. It provides a hierarchical data structure and related functions to help with calculating volumetric effects in CGI applications. Volumetric effects apply to volumes, as opposed to just on surfaces. An example is fog.

Specifically catering for feature film production, the library was originally developed by DreamWorks Animation and is currently maintained by the Academy Software Foundation (ASWF). The primary authors are Ken Museth, Peter Cucka, Mihai Aldén, and David Hill. OpenVDB is written in C++ and has Python bindings.

OpenVDB is supported in a wide range of CGI software, such as Blender (since April 2016), Cinema 4D, Houdini, and RenderMan. It was used in the films Puss in Boots (2011) and Rise of the Guardians (2012). 

https://www.openvdb.org
https://academysoftwarefoundation.github.io/openvdb/
https://github.com/AcademySoftwareFoundation/openvdb

A tool that converts openPMD files from hdf5 to adios and vice versa would be useful since:

• specific pre- and post-processing tool might only work with one or the other
• some PIC codes (and others) might only have implemented one output format thus this would ease interchange of openPMD files between codes (that only support one version)

As an example:
I have implemented an electron bunch initialization that only works with hdf5 and writes out a openPMD checkpoint that PIConGPU can use to continue simulating the bunch evolution. However since the simulation is quite large and memory consuming, adios would be a better choice for file IO and using hdf5 comes with a performance detriment. Thus, I have right now three options: (1) update my code to the latest dev where PIConGPU supports mixed output formats, (2) include adios as output to my pre-processing tool, or (3) convert the generated hdf5 to adios before using PIConGPU with adios only.

However, since such a conversion tool might be useful for more users, my question is whether you @RemiLehe and @ax3l think it is useful too and if you think a conversion between openPMD hdf5 and openPMD adios is possible in general.

@ax3l i sent a PR to this repo. about the SDF to openPMD project. If there is some problem to merge, please tell me.

VizSchema - TechX: HDF5 (NetCDF?)

Add a script similar to #1 that automatically adds the markup for VizSchema (vsAttributes).

(Not all documentation and repositories of VizSchema are public...)

This is not the same as:

ADIOS Visualization Schema

Visualization Schema