Comments (4)
cc'ing @PrometheusPi, @ax3l
from adios.
Minimal "working" example, e.g. with the file from ax3l's issue #69
import adios as ad
f = ad.file("data_compressed.bp")
f["xy"][0, :6:2] == f["xy"][0, :3]
gives [True, True, True].
I think this should be considered a bug. The proper way of handling this should be raising a ValueError
or a NotImplementedError
(if it's possible that adios will support stepping at some time - which could be useful if you do not think about performance but the array does not fit into RAM - which was the case for Marco).
The current implementation masks the problem, since the shape of the returned array is as expected, so in the worst case a user just doesn't notice that he got the wrong data!
A fix in python (the wrapper is not written in python, is it? I will post some pseudocode anyway^^) would be:
def __getitem__(self, key):
if isinstance(key, slice) and slice.step is not None:
raise NotImplementedError
return super().__getitem__(key)
from adios.
@n01r @ax3l @wmles we are currently supporting in ADIOS2 the Python low-level and high-level APIs. You can see then in action using Jupyter Notebooks running on MyBinder (see ReadMe): https://github.com/ornladios/ADIOS2-Jupyter
A few things about high-level APIs:
- We don't have slicing/stride numpy syntax, high-level APIs are write/read based and are selection/random access (
np_array = read("var", start, count, step_start, step_count)
) and/or step-by-step based (usingfor step in stream
). As in the spirit of Python we heavily overload these 2 functions, but read always returns a numpy array. For examples see:
https://github.com/ornladios/ADIOS2/blob/master/testing/adios2/bindings/python/TestBPWriteTypesHighLevelAPI.py - We are maintaining these APIs and potentially adding more features. e.g. compression: https://github.com/ornladios/ADIOS2/blob/master/testing/adios2/bindings/python/TestBPZfpHighLevelAPI.py
Let me know if this is something of interest, ADIOS2 readers should be able to handle ADIOS1.x generated files. Thanks!
from adios.
You really do not want to stride during reads anyway for performance. Read all into a temporary var and stride in RAM :)
from adios.
Related Issues (20)
- ADIOS_READ_METHOD_BP_AGGREGATE not working in read API HOT 1
- bpdump crash on attribute file written by machine of different en HOT 4
- runconfig fail on theta@alcf HOT 3
- python 3 support broken by print statements HOT 1
- SZ transformation build error HOT 2
- adios_mpi.cpp doesn't work with Python 3.7 HOT 2
- Support ZFP 0.5.1+ HOT 2
- Possible bug in bp2ncd utility HOT 1
- GFortran 10 Argument Mismatch HOT 2
- Question about timers HOT 4
- ADIOS Spack package broken on Summit HOT 5
- ZFP_INCLUDE_DIR not set HOT 1
- Compilation error HOT 6
- Attribute of Array of Complex
- Python DeprecationWarning: np.asscalar(a) is deprecated since NumPy v1.16, use a.item() instead HOT 1
- Python: wrapper of adios.File close errors
- Python 3.9 Build Error HOT 15
- Bug writing empty data sets? HOT 13
- Subsetting Data for in situ visualization HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from adios.