Comments (11)
I want to implement a simple set of wrappers that use np.loadtxt and np.savetxt to load and save txt, dat, csv format files. Mostly metadata agnostic of course.
The issue with having a generic io plugin for txt/csv files is that there is no specification and therefore the plugin can't do much:
- loading data is trivial
- nothing to be done with restructuring the data
- not possible to set axes
- not possible to add metadata.
It would be helpful to import/transfer data from systems where the native file system is so far not implemented in hyperspy (e.g. Jobim Yvon spectrometer software or systems with self-programmed data collection), but a text format export is possible.
If there are specifications or it follows a specific know pattern, then this is different from the case above and, it makes sense to have a plugin - see for example msa
.
from lumispy.
Having examples would be good, see
https://github.com/hyperspy/hyperspy/pull/2306/files#diff-82b6f06aa6e3647b596d6dc24789557125d85d87ce203ad9a660ffc5c5ec78f4
from lumispy.
I see your point with the 'specs' and it is true that for single spectra msa
does the job. At least for saving I have my wrapper routines in a local script and think a generalized version could be nice to have in lumispy for others to use.
For the examples, I'll maybe add one for reading linescan data in txt format.
from lumispy.
I agree that we need some sort of function to integrate better with the Origin crowd.
Gunnar (here in Cambridge) has asked me for this functionality many times, as collaborators may not use Python.
What about having some general functions in the luminescence class to export the data? I envision some very simple function that simply takes the s.data
and the s.axes_manager.signal_axes[0].axis
and puts it in a simple .csv
file that Origin can load easily?
Would that be fine? Or are you thinking something more complex?
And as for the "examples", would it make sense to have it in the lumispy-demos
section? Or within the lumispy
package?
from lumispy.
I can contribute such a function, as I basically have some functions doing the job. I would just generalize them to decide the format based on the dimensions within the function instead of having several different functions:
- Spectrum: simple
.csv
(with options to choose the delimeter, maybe also to add a simple header) - Linescan: signal axis as first column and navigation axis as first row
- 2D navigation, 0D signal (e.g. single attributes from a fit): simple matrix (with option to put axes in first row and column)
- 2D: Could in principle be represented by consecutive matrices. Do we need that? It is usually no fun to handle multidimensional data in Origin, so up to now I did not pass that type of data to anyone in a text format. I would just raise an error if dimensions exceed the three cases above.
I'll do it for LumiSpectrum
for the moment. For LumiTransient
, I would create a separate function eventually (alternative would be to generalize the above function and then put it in CommonLumi
, but I think a separate function would be clearer).
- Basically, a single point dataset could be represented the same way as a spectral linescan above
- It could be interesting to save fit results, where one or multiple lifetimes are added to a simple
.csv
file - Anything of higher dimension
from lumispy.
I think the "examples" was more as an alternative for the HyperSpy case, where one could add an example for linescans. I think that is independent of whether we implement it here.
Apart from that, I would eventually include this functionality in one of the demos on lumispy-demos
.
from lumispy.
I'm happy for you to go ahead.
I also have some code for the 2D fitting results to pandas arrays. But please go ahead.
I absolutely agree that Origin is a big no for more than linescans. So just raise an error indeed.
Let me know if you want me to help out with this eventually.
from lumispy.
As a comment, before you go too deep into certain directions. Origin can without problems load hdf5 files and is how I am handling data export at the moment in my local scripts for extracting spot spectra. hdf5 in this case would have the advantage that you can just append more and more points to it (should make a linescan easy) and can at least theoretically store metadata (I haven't had time to go that much into detail on the hdf5 file format...which is mre or less my story with all of this, not having enough time). In case I am not completely misreading what you would like, I am happy to send you the bit I wrote @jordiferrero for you to look at and potentially implement here.
from lumispy.
I often have colleagues who want the data in a simple text format, that is why I have written some small functions. Nothing elaborate, but the main point is in putting the different vectors and the data matrix together the correct way.
I can put together a rough proposal and let you comment along the way before adding all the nice-to-have details such as choosing the delimiter.
If Origin can handle hdf5, all the better, as it is the native format of hyperspy so no specific functions needed for export, I guess - but I fear there are some colleagues who will still prefer a simple text format.
from lumispy.
I agree with both of you.
Then I wait for your proposal and will comment on that.
from lumispy.
See #56 for a the idea of implementation.
from lumispy.
Related Issues (20)
- Docstring formatting in RTD HOT 11
- Transient signal classes HOT 3
- LumiSpy v0.2 HOT 2
- Create a `remove_background_signal` function HOT 1
- Extend functionality of crop_edges HOT 3
- Implementation of Spectroscopy File Readers HOT 18
- Increase coverage HOT 2
- Webhook failing HOT 3
- Adding an interactive way to slice HS over a wavelength range, and view the result spatially! HOT 12
- Consolidate axis conversion codes
- Failure with numpy 1.24.dev
- Slicing of energy/wavenumber signal with isig fails
- Failure with numpy dev HOT 1
- Documentation is broken HOT 3
- Azure tests failing HOT 2
- Find maximum and width of a peak HOT 1
- Reminder: Change doc-links to sphinx
- Fix test for `remove_spikes` HOT 4
- Slicing TransientSpec HOT 22
- `to_eV()` function and non-uniform axis HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from lumispy.