Just as a heads up.

A couple tests from one test suite (the one where histogramnd is compared to histogramdd) can sometimes fail due to some precision error (when using np.all_close when comparing both results).
This suite was only added in order to test histogramnd with large data sets (everything is pretty much already covered by the other tests, i think, but with very small hand written data sets).

The failure rate is low (as far as I know), but still...

Possible solutions :

• at the very least print the state of the random number generator (RNG) when there is a fail so we can reproduce the result (@kif's suggestion)
• remove those tests from the automatically executed tests (run them manualy when needed)
• generate a data set that works, manualy validate the results, and then provide that data with the tests (or actually just the RNG state)

Currently, if setup.py is called with the environment variable WITH_CYTHON set to

• True : setup doesn't use cython, the pre-generated C files are used.
• False : setup uses cython

Judging by the code (the other lines all set WITH_CYTHON to False when not using cython) this should probably be the opposite.

I did not find a proper way to run the test suite using only installed modules.

I found I only could execute silx.test.test_version when I was expecting something like silx.test.test_all or similar to implement the equivalent of the run_tests.py script found with the sources.

Did I miss something?

This fails :

#S 1  timescan 1 0
#D Tue May 03 17:13:44 2016
#T 1  (Seconds)
#G0 0
#G1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
#G3 0 0 0 0 0 0 0 0 0
#G4 0
#Q 
#P0  0 0 0.2 47.4 3.0905 1100 -0.553
#P1 11.831251 0 333.66188 -78.575001 -1896.0207 280.76377 3.37 60
#P2 122.78406 200 0 260 0 0 18.99 2.5e-05
#P3 9.9149625 6.25e-07 6.25e-07 2.845 61.692125 61.074125 
#UMI0     Current AutoM      Shutter     PPU13A_GAP W150B_GAP W150B_TAPER U17-6c_GAP
#UMI1       -0.01   OFF   FE closed        200.0000  260.0000      0.0000    19.0000
#UMI2 Refill in 0 sec, Fill Mode: 16 bunch / Message: May  3 09:34 MDT; Next injection at 09:40
#N 14
#L Time  Epoch  IC  avg  std  integ  VM_1  VM_2  T_simap  Storage Ring A  Single Bunch A  pico  Monitor  Seconds
#C Tue May 03 17:14:03 2016.  Scan aborted after 0 points.

Refactor existing PyMca module.

Problem

• Specfile build crashes under windows

Solution

• When dealing with multiplatform developments, one has to take care and systematically use os.path.join to build paths.

In at least 1 CI system:

• Building using sdist generated tarball.
• Building without Cython installed.

See PR #10

@dnaudet

With a python setup.py install, the generated histogramnd.c code is overwritten and git complains.

The default file should be provided in a default folder or under a different name and used if no cython is going to be used. See the setup.py of https://github.com/vasole/fisx to get an idea if necessary.

Prepare automatic packaging of silx.
The target is debian8-backports or newer.
I am on it.

@t20100

After installation from master branch I get an error running the run_tests.py script.

ERROR:silx.gui.plot.PlotTools: Error while converting coordinates (46.87500, 52.246696)with converter 'Exception'

HDF5 is part of the core of the data policy at ESRF. The natural binding for hdf5 is h5py.
The idea is to provide an API similar to h5py in silx when addressing images and specfiles.

Today: there is 1 binding for SpecFile and 2 for images: EdfImage and fabio.
H5py offers a Dataset class with attrs which matches pretty well with the data/header dichotomy in image formats. Re-implementing the behavior (or subclassing?) Dataset solves most of the discussion about the API. The second advantage is about the formation to silx as trainee have to learn only one API.

Difference between HDF5/other format

The main difference between HDF5 and other is the hierarchical structure which is exposed as a Python dictionary where most of the other formats are using a linear structure. The natural design would then be to use (/expose?) a Python list as container to Dataset instances in a File object.

About file/file-series

The h5py File module provides a "driver" option to select the way data are saved. Moreover, file series would be represented as "prefix_%04d.img" to create a unified view, like what is done in h5py.

Work to be performed:

• Implement Dataset, Attribute, and File
• Provide drivers for the various file formats (re-using the code existing in the pre-existing libraries)

Advantages:

The transition to HDF5 over the period 2010-2015 has been very slow, most scientists being:

• reluctant to change something which already works
• don't want to learn yet another API
  If one wants actually to promote HDF5, offering the same API between HDF5/legacy would allow to bridge the gap.

As discussed in our meeting, I'm working on a silx command line interface to be able to execute a main function in any module. Any criticism is welcome !

I tried two different ways for finding all available modules, and the most likely option is to browse through the silx package with os.walk. I have a heavily commented prototype to explain how it works: https://github.com/PiRK/silx/blob/silx_cli/scripts/runmodule_oswalk.py

All .py and .so files in a subfolder with a __init__.py file are considered to be modules.

A modules can be called by its short name (e.g. PlotWindow) or it's long name (e.g. silx.gui.plot.PlotWindow). The long name is mandatory in case of ambiguity.

The -h option used together with a module name causes the docstring of the main() function to be printed.

A -l option can be used to print a complete alphabetically sorted list of long module names. This list is already very long, so this option might soon become hard to use.

A -s option can be used to search a module by providing part of its name. All modules whose long name contain the provided search string are listed. Maybe I should change the search to do it on short names? This would help restrict the number of matches, but the downside would be that we could not search all sub-modules of a given module (e.g python runmodule_oswalk.py -s math to find all silx.math.* modules)

So far, the main function should have following signature:

int main(*args)

The return value is used as the exit code (sys.exit(main(*args))).

args is the list of all arguments specified on the command line after the module name. For the moment I pass them as raw strings to the main() function. Do you think it would be a good idea to try to guess the type of the arguments, or should we just let the main() function handle all typecasting?

This cannot go through in Debian as it would be considered "FBFS" (failed to build from scratch)

WARNING:run_tests:Patched sys.path, added: '/home/kieffer/workspace/silx/package/silx-0.0.1a3/.pybuild/pythonX.Y-dbg_2.7/build'
WARNING:run_tests:Test silx 0.0.1-dev3 from /home/kieffer/workspace/silx/package/silx-0.0.1a3/.pybuild/pythonX.Y-dbg_2.7/build/silx
INFO:run_tests:Import silx.test
WARNING:silx.io.test.test_specfile:de_DE.utf8 locale not installed on your system. An important i18n test will be skipped.
Traceback (most recent call last):
File "run_tests.py", line 296, in
test_module = importlib.import_module(test_module_name)
File "/usr/lib/python2.7/importlib/init.py", line 37, in import_module
import(name)
File "/home/kieffer/workspace/silx/package/silx-0.0.1a3/.pybuild/pythonX.Y-dbg_2.7/build/silx/test/init.py", line 46, in
from ..io.test import suite as test_io_suite
File "/home/kieffer/workspace/silx/package/silx-0.0.1a3/.pybuild/pythonX.Y-dbg_2.7/build/silx/io/test/init.py", line 32, in
from .test_dicttoh5 import suite as test_dicttoh5_suite
File "/home/kieffer/workspace/silx/package/silx-0.0.1a3/.pybuild/pythonX.Y-dbg_2.7/build/silx/io/test/test_dicttoh5.py", line 30, in
import h5py
File "/usr/lib/python2.7/dist-packages/h5py/init.py", line 13, in
from . import _errors
ImportError: /usr/lib/python2.7/dist-packages/h5py/_errors.x86_64-linux-gnu.so: undefined symbol: Py_InitModule4_64
[92317 refs]
E: pybuild pybuild:256: test: plugin custom failed with: exit code=1: env PYTHONPATH=/home/kieffer/workspace/silx/package/silx-0.0.1a3/.pybuild/pythonX.Y-dbg_2.7/build python2.7-dbg run_tests.py -i
dh_auto_test: pybuild --test -i python{version}-dbg -p 2.7 -s custom --test-args=env PYTHONPATH={build_dir} {interpreter} run_tests.py -i --dir . returned exit code 13
debian/rules:34: recipe for target 'override_dh_auto_test' failed
make[1]: *** [override_dh_auto_test] Error 13
make[1]: Leaving directory '/home/kieffer/workspace/silx/package/silx-0.0.1a3'
debian/rules:13: recipe for target 'build' failed
make: *** [build] Error 2
dpkg-buildpackage: error: debian/rules build gave error exit status 2
Build failed, please investigate ...

I'm creating this issue to discuss data packaging.

Initially we could gather all the data in a silx/data directory and make setup.py include it with the build. Having all the data (for the moment only GUI icons, but possibly also calibration data in the future, and maybe test related data) in a a single directory will make it easier to separate data and code for Debian packaging. Accessing data in our code could be done using from silx.resources import data_dir.

This is a draft for resources.py: https://github.com/PiRK/silx/blob/master/silx/resources.py

One alternative way of doing, instead of gathering all data in a sub-directory of the package, would be to store data together with the modules using it (e.g. putting icons in a subfolder of silx/gui) and setting data_dir to the root package directory by default. And when we need to separate data and code, we could replicate the project structure in the data directory (e.g. /usr/share/silx/gui/icons). It would make separating data and code slightly more complicated.

relative to absolute imports.

prepare continuous build on Deb8 and automatic deployment

Functions (list taken from PyMca MaskImageWidget):

• Operation on the whole mask:
  • Clear
  • Inversion
• Drawing tools:
  • Brush (with settable width in image pixels)
  • Rectangle
  • Polygon
  • Above threshold
  • Below threshold
  • Within range
• Each draw tool can either paint or erase
• Multiple level of exclusive masks

Proposal: Regarding the number of tools and the fact that more tools will probably be needed, I'm thinking of making a DockWidget as for ROIs, e.g.:

Ideas for future:

• History/undo/redo
• Drawing tools: circle, ring
• Edition of shapes as vector graphics
• shrink/expand mask
• show/hide different level of mask
• Load (replace of add) and Save masks

[]- Test supplied PyQt5 wheels.

[]- Figure out under which platforms are useful

Please do not install them on reference Mac. I need universal builds in order to be able to generate frozen packages.

Message from PyQt5 mailing list follows:

I've created a new set of wheels for the latest snapshot of PyQt5 v5.6.

Wheels for OS X and Windows can now be installed with pip3 from PyPi. Just do...

pip3 install PyQt5

...which will also automatically install the approprate sip wheel as well.

PyPi doesn't host Linux wheels so that can be downloaded from...

https://sourceforge.net/projects/pyqt/files/PyQt5/PyQt-5.6/

There is a link to the Linux wheel for sip on the sip download page.

The OS X and Linux wheels only have internal changes. If you installed the previous versions then there's not much point in installing these new ones.

The Windows wheels should now include the missing MSVC DLL and the OpenSSL DLLs.

Phil

When running python setup.py build on a system (or virtualenv) without cython being installed, C source files ended up not being linked during compilation. This caused following error when trying to run tests:

from silx.io.specfile import SpecFile, Scan
ImportError:
/users/knobel/nocython/local/lib/python2.7/site-packages/silx-0.0.1a2-py2.7-linux-x86_64.egg/silx/io/specfile.so:
undefined symbol: SfError

When panning with arrow keys, overlays like crosshair cursor and area displayed while drawing are not update.

Try to send a mouse move event when panning this way to sync the overlays.

As there is a growing bunch of optional dependencies, what about having a requirements.txt file with all those dependencies at the root of the project to document this + allow to install silx & optional dependencies with: pip install -r requirements.txt...

Enhancements for PlotWindow profile toolbar:

• Disable profile actions when no image is displayed
• Update profile when image is updated
• Add draggable anchors to free profile to allow changing an already defined profile.

@PiRK

With a python setup.py install, the generated specfile.c code is overwritten and git complains.

The default file should be provided in a default folder or under a different name and used if no cython is going to be used. See the setup.py of https://github.com/vasole/fisx to get an idea if necessary.

Pointed out by @kif: in silx/silx/math/setup.py os.path.join should be used to generate the paths.

IIRC os.path.join wasn't used because distutils handles both slash and backslash on windows (I think...). But still, it's better to use os.path.join.

histo_src = ['/'.join([histo_dir, srcf])
                 for srcf in ['histogramnd.pyx',
                              'src/histogramnd_c.c']]
histo_inc = ['/'.join([histo_dir, 'include']),
                 numpy.get_include()]

The Legend pane does not take into account the active curve and so the color of the active curve is different between the legend pane and the plot area.

• I expect an icon to reflect the action is going to apply and not the current status. To me the shown icons when toggling Y axis origin and when toggling aspect ratio are the wrong ones.

@t20100

• When active curve handling is enabled, there is an active curve, and points are shown, make a difference between mouse coordinates and point coordinates. That behavior was requested by beamlines (and implemented) for PyMca ScanWindow.

I found very convenient to be able to execute a module in order to have an idea about its typical use. Most PyMca widgets where implementing if name == "main": and that was quite handy for both, documentation purposes and for testing the generation of signals.

For instance, I was expecting to be able to get something when doing python -m silx.gui.plot.PlotWindow but one gets absolutely nothing.

I would really prefer to have some modules, and particularly most of those implementing widgets, implementing a def main() function so that one could execute them as "silx ModuleName" without even having to know where they are located in the silx tree. For instance, "silx PlotWindow" would find the PlotWindow module and call its main function if present.

If none of those approaches is to be followed, a whole battery of demos is to be prepared.

Implement saving image data via third party modules for TIFF and EDF

Discussion Introduction

Plot1D2D.pdf

Conclusion

• Make clear we aim to have visualization widgets that implement common needs with an added value than just display. Example: there are many solutions to visualize an image, but if we want to offer the end user the possibility to change display options, extract profiles, calculate masks and more, it is better to have a common solution maintained by all of us than a set of individual solutions.
• To have an OpenGL backend is certainly a warranty for performance. High quality output can be assured by the matplotlib backend for instance.
• Current exposed methods are not considered fundamentally wrong. From a practical point of view they can be taken as starting point.
• New methods need to be added to accommodate developers’ need to access the items in the plot in order to efficiently update them.
• Current approach of choosing PlotWindow functionality at instantiation time needs to be completed by a simple enabling procedure.
• Missing (already foreseen) features: error bars, contour plots, iso-lines, histograms.
• Other missing features: maximum filter for image display, copy widget window (to clipboard, to file, ...)
• Logo contest: It was decided to have a logo based on the “vertical silex stone” with letters model.
• There is need for early training and documentation.

• Export single curves as CSV and SPEC file
• Export multiple curves in SPEC file format

see:
silx-kit/pyFAI#307

todo ... by the way which is the chosen one ?

The goal is to use it when extracting profiles on images.

• Icon for handling the clipboard is missing.

Can we change it to MIT?
Otherwise, add an entry for it in copyright file to remember it.

Reading back empty strings causes a crash on all version of python (easy to fix).

Having a "%" character inside a string works with python2 but not python3 (we probably need to escape % at some stage).

Reading back None yields the string "None" (probably easy to fix).

When calling multiple times::

python setup.py build

Each time the cython file is re-generatd and re-compiled which makes the build time huge. Other project don't have this behaviour, it is probably a bug in the local setup.py

@t20100

The ROI widget should appear below the Options and Coordinates line.

This is a bug in my code.

When reading a SpecFile and interpreting it as a HDF5 with silx.io.spech5, HDF5-like attributes are attached to some groups and datasets as a dictionary. For instance, the root group has the following .attrs: {"NX_class": "NXroot"}.

These attributes are currently not being written to the output HDF5 file when converting a SpecFile using functions from silx.io.spectoh5.

When visualizing a curve with negative values, the values are not shown when setting the axis as being logarithmic, that is desired behavior. However, when coming back to linear axis, the negative values are lost/not show.

Allow an icon toggling (like with aspect ratio) associated to the width (for instance a "w" and a "w" with a horizontal bar on top) in order to allow the user to choose if the average profile or the sum is desired.

To be ready for first release.

Proposition to add to the plot and use as default colormap 'viridis', the future matplotlib 2.0 default colormap.
See http://bids.github.io/colormap/

The chosen abstract (single liner) is:
Toolbox for X-Ray data analysis

May be replaced by smth better

Question: How to handle precision by default in the save function when saving data to ASCII?

Does it need to be conservative and safe for all types (my opinion)?
Does it need to handle safely mixing types, e.g., for a curve x as int and y as float (also my opinion)?

Some examples:

• '%.7g' is safe for float32 but not float64 and int

• By default numpy.savetxt saves with '%.18e'. This is safe for float32 and float64, but not for int

  >>> numpy.savetxt('test.txt', (10000000000000001,))
  >>> numpy.loadtxt('test.txt')
  array(1e+16)
  

I know it is too early, but I put this issue to keep in mind that at each release we have to perform tests to check if we can generate the library from the sources obtained with python setup.py sdist.

in spech5.py file:

Change :

months = ['Jan', 'Feb', 'Mar', 'Apr', 'Jun', 'Jul',
              'Aug', 'Sep', 'Oct', 'Nov', 'Dec']

To :

months = ['Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun', 'Jul',
              'Aug', 'Sep', 'Oct', 'Nov', 'Dec']