pyxem / pyxem Goto Github PK

Current methods put the brightest beam in the center. Given the reasonableness of the physical assumption (the direct beam will generally lie near the center of the image square) I think we could automated the "draw a square around the direct beam" idea.

get_radial_profile in electron_diffraction does not work properly if centers is not constant over the stack. The reason is that a change in the center values could change the length of the average profile, so that the size of the signal axes changes over the stack. This leads to the signal axes not being created at all, and thus the radial profile stack cannot be returned as a signal.

A proposed solution is to use the center values to calculate the max length and use that for all the average profiles in the stack, by appending zeros where the length is below the max. I have tested it for zero, one and two navigation axes, and it seems to work fine. I will make a pull request.

In pyxem/signals/diffraction_vectors.py the function get_diffracting_pixels_map() appears to result in an error, seemingly due to the handling of the diffraction vectors signal (and/or how it interacts with map and as_signal2D()). Changing to the transpose of the diffraction vectors seems to resolve the problem.

get_diffracting_pixels_map():

Current:
crystim = self.map(get_npeaks, inplace=False).as_signal2D((0,1))

Suggested change:
crystim = self.T.map(get_npeaks, inplace=False).as_signal2D((0,1))

We should make some tutorial jupyter notebooks to illustrate key pyxem functionality.

Currently there are several python based data processing libraries for working with pixelated STEM/4D STEM/precession, which all have some kind of overlap in data processing functionalities.

For example, radial integration is implemented in both pixStem, fpd and pyXem:

• https://gitlab.com/pixstem/pixstem/blob/master/pixstem/pixelated_stem_class.py#L457
• https://gitlab.com/fpdpy/fpd/blob/master/fpd/fpd_processing.py#L696

• pyxem/pyxem/signals/electron_diffraction.py

  Line 375 in 7bdac8d

  def get_radial_profile(self,cython=False,inplace=False,**kwargs):

And likewise with removal of dead pixels.

Would there be any interest in combine these efforts into one "backend" library, which handles these types of more generic processing like radial integration, centre of mass, dead pixels, ...

Ideally it would only depend on the more common scientific python stack, so potentially other non-HyperSpy based projects could also use them. I'm tempted to also include dask in this, since it has proven very useful in working with the large pixelated STEM datasets I've got.

Symmetry reduced equispaced grids of orientation space, for orientation mapping including all non-equivalent orientations, can be found here with 1 degree resolution for all proper rotational point groups:

https://drive.google.com/open?id=1njQ3AXy3ZDzLaa4gqbN4K5TQULsXuyvW

Eventually we will add utility functions to the code to make the generation of template libraries more user friendly, but in this short term this should help people make progress.

To perform orientation mapping by template matching you need a list of orientations for which to simulate templates. Forming such a list that samples orientation space evenly and accounts for crystal symmetry correctly is a little non-trivial so we need to write a function to make these easily accessible to users in general.

Currently if you save an object from pyxem it saves as a .hspy file. If this object is supposed to be one of the pyxem classes that is not ElectronDiffraction it is loaded incorrectly because pxm.load() assumes it will be opening ElectronDiffraction data.

We need to re-write the load method to handle all the pyxem classes and may need to look at the save function too.

Seems increasing that a small test dataset would be useful for testing - we'd previously tried to avoid this, but should now pick something minimal that gives sufficient test flexibility.

Masked Orientation Mapping has been requested. This would allow Regions of non-interest (vacuum or areas of sample known to be produce bad matches) to be removed in advance (with associated speed up). I'll be looking into this early next week.

Anyone with strong opinions on how this should work, speak now...

This is probably coming from:

dp.apply_affine_transform()

or another similar pre-processing stage.

A checklist of miscellaneous actions that should be done for a tidy 0.6 release:

• Finalize namespace clarity (dnjohnstone)
• Finalize docstrings
• Finalize web documentation
• Finalize pyxem release procedure
• Package for Conda
• Zenodo dois

Loading an hspy (HDF5) file saved from pyxem again using 'load()' results in the data being loaded as Signal2D rather than as ElectronDiffraction (or 'electron_diffraction') even when passing the signal_type='electron_diffraction' option with load() as described in the documentation of load().

Syntax like "dp.set_signal_type('electron_diffraction') also appears to have no effect.

Trying to match SED data to simulated ED patterns from a .cif file + list of rotation:

%matplotlib tk
import pycrystem as pc
import numpy as np
import transforms3d

Loading the data:

dp = pc.load('QuantumDots\QD-03.blo')
dp.apply_affine_transformation(D = np.array([[0.99, 0.00, 0.00],
                                             [0.00, 0.69, 0.00],
                                             [0.00, 0.00, 1.00]]))
dp.set_calibration(0.032)
structure = pc.Structure.from_file("CsPbBr3_1.cif")

Initially I used a full list of 32000 (or so) combinations of Euler angles to define a rotation list:

rot_array = np.loadtxt('mmm_grid_euler')
rot_list = rot_array.tolist()
edc = pc.ElectronDiffractionCalculator(300, 0.025)
diff_gen = pc.DiffractionLibraryGenerator(edc)
struc_lib = dict()
struc_lib["CsPbBr3"] = (structure, rot_list)
library = diff_gen.get_diffraction_library(struc_lib,
                                            calibration=1.2/128,
                                            reciprocal_radius=1.,
                                            representation='euler')

The feedback I got:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-11-7c406103d17d> in <module>()
----> 1 match_results = indexer.correlate(parallel = True)

~\Anaconda3\lib\site-packages\pycrystem-0.4-py3.6-win-amd64.egg\pycrystem\indexation_generator.py in correlate(self, n_largest, *args, **kwargs)
    119                                       n_largest=n_largest,
    120                                       inplace=False,
--> 121                                       *args, **kwargs)
    122         return MatchingResults(matching_results)
    123 

~\Anaconda3\lib\site-packages\hyperspy\signal.py in map(self, function, show_progressbar, parallel, inplace, ragged, **kwargs)
   3530                                     parallel=parallel, inplace=inplace,
   3531                                     ragged=ragged,
-> 3532                                     **kwargs)
   3533         if inplace:
   3534             self.events.data_changed.trigger(obj=self)

~\Anaconda3\lib\site-packages\hyperspy\signal.py in _map_iterate(self, function, iterating_kwargs, show_progressbar, parallel, ragged, inplace, **kwargs)
   3650                            show_progressbar)
   3651         for ind, res in zip(range(res_data.size),
-> 3652                             thismap(func, zip(*iterators))):
   3653             res_data.flat[ind] = res
   3654             if ragged is False:

~\Anaconda3\lib\concurrent\futures\_base.py in result_iterator()
    584                     # Careful not to keep a reference to the popped future
    585                     if timeout is None:
--> 586                         yield fs.pop().result()
    587                     else:
    588                         yield fs.pop().result(end_time - time.time())

~\Anaconda3\lib\concurrent\futures\_base.py in result(self, timeout)
    430                 raise CancelledError()
    431             elif self._state == FINISHED:
--> 432                 return self.__get_result()
    433             else:
    434                 raise TimeoutError()

~\Anaconda3\lib\concurrent\futures\_base.py in __get_result(self)
    382     def __get_result(self):
    383         if self._exception:
--> 384             raise self._exception
    385         else:
    386             return self._result

~\Anaconda3\lib\concurrent\futures\thread.py in run(self)
     54 
     55         try:
---> 56             result = self.fn(*self.args, **self.kwargs)
     57         except BaseException as exc:
     58             self.future.set_exception(exc)

~\Anaconda3\lib\site-packages\hyperspy\misc\utils.py in func(*args)
    973     def func(*args):
    974         dat, these_kwargs = figure_out_kwargs(*args)
--> 975         return function(dat, **these_kwargs)
    976 
    977     return func, iterators

~\Anaconda3\lib\site-packages\pycrystem-0.4-py3.6-win-amd64.egg\pycrystem\indexation_generator.py in correlate_library(image, library, n_largest)
     45         correlations = dict()
     46         for orientation, diffraction_pattern in library[key].items():
---> 47             correlation = correlate(image, diffraction_pattern)
     48             correlations[orientation] = correlation
     49         res = nlargest(n_largest, correlations.items(), key=itemgetter(1))

~\Anaconda3\lib\site-packages\pycrystem-0.4-py3.6-win-amd64.egg\pycrystem\utils\__init__.py in correlate(image, pattern, include_direct_beam, sim_threshold, interpolate, **kwargs)
     77         image_intensities = image.T[pixel_coordinates[:, 0][in_bounds], pixel_coordinates[:, 1][in_bounds]]
     78     pattern_intensities = pattern_intensities[mask]
---> 79     return np.nan_to_num(_correlate(image_intensities, pattern_intensities))
     80 
     81 

~\Anaconda3\lib\site-packages\pycrystem-0.4-py3.6-win-amd64.egg\pycrystem\utils\__init__.py in _correlate(intensities_1, intensities_2)
    114 
    115 def _correlate(intensities_1, intensities_2):
--> 116     return np.dot(intensities_1, intensities_2) / (
    117         np.sqrt(np.dot(intensities_1, intensities_1)) *
    118         np.sqrt(np.dot(intensities_2, intensities_2))

ValueError: shapes (14,) and (13,) not aligned: 14 (dim 0) != 13 (dim 0)

I found that there is a special triplet of Euler angles that causes that problem.

rot_list_test = [(2.692794, 1.352630, 4.288523)]
rot_list_test
edc = pc.ElectronDiffractionCalculator(300, 0.025)
diff_gen = pc.DiffractionLibraryGenerator(edc)
struc_lib = dict()
struc_lib["CsPbBr3"] = (structure, rot_list_test)
library = diff_gen.get_diffraction_library(struc_lib,
                                            calibration=1.2/128,
                                            reciprocal_radius=1.,
                                            representation='euler')
indexer = IndexationGenerator(dp, library)
match_results = indexer.correlate(parallel = True)

Which gives the same error as above. After removing this one set of angles from original file everything works fine. Even changing one of the angles slightly can fix the issue (for example changing the last one from 4.288523 to 4.288525. The issue repeats for different files with this set of angles, is fixed when the set is removed.

@to266

ElectronDiffraction as a lazy signal won't run .compute(), making them useless for decomposition.

Steps that should work:

>>> from pycrystem import ElectronDiffraction
>>> import numpy as np
>>> data = np.random.random((50, 50, 100, 100))
>>> ed = ElectronDiffraction(data)
>>> ed.compute()

AttributeError                            Traceback (most recent call last)
<ipython-input-6-5de62f25ef12> in <module>()
----> 1 ed.compute()

/home/bm424/Documents/phd/hyperspy/hyperspy/_signals/lazy.py in compute(self, progressbar)
     90             cm = dummy_context_manager
     91         with cm():
---> 92             self.data = self.data.compute()
     93         self._lazy = False
     94         self._assign_subclass()

AttributeError: 'numpy.ndarray' object has no attribute 'compute'

I think they're supposed to be dask arrays for this? I don't know much about lazy signal processing. I've tried casting it to a Signal2D, and then this will work, but having the ElectronDiffraction class is helpful.

We have a number of unused imports throughout the code from recent trimming of functionality - we should tidy this up.

There appears to be a discrepancy in how the center of the diffraction pattern is handled. In the blur-method for center_direct_beam the center appears to be at half the pixel dimensions (at 128, 128 for (256,256)). In the radial integration, however, (in radial_average in utils/expt_utils) there is a correction (-0.5) to put the center at 127.5,127.5. I think this should be handled consistently - I'm not sure if the diffraction pattern center shows up anywhere else (will keep looking...).

Hi guys,
I have noticed for the last few weeks that every time I update my copy of pyXem that stuff that I rely on is removed which breaks my code (i.e. https://github.com/stefsmeets/problematic). First, I saw that it is no longer possible to obtain the centers of the diffraction patterns (get_beam_center if I'm not mistaken), and when I updated today the peak finder I like to use (find_peaks_regionprops) is no longer available. What is the reasoning behind all this?

The hyperspy map() function can return a number of data structures now, including ragged arrays. Assigning all of these outputs to a new class e.g. DiffractionVectors is providing difficult without specifying each case individually or getting a brittle result.

Probably surmountable but needs a bit of careful thought and checking hyperspy...

The prevailing convention on this is fairly strong, and most people natural instinct is to think in degrees. However this will require quite a lot of careful adjustments so as not to break anything.

Personally I am pretty convinced this is the correct thing to do.

Now that we have a fairly comprehensive set of tests in terms of coverage we should make some efforts to write more tests that verify important physical behaviours.

get_unique_vectors does not find all unique vectors. It considers each vector element x and y separately. If one of those is already in the gvlist, the considered vector will not be added, even if it is a unique vector.
..................................................................

Example of issue

When testing the vector i[0][j] against the list of unique g-vectors, gvlist, the code will pass, meaning that this vector is not added to the list of unique vectors. The reason is that one of the values in i[0][j]; -0.11913, is in gvlist. However, this is not the same g-vector, and i[0][j] should have been added.

i[0][j] = [ 0.26334 -0.11913]
gvlist = [[ 0.35112 -0.11913]
..................................................................

Solution

A solution could be (using the same notation) to change the if-statement from:

if np.asarray(i[0][j]) in np.asarray(gvlist):

to

if sum(map(lambda x: np.array_equal(np.asarray(i[0][j]),x), np.asarray(gvlist))):

This seems to works for me. Of course, since gvlist then becomes a lot larger, everything you do from there takes so much more time and space...

Incorporating distance threshold

Therefore, I tried to complete the to-do-task of adding the distance_threshold. I think this can be easily done using np.allclose. Then, instead change the above if-statement to:

if sum(map(lambda x: np.allclose(np.asarray(i[0][j]), x, rtol=0, atol=distance_threshold, equal_nan=False), np.asarray(gvlist))):

The complete function then becomes:

    def get_unique_vectors(self,
                           distance_threshold=None):

        gvlist=[]

        for i in self._iterate_signal():

            for j in np.arange(len(i[0])):
                
                if distance_threshold is not None:
                    check = sum(map(lambda x: np.allclose(np.asarray(i[0][j]),
                                                          x, 
                                                          rtol=0, 
                                                          atol=distance_threshold, 
                                                          equal_nan=False), 
                                    np.asarray(gvlist)))
                        
                else: 
                     check = sum(map(lambda x: np.array_equal(np.asarray(i[0][j]),x), 
                                     np.asarray(gvlist)))
                                
                if check:
                    pass
                
                else:
                    gvlist.append(i[0][j])

        unique_vectors = np.asarray(gvlist)
        
        return unique_vectors

Hopefully, someone else has the time to test this as well.

Test coverage is improving, but we have work to do, this is just a reference list of the worst offenders in terms of coverage:

• DiffractionComponent
• ScalableReferencePattern
• DiffractionGenerator
• Indexation Generator
• DiffractionVectors
• IndexationUtils
• SimUtils

Indexation of diffraction profiles / lists of magnitudes does not currently work in the mapping case. This functionality should be added.

It would be nice having pyxem packaged in PyPI, as currently (I think) it is only available pre-packaged via conda-forge?

Having run the following as set up.

import numpy as np
import pycrystem as pc
import hyperspy.api as hs

x = np.random.rand(2,2,10,10) # two by two in navigation and 10 by 10 in signal

#generate some maxima 

x[0,0][0][9] = 5
x[1,0][9][0] = 5
x[0,1][9][9] = 5
x[1,1][0][0] = 5

dp = pc.ElectronDiffraction(x)

Now moving to a console:

>>> dp
<ElectronDiffraction, title: , dimensions: (2, 2|10, 10)>

>>> dp.find_peaks('skimage')
<DiffractionVectors, title: , dimensions: (2, 2|)>

Which is in my opinion is/should be the desired behaviour. However

>>> dp.find_peaks('skimage',num_peaks=1)
<DiffractionVectors, title: , dimensions: (2, 1, 2, 2|)>
>>> dp.find_peaks('max')
<DiffractionVectors, title: , dimensions: (2, 1, 2, 2|)>

Regardless of what we chose, I think we should try to have .find_peaks() return a consistent object (and write tests to confirm this)

Sub-pixel peak finding would be useful - most of the current peak finders are not so precise for that. A cross-correlation based approach similar to that used for diffraction pattern centering is probably the correct solution, could also fit a function at each peak.

A crystallographic map produced by the save_map function in pyXem creates a horizontally flipped image when imported as an EBSD file in MTEX. I'm guessing this is because in pyXem the origin point is at the top left corner, while in MTEX it's at bottom left. A possible solution is to convert all the y-indices to (max y-index - original y-index), but I'm not sure how easy/difficult it'd be to put this into the code.

We experienced an issue with installing pyxem as it seems to rely on availability of a C compiler. After comparing with pycrystem setup file, we could have it work by modifying the setup.py file, commenting out the lines calling Cython as follows (please kindly advise if there is a more elegant solution to this!):

#!/usr/bin/env python

import os, sys
from setuptools import setup, Extension

exec(open('pyxem/version.py').read())  # grab version info 

#from Cython.Build import cythonize
#import numpy as np

# if sys.platform == "win32":
    # extensions = [
        # Extension('pyxem.utils.radialprofile', ['src/radialprofile.pyx'],     include_dirs=[np.get_include()])
    # ]
# else:
    # extensions = [
        # Extension('pyxem.utils.radialprofile', ['src/radialprofile.pyx'],     include_dirs=[np.get_include()])
    # ]
# ext_modules = cythonize(extensions)


setup(
    name='pyxem',
    version=__version__,
    description='An open-source Python library for crystallographic electron'
                'microscopy.',
    author=__author__,
    author_email=__email__,
    license="GPLv3",
    url="https://github.com/pyxem/pyxem",

    classifiers=[
        'Programming Language :: Python :: 3.6',
    ],

  #  ext_modules = ext_modules,

    packages=[
        'pyxem',
        'pyxem.utils',
    ],

    package_data={
        "": ["LICENSE", "readme.rst", "requirements.txt"],
        "pyxem": ["*.py"],
    },
)

We need to start thinking about tests and implementing them...

Finding a way to get library matches from dictionary keys that is more stable.

(0.0,0.0,0.0) is [1e-16,0,0] or what have you.

We want to have some full dynamical diffraction simulations callable from PyCrystEM. Following e-mail correspondence the plan is to go for a multislice implementation and a Bloch wave implementation based on code from Christoph Koch's group. @woutervandenbroek and @jacobjma

The multislice code is here:
https://github.com/woutervandenbroek/FDES

A python wrapper for some of the desired functionality is here:
https://github.com/jacobjma/PyQSTEM/blob/master/examples/HRTEM_dft.ipynb

The DP shown in find_peaks_interactive at inav[x,y] is mirrored with respect to the original signal plot from s.inav[x,y].

To make the architecture more consistent virtual images for a series of integration window positions should be produced using a VDFGenerator class.

I have implemented this and will PR soon.

While trying to plot patterns from a library through library.plot() following error turns up:

AttributeError Traceback (most recent call last)
in ()
----> 1 library_test.plot()

~\Anaconda3\lib\site-packages\pyxem-0.5-py3.6-win-amd64.egg\pyxem\signals\diffraction_library.py in plot(self)
44 for key in self.keys():
45 for ori in self[key].keys():
---> 46 dpi = self[key][ori].as_signal(128, 0.03, 1)
47 sim_diff_dat.append(dpi.data)
48 ppt_test = ElectronDiffraction(sim_diff_dat)

AttributeError: 'dict' object has no attribute 'as_signal'

This is just an artifact of the old library I assume.
@pc494

We should write the following guides:

1. Issue template
2. Pull request template
3. Contributor guide

It always helps having either a pretty result or processing step in the readme file to attract users. I suggest people screenshot whenever they have an interesting result on screen to make finding one easier.

Take for example:

    def apply_affine_transformation(self,
                                    D,
                                    order=3,
                                    inplace=True,
                                    *args, **kwargs):
        """Correct geometric distortion by applying an affine transformation.
        Parameters
        ----------
        D : array
            3x3 np.array specifying the affine transform to be applied.
        inplace : bool
            If True (default), this signal is overwritten. Otherwise, returns a
            new signal.
        Returns
        -------
            ElectronDiffraction Signal containing the affine Transformed
            diffraction patterns.
        """
        return self.map(affine_transformation,
                        matrix=D,
                        order=order,inplace=inplace)

Now, things like show_progressbar won't be carried over to map, as kwargs aren't handled here. Likely to be other methods like this. Will run a sweep as and when.

Currently our Travis works from pip while our install instruction work from conda. Realistically we should probably support both of these options.

However, this will come under the body of work conducted after hyperspy is split so that we only have to do it once.

Files loaded with pyXem at present will be in a HyperSpy class e.g. Signal2D class not a pyXem class e.g. ElectronDiffraction. This can be resolved by the following:

dp = pxm.load(file)
dp = pxm.ElectronDiffraction(dp)

Plot functions are not currently well tested - tests verify that the code runs, but not that the plots are correct. This could be addressed by: https://stackoverflow.com/questions/27948126/how-can-i-write-unit-tests-against-code-that-uses-matplotlib

To reproduce:

import pycrystem as pc
ds = pc.diffraction_generator.DiffractionSimulation(coordinates = (np.random.random((10,2))))
ds.as_signal(10, 3, 1.0)

Gives error:

ImportError: can't import 'None' with 'import_module('numexpr')' command

Here is the full traceback:

ImportError                               Traceback (most recent call last)
<ipython-input-194-b24246395d7b> in <module>()
      1 ds = pc.diffraction_generator.DiffractionSimulation(coordinates = (np.random.random((10,2))))
----> 2 ds.as_signal(10, 3, 1.0)

c:\users\stef\python\pycrystem\pycrystem\diffraction_generator.py in as_signal(self, size, sigma, max_r)
    276         x, y = np.meshgrid(l, l)
    277         coords = self.coordinates[:, :2]
--> 278         g = Expression(_GAUSSIAN2D_EXPR, 'Gaussian2D', module='numexpr')
    279         for (cx, cy), intensity in zip(coords, self.intensities):
    280             g.intensity.value = intensity

C:\Users\stef\AppData\Local\Continuum\Miniconda3\lib\site-packages\hyperspy\_components\expression.py in __init__(self,     expression, name, position, module, autodoc, add_rotation, rotation_center, **kwargs)
    119         self._str_expression = expression
    120         if rotation_center is None:
--> 121             self.compile_function(module=module, position=position)
    122         else:
    123             self.compile_function(module=module, position=rotation_center)

C:\Users\stef\AppData\Local\Continuum\Miniconda3\lib\site-packages\hyperspy\_components\expression.py in compile_function(    self, module, position)
    186         variables = [x, y] if self._is2D else [x]
    187         self._f = lambdify(variables + parameters, eval_expr,
--> 188                            modules=module, dummify=False)
    189 
    190         if self._is2D:

C:\Users\stef\AppData\Local\Continuum\Miniconda3\lib\site-packages\sympy\utilities\lambdify.py in lambdify(args, expr,     modules, printer, use_imps, dummify)
    377     namespace = {}
    378     for m in namespaces[::-1]:
--> 379         buf = _get_namespace(m)
    380         namespace.update(buf)
    381 

C:\Users\stef\AppData\Local\Continuum\Miniconda3\lib\site-packages\sympy\utilities\lambdify.py in _get_namespace(m)
    469     """
    470     if isinstance(m, str):
--> 471         _import(m)
    472         return MODULES[m][0]
    473     elif isinstance(m, dict):

C:\Users\stef\AppData\Local\Continuum\Miniconda3\lib\site-packages\sympy\utilities\lambdify.py in _import(module, reload)
    158 
    159         raise ImportError(
--> 160             "can't import '%s' with '%s' command" % (module, import_command))
    161 
    162     # Add translated names to namespace

ImportError: can't import 'None' with 'import_module('numexpr')' command

There was a function to reproject electron diffraction data to polar coordinates - this has been removed because the overall architecture was wrong. It should be re-added with a new class for polar reprojected diffraction data.

Is there a particular reason pycrystem is requiring specific versions of all software? It makes it difficult to install it in an environment previously used for other things as well.

A full/stable release of pyXem is a goal for the first quarter of 2018. This issue documents anticipated functionality and progress towards implementation.

Functionality:

• 2D peak finding included and returning in calibrated units
• Update atomic scattering parameters & calculation to Doyle-Turner
• Vector based orientation & strain mapping
• Finish strain mapping methods
• Availability of orientation grids for all point groups
• Write save() function for crystallographic maps compatible with MTEX
• Update Correlations class for multi-phase and implement appropriate plotting
• Implement more advanced simulation methods
• Basic variance imaging
• Pattern centre determination routines
• VDF image segmentation routines
• PDF & FEM implementations
• Stable pre-processing and background subtraction

Documentation & Demonstrations:

• Reference frames consistent and documented
• Document ElectronDiffraction class & basic SED data analysis
• Document Orientation Mapping
• Document Strain Mapping
• Document Peak Finding based methods
• Create a series of tutorial notebooks

Testing (aim >75%):

• Tests for ElectronDiffraction
• Tests for LibraryGenerator
• Tests for DiffractionSimulator
• Tests for Orientation Mapping
• Tests for Strain Mapping

When building the docs, I get the following warnings:
Ignore the hyperspy ones.

C:\Users\thomasaar\Downloads\pycrystem-master\docs
(cryst) λ sphinx-build -b html source build
Running Sphinx v1.6.3
making output directory...
The fast cython based bcf unpacking library were found
WARNING: The ipywidgets GUI elements are not available, probably because the hyperspy_gui_ipywidgets package is not installed.
WARNING: The traitsui GUI elements are not available, probably because the hyperspy_gui_traitui package is not installed.

c:\programdata\miniconda3\envs\cryst\lib\site-packages\pymatgen\util\plotting_utils.py:3: UserWarning: pymatgen.util.plotting_utils has been renamed pymatgen.util.plotting. This stub will be removed in PMG 5.0.
  warnings.warn("pymatgen.util.plotting_utils has been renamed "
loading pickled environment... not yet created
loading intersphinx inventory from https://docs.python.org/objects.inv...
intersphinx inventory has moved: https://docs.python.org/objects.inv -> https://docs.python.org/2/objects.inv
loading intersphinx inventory from http://docs.scipy.org/doc/numpy/objects.inv...
intersphinx inventory has moved: http://docs.scipy.org/doc/numpy/objects.inv -> https://docs.scipy.org/doc/numpy/objects.inv
loading intersphinx inventory from http://hyperspy.org/hyperspy-doc/current/objects.inv...
loading intersphinx inventory from https://docs.scipy.org/doc/scipy/objects.inv...
WARNING: intersphinx inventory 'https://docs.scipy.org/doc/scipy/objects.inv' not fetchable due to <class 'requests.exceptions.HTTPError'>: 404 Client Error: Not Found for url: https://docs.scipy.org/doc/scipy/objects.inv
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 12 source files that are out of date
updating environment: 12 added, 0 changed, 0 removed
reading sources... [100%] strain_mapping
C:\Users\thomasaar\Downloads\pycrystem-master\pycrystem\diffraction_generator.py:docstring of pycrystem.diffraction_generator.DiffractionSimulation.direct_beam_mask:1: WARNING: Inline interpreted text or phrase reference start-string without end-string.
C:\Users\thomasaar\Downloads\pycrystem-master\pycrystem\indexation_generator.py:docstring of pycrystem.indexation_generator.IndexationGenerator.correlate:6: WARNING: Inline strong start-string without end-string.
C:\Users\thomasaar\Downloads\pycrystem-master\pycrystem\indexation_generator.py:docstring of pycrystem.indexation_generator.IndexationGenerator.correlate:8: WARNING: Inline strong start-string without end-string.
WARNING: C:\Users\thomasaar\Downloads\pycrystem-master\pycrystem\indexation_generator.py:docstring of pycrystem.indexation_generator.MatchingResults.get_phase_results:4: (SEVERE/4) Unexpected section title.

Paramters
---------
WARNING: C:\Users\thomasaar\Downloads\pycrystem-master\docs\source\pycrystem.rst:65: (WARNING/2) autodoc: failed to import module 'pycrystem.orientation_map'; the following exception was raised:
Traceback (most recent call last):
  File "c:\programdata\miniconda3\envs\cryst\lib\site-packages\sphinx\ext\autodoc.py", line 657, in import_object
    __import__(self.modname)
ModuleNotFoundError: No module named 'pycrystem.orientation_map'
C:\Users\thomasaar\Downloads\pycrystem-master\pycrystem\tensor_field.py:docstring of pycrystem.tensor_field.DisplacementGradientMap.polar_decomposition:3: WARNING: Unexpected indentation.
C:\Users\thomasaar\Downloads\pycrystem-master\pycrystem\utils\expt_utils.py:docstring of pycrystem.utils.expt_utils.refine_beam_position:4: WARNING: Unexpected indentation.
C:\Users\thomasaar\Downloads\pycrystem-master\pycrystem\utils\expt_utils.py:docstring of pycrystem.utils.expt_utils.refine_beam_position:5: WARNING: Block quote ends without a blank line; unexpected unindent.
C:\Users\thomasaar\Downloads\pycrystem-master\pycrystem\utils\peakfinders2D.py:docstring of pycrystem.utils.peakfinders2D.find_peaks_stat:6: WARNING: Unexpected indentation.
C:\Users\thomasaar\Downloads\pycrystem-master\pycrystem\utils\peakfinders2D.py:docstring of pycrystem.utils.peakfinders2D.find_peaks_stat:7: WARNING: Block quote ends without a blank line; unexpected unindent.
WARNING: C:\Users\thomasaar\Downloads\pycrystem-master\docs\source\pycrystem.utils.rst:50: (WARNING/2) autodoc: failed to import module 'pycrystem.utils.strain_utils'; the following exception was raised:
Traceback (most recent call last):
  File "c:\programdata\miniconda3\envs\cryst\lib\site-packages\sphinx\ext\autodoc.py", line 657, in import_object
    __import__(self.modname)
ModuleNotFoundError: No module named 'pycrystem.utils.strain_utils'
WARNING: C:\Users\thomasaar\Downloads\pycrystem-master\pycrystem\utils\__init__.py:docstring of pycrystem.utils.correlate:7: (SEVERE/4) Unexpected section title.

Parameters
----------
C:\Users\thomasaar\Downloads\pycrystem-master\pycrystem\utils\__init__.py:docstring of pycrystem.utils.correlate:17: WARNING: Inline strong start-string without end-string.
C:\Users\thomasaar\Downloads\pycrystem-master\pycrystem\utils\__init__.py:docstring of pycrystem.utils.correlate:19: WARNING: Definition list ends without a blank line; unexpected unindent.
WARNING: C:\Users\thomasaar\Downloads\pycrystem-master\pycrystem\utils\__init__.py:docstring of pycrystem.utils.correlate:20: (SEVERE/4) Unexpected section title.

Returns
-------
C:\Users\thomasaar\Downloads\pycrystem-master\pycrystem\utils\__init__.py:docstring of pycrystem.utils.correlate:23: WARNING: Definition list ends without a blank line; unexpected unindent.
WARNING: C:\Users\thomasaar\Downloads\pycrystem-master\pycrystem\utils\__init__.py:docstring of pycrystem.utils.correlate:24: (SEVERE/4) Unexpected section title.

References
----------
C:\Users\thomasaar\Downloads\pycrystem-master\pycrystem\utils\__init__.py:docstring of pycrystem.utils.correlate:26: WARNING: Enumerated list ends without a blank line; unexpected unindent.
WARNING: C:\Users\thomasaar\Downloads\pycrystem-master\pycrystem\utils\__init__.py:docstring of pycrystem.utils.correlate_component:9: (SEVERE/4) Unexpected section title.

Parameters
----------
WARNING: C:\Users\thomasaar\Downloads\pycrystem-master\pycrystem\utils\__init__.py:docstring of pycrystem.utils.correlate_component:17: (SEVERE/4) Unexpected section title.

Returns
-------
WARNING: C:\Users\thomasaar\Downloads\pycrystem-master\pycrystem\utils\__init__.py:docstring of pycrystem.utils.correlate_component:22: (SEVERE/4) Unexpected section title.

References
----------
C:\Users\thomasaar\Downloads\pycrystem-master\pycrystem\utils\__init__.py:docstring of pycrystem.utils.correlate_component:24: WARNING: Enumerated list ends without a blank line; unexpected unindent.
C:\Users\thomasaar\Downloads\pycrystem-master\docs\source\setup.rst:4: WARNING: autodoc: failed to import module 'setup'; the module executes module level statement and it might call sys.exit().
looking for now-outdated files... none found
pickling environment... done
checking consistency... C:\Users\thomasaar\Downloads\pycrystem-master\docs\source\roadmap.rst: WARNING: document isn't included in any toctree
done
preparing documents... done
writing output... [100%] strain_mapping
generating indices... genindex py-modindex
highlighting module code... [100%] pycrystem.utils
writing additional pages... search
copying static files... done
copying extra files... done
dumping search index in English (code: en) ... done
dumping object inventory... done
build succeeded, 27 warnings.

I'm planning on refactoring this for speed; but I've raised this for some clarifications, probably from @smc204 (function reproduced below)

The slow step is register_translation so

1. Can the step below comment be factored out of the for loop. This would halve the run time.
2. In signals with varying direct beam size is it possible to identify in advance how large the direct beam is likely to be? If not the current route is fine, but if you had beam sizes ranging from 4 to 8 you could save another factor (of 4 or so) by only searching at the correct(ish) size.

    radiusList = np.arange(radius_start,radius_finish)
    errRecord = np.zeros_like(radiusList,dtype='single')
    origin = np.array([[round(np.size(z,axis=-2)/2),round(np.size(z,axis=-1)/2)]])

    for ind in np.arange(0,np.size(radiusList)):
        radius = radiusList[ind]
        ref = reference_circle(origin,np.size(z,axis=-2),np.size(z,axis=-1),radius)
        h0= np.hanning(np.size(ref,0))
        h1= np.hanning(np.size(ref,1))
        hann2d = np.sqrt(np.outer(h0,h1))
        ref= hann2d*ref
        im = hann2d*z
        shift, error, diffphase = register_translation(ref,im, 10)
        errRecord[ind] = error
        index_min = np.argmin(errRecord)

        #the step below this
        ref = reference_circle(origin,np.size(z,axis=-2),np.size(z,axis=-1),radiusList[index_min]) 
        ref= hann2d*ref
        shift, error, diffphase = register_translation(ref,im, 100)

    return shift

The specification of install_requires in setup.py is causing an issue at the moment - most easily solved by commenting this out of the setup file but not clear what the underlying issue is.

I've spent some time on this recently, and I think we now have a far more correct implementation.

The next steps are as I see it:

1a) Refactoring the template library in such a way as to store the relevant data for correlate (at the moment pattern.calibrated_coordinates and pattern.intensities) and prevent these operations being tackled for each and every image.

1b) Removing some of the internal safeguards from correlate, for example:
in_bounds = np.product((pixel_coordinates > 0)*(pixel_coordinates < shape[0]), axis=1)
checks that each template peaks lies within bounds of the image, which seems like a good idea. Except that if they don't an error is thrown, which in my opinion is sensible behavior. Reciprocal radius (generating the library) should be such that all peaks land within the image, if this isn't the case it's likely you'll generate poor results as it is.

2x) Making matching_results smarter, probably by converting Euler to cubochoric. This would allow us to calculate the distances between various results 'by eye'. texpy make be the saving grace here.

3experimental) Some improvement on the matching formulae. Either for speed or accuracy. For example you can drop the image normalization if you don't need to do reliability work.

Anyway, updates and thoughts are going to be here going forward. Probably starting with a profiling of the current code.

Coverage is not being properly tracked by coveralls. Just needs linking properly.