tofuproject / tofu Goto Github PK

Useful for tIGNITRON on WEST
Wait for imas to handle events

Add the possibility of plotting Equilibrium and 2D map on :

tf.data.Data.plot_spectrogram()
tf.data.Data.plot_svd()

All tests should be updated with:

• Explicit import (import tofu.geom....) instead of relative paths (from . import ...) because tests should not assume they are at the right location*
• There is no need to tests several times the same routine (when the same core routine is called by multiple different methods)

Since I merged with the latests version of master (with the new module for imas) I get the following error when doing python setup.py nosetests. I don't get it when just calling nosetests [options]. It seems when nosetests is ran from distutils there can be this kind of problems (see this SO post).
Error:

======================================================================
ERROR: Failure: Exception (IMAS python API issue
imas could not be imported into tofu ('import imas' failed):

• it may not be installed (optional dependency)

• or you not have loaded the good working environment

  => the optional sub-package tofu.imas2tofu is not usable
  )

---

Traceback (most recent call last):
File "/home/u2/mendoza/.conda/envs/tofu3/lib/python3.7/site-packages/nose/failure.py", line 39, in runTest
raise self.exc_val.with_traceback(self.tb)
File "/home/u2/mendoza/.conda/envs/tofu3/lib/python3.7/site-packages/nose/loader.py", line 417, in loadTestsFromName
addr.filename, addr.module)
File "/home/u2/mendoza/.conda/envs/tofu3/lib/python3.7/site-packages/nose/importer.py", line 47, in importFromPath
return self.importFromDir(dir_path, fqname)
File "/home/u2/mendoza/.conda/envs/tofu3/lib/python3.7/site-packages/nose/importer.py", line 94, in importFromDir
mod = load_module(part_fqname, fh, filename, desc)
File "/home/u2/mendoza/.conda/envs/tofu3/lib/python3.7/imp.py", line 244, in load_module
return load_package(name, filename)
File "/home/u2/mendoza/.conda/envs/tofu3/lib/python3.7/imp.py", line 216, in load_package
return _load(spec)
File "", line 696, in _load
File "", line 677, in _load_unlocked
File "", line 728, in exec_module
File "", line 219, in _call_with_frames_removed
File "/home/u2/mendoza/tofu/tofu/imas2tofu/init.py", line 16, in
raise Exception(msg)
Exception: IMAS python API issue
imas could not be imported into tofu ('import imas' failed):

• it may not be installed (optional dependency)

• or you not have loaded the good working environment

  => the optional sub-package tofu.imas2tofu is not usable

Make the input polygons of Ves and Struct more flexible (type):

• Should work with any 2D iterable with at least one shape ==2
• Automatically convert to np.float_

The dependency on git during setup is an issue wen installing with easyconfig:
see https://jira.iter.org/browse/IMAS-2287?filter=-3 for context

Explanation:

I got your email Vezinet Didier, asking how to make the package easier (i.e. getting rid of the patch).

What the patch does is remove the part that requires git commands during the python setup.py build step. Why? because the .eb file downloads the tarball – so there is no git repository inside.

How the packaging would be better: remove the need for git. You can probably include a test that inspects whether setup.py is run from a git repo or not. E.g. check for the .git directory's existence, or make git describe fail gracefully and continue with the default version – now it breaks

It is not possible now to choose which LOS is used for computing the LOS signal.

Idem dans :
GDetect.calc_Sig()
GDetect.plot_Sig()

The longitudinal sampling of a large number of LOS can be used for:

• Visual representation of the LOS in a poloidal projection (hyperbola)
• Computing of an emissivity / other field at each point along the LOS
• Computing the integral for synthetic diagnostics

When possible the sampling is vectorized.
But in many cases it cannot be vectorized (if the number of samples it different on each LOS).
In this case a for loop is used (Cython)

This algorithm is not optimal and can be optimized and parallelized.

Furthermore, 3 different cases are possible for the output format. This is not necessary, a single format (concatenated array) would result in simpler code and more efficient computing.

The concerned routines are (in tf.geom._GG02.pyx):

• def LOS_get_sample()
• def LOS_calc_signal()

Two algorithms are envisaged for this:

1/ make semi-plane from LOS in any direction, count intersections with polygon (even vs odd)
2/ triangulate polygon from barycenter, count intersections of LOS with triangulated surface (even vs odd)

Tasks:

• See if other algorithms are possible (discuss with colleagues)
• Implement both and tests on simple cases (performance), unless the faster one is clear even without implementing / testing (bibliography, benchmarks by other people...)

There's an error when cam is IN device with "optimized" algo, the results are always NaN

To answer need expressed by TFL on WEST
Created branch feature/magfieldlines for this

When calling _tfg_c._Detect_SigSynthDiag() with PreComp=False and non-standard resolution parameters, the following error was raised:

>> NameError: global name 'ConeWidth_k' is not defined

Indeed, checking _tfg_c._Detect_SigSynthDiag() showed that variable ConeWidth_k had not been defined as one of the input arguments !

The runtime for the current method is longer than a minute or more depending on the file size

Identical to without unit vector, one extra output

The function works fine when unprocessed video is used i.e., the raw data from camera but has problems reading the metadata of videos previously processed my OpenCV

Cython is necessary here for memory / speed performance.

We will intergate toroidally over a potentially large plasma sub-domain (or the whole domain).
It implies:

• Determining the plasma volume to be sampled (from user input)
• Sampling the volume (copy the already existing algorithm in _GG02.pyx)
• As the volume is being sampled, include a few extra steps:
  * sampling has to be done by looping on R (outer main loop), because R determines the number of toroidal points
  * then loop on Z
  * and integrate toroidally by updating a cumulative toroidal sum of solid angles

Important : for each point, only compute solid angle if there is no vignetting due to structral elements!
i.e.: use algorithms for LOS_intersection where appropriate ! (to be discussed)

The output should be a 2D (R,Z) map of toroidally integrated solid angles (one map for each M point on the trajectory, so in fact the output should be a (M,R,Z) array).

The question of where to put the loop on the M points remains to be determined.
Maybe inside (innermost loop) so the parallelization is done on R ?

The KeyHandler class in tofu should be improved to:

• Make it handle properly the "home" button in the figure
• Make "Clic" move the last item (like "arrow", to be consistent)
• Make "Shift + clic" and "Shift + arrow" add a new item
• Make "Ctrl+clic" and "Ctrl+arrow" go back to 1 item
• Make "Alt+arrow" increment by N steps instead of 1 (parametrize N with default value at 10)

• Include the possibility of directly providing the shot, user.... (and keep dids, just convert to it in the function)

• Make sure diag names match the imas diag names

• Create default Name for objects if not provided

A segmentation fault appeared

VideoCapture class reads videos through default 4 channels and any forced tweaking have not worked yet. This results in an additional grayscale conversion step before doing any operation. Although it does not provide any noticeable increased processing time still a fix would render dumpro more efficient.

To provide the background image for field line tracing

A base method shall be included in Struct, then Config just loops over it

Take attribute Poly_closed from Struct subclass instances

No tool (neither distutils nor setuptools) has been used yet to ensure easy installation of the release (to be defined)

In its current form, it is not possible to choose the parameters of Detect.set_SigPreComp() at the creation of the Detect.

Several problems:

• Authentication (travis-encryp password)
• Version number of tofu must be PEP440-compatible (i.e.: remove 'v' from git describe)
• Name conflict with another tofu package

When trying to re-create the WEST SXR camera from the real geometry, tf.geom.CamLOS1D() mixes up 1D vs D in _complete_dX12(self, dgeom) (line 2982).

Probably due to the fact that the Ds are not aligned but on a berth

This version can be done with numpy, no need for cython, a priori

Take time to implement the exact computation and the approximation

Support for matlab users

Create / optimize / parallelize an algorithm for computing the distance between N LOS and M closed polygons (with LIm=None, e.g.: isoflux surfaces).

As specified in Milestone "LOS-to-isoflux distance computation", should exist in several versions (distance, bool, closest LOS, closest isoflux)

Tests are not fully passed on travis, due to hard-coded path for tests inputs
They are also too verbose (print of all save / loads)

In some geometrical configurations, the direction in which the vector perpendicular to a Detect of Optics instance (attribute nIn) may not be obvious and the algorithm may point it towards the wrong direction upon creation of the instance.

This is a problem for later computations, which rely on nIn to estimate the volume of sight (e.g.: k*nIn) and lay thus look in the wrong direction.

Profile, optimize and parallelize the best algorithm

Make the structure of the code (inlined basic functions...) allows for creating other version later if necessary

Make available through a single python function, for N LOS and M polygons, return a (N,M) array of booleans

Create a new IMAS sub-package for handling tofu-IMAS generic compatibility
Inspired from the prior tfou_west plugin
Should allow to load and visualize interactively:

• Configurations
• core_profiles
• diags

Should also allow to compute synthetic diag data from core_profiles and equilibrium

To allow direct issue opening by users

That issue was revealed by an ITER test case implemented by Gabor.

On figure01, page 1 one notices that there is part of the VOS on the opposite side of the Tokamak.
How is this possible ?

When estimating the VOS of a detector+aperture, one of the first steps that ToFu takes is to plot many LOS from the detector through the aperture, to ‘sample’ the VOS.
Now, since we are dealing with a torus, there is always a risk that a LOS will enter the torus twice : once in the poloidal cross-section of interest, and once more in the opposite poloidal cross-section (cf. figure01 page 2 where several examples of LOS are represented)
Of course ToFu can handle this, and it ‘knows’ that each LOS should enter the Torus only once, thus, it disregards the second part in the opposite cross-section.

But detector 1805-00 is in a very special configuration : it is such that some LOS can enter the opposite cross-section without having first entered the relevant one, because the detector is so close to the edge and looking up.
Thus, for these LOS ToFu does not know that they should be disregarded because the usual criterion (i.e. disregard if it is the second time they enter the torus) is not fulfilled.

To solve this problem in the former ToFu version available at IPP, two types of routines were added to General_Geom_cy.pyx (each in 1D and 2D version).
One returns the three planes and 2 points defining the forbidden area.
One check whether a points to this area.

See modified part attached => to be implemented

Debug_1805-0_Corrected.pdf
Figure01.pdf

tofu is not yet available as a pre-compiled conda package for windows 64 bits, so Arpan tried to install it using pip instead.
After creating a virtual environment dedicated to that installation, and running pip (from the conda pip), the following happened (see png attached):

> pip install tofu
Traceback (most recent call last):
File "", line 1, in
File "C:\Users\napra\AppData\Local\Temp\pip-install-6rkhfbb8\tofu\setup.py", line 70, in
not_openmp_installed = check_for_openmp(os.environ['CC'])
File "C:\Users\napra\AppData\Local\Temp\pip-install-6rkhfbb8\tofu\setup.py", line 43, in check_for_openmp
stdout=fnull, stderr=fnull)
File "C:\Users\napra\Anaconda3\envs\tofu140\lib\subprocess.py", line 317, in call
with Popen(*popenargs, **kwargs) as p:
File "C:\Users\napra\Anaconda3\envs\tofu140\lib\subprocess.py", line 769, in init
restore_signals, start_new_session)
File "C:\Users\napra\Anaconda3\envs\tofu140\lib\subprocess.py", line 1172, in _execute_child
startupinfo)
FileNotFoundError: [WinError 2] The system cannot find the file specified
----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in C:\Users\napra\AppData\Local\Temp\pip-install-6rkhfbb8\tofu\

When creating a 1D cam with only one ray, ie:

D = [0,0,0]
u = [0.1, 0.1, 0.1]
cam = tf.geom.CamLOS1D(Exp="Dummy", Name="Dummy", dgeom=(D, u),
config=config, Diag='Test', method="ref", plotdebug=False)

There is an error, and Cam2D is called

I was trying to configure Travis tests with Python version 3.7, after some configurations, I just realized there is a bottom line:

conda.exceptions.UnsatisfiableError: The following specifications were found to be in conflict:

• polygon3 -> python[version='>=3.6,<3.7.0a0']
• python=3.7

So we either try to move forward without the polygon library or we drop the support of python 3.7 (knowing that for the future this still will have to be solved)

versus:

• geom=True and data=False (define chans) in load_Diag()
• user providing ids or dids or (shot, user....) => implement methods in objects using utils
• use same names of diagnostics as IMAS

Hi @Didou09,

I'm currently trying to install tofu.
I tried to install using
python setup.py develop
I first had to correct setup.py since line 128 should be commented out.
Then the cython build process failed on the following:

File "/var/folders/r9/s1k2nd110xzfxyfym7jmyb8r0000gn/T/easy_install-v02kmvno/Polygon2-2.0.8/setup.py", line 29
print "Using NumPy extension!"

I'm using Python 3. So this explains the above error.
When I run 2to3 on the code I get this output:

RefactoringTool: Files that need to be modified:
RefactoringTool: tofu/init.py
RefactoringTool: tofu/defaults.py
RefactoringTool: tofu/helper.py
RefactoringTool: tofu/pathfile.py
RefactoringTool: tofu/setup.py
RefactoringTool: tofu/geom/init.py
RefactoringTool: tofu/geom/_compute.py
RefactoringTool: tofu/geom/_core.py
RefactoringTool: tofu/geom/_plot.py
RefactoringTool: tofu/geom/setup.py
RefactoringTool: tofu/plugins/ITER/init.py
RefactoringTool: tofu/treat/init.py
RefactoringTool: tofu/treat/_compute.py
RefactoringTool: tofu/treat/_core.py
RefactoringTool: tofu/treat/_plot.py

I also had to modify some of the dependencies to get them to work with Python 3.
I'll submit a PR with the modifications for Python 3 compatibility.

Are there any tests I can run to see if everything works correctly now that it seems installed?
Best regards,
Florian

Write the fastest possible algorithm for computing the solid angle subtended by an arbitrary (flat, non-complex) polygon as seen from any point, with a side-discrimination (unit perpendicular vector provided).

Discuss necessity to write several versions for:

• Flat polygons
• 3d (non-flat) polygons that can, or cannot, be projected on a plane
• Direct computation ? planar projection ? sphere projection (non-euclidian geometry) ?

Possible strategy : decompose into triangles (with barycenter as common summit) and use exact formulation (see existing algo in tofu and this page :
https://en.wikipedia.org/wiki/Solid_angle
)

Inline that algorithm

A saved GDetect instance for which only the LOS and Etendues had been computed did not have a Sino_RefPt (None).
When trying to load it, an error was raised because the saving routine had turned it into:
np.array(None)

That array has shape=(), but it is not None, thus raising an error in the _Detect_checkinputs() routine called when re-creating the Detect instances from the saved file.

ZeroDivisionError raised from tf.geom._GG02.pyx in kInOut calculation with optimized algorithm in Linear case due to double identical points in concatenated Struct polys.

Reproduce with (on WEST):

import tofu as tf

dids = dict(shot=53069)
interf = tf.imas2tofu.load_Diag(diag='interfero', dids=dids, Name='test')
bolo = tf.imas2tofu.load_Diag(diag='bolo', dids=dids, geom=False, Name='test')
ece = tf.imas2tofu.load_Diag(diag='ece', dids=dids, geom=False, Name='Test')

ece.plot_combine([interf, bolo])

_Ves_mesh_dlfromL_cython
_Ves_meshCross_FromD

Improve the travis meta to handle a larger diversity of builds using a build matrix to get more control of:

• Python versions (finer granularity: 2.7.0, 2.7.1, 2.7.2..., 3.6.0, 3.6.1, .... 3.7.0)
• Libraries versions (matplotlib 2 vs matplotlib 3, numpy scipy in particular)
• Distros (linux, mac, Windows) and associated GLIBC versions

Apply to conda recipe and pip wheels

Not very urgent

Think about a replacement strategy

indentation error when defining methods for video class.
There is problem with checking whether path exists or not

On certain configurations, it appears that the projections of the viewing cone computed via tofu.geom._compute._Detect_set_ConePoly() are not consistent with a computation of the solid angle of those points taking into account Colis=True by tofu.geom._compute._Detect_SAngVect_Points().

See figure attached for an example on ITER, in the divertor region (the points and thoses stored in Detect._SynthDiag_Points)