If no headers are given, the return value of _headers_repr() is None as the return statement (

Feature Description

As reported by some users, the validation step can be very slow on large dataframes (for instance sliced FCC lattice).

In #107 the validation step was made optional. This is a first step but it would be nice for users who wish to validate to be able to do so faster.

Possible Implementation

The main culprit is is_not_finite(x) in the validate function, which is slow and applied with applymap, also very slow.

This can be sped-up (see implementation).

Feature Description

Option to check for madx-compatibility on writing

Possible Implementation

into the writer add a flag: madx_compatibility with the options None (default), warn or error.
If not none it should be checked that:

a) the header contains TYPE
b) header-names do not contain spaces

... maybe more?

Currently tfs.tools.significant_digits returns strings, for further use and consistency the possibility to return floats would be advantageous

If a column with strings has spaces... that's problematic

We're declaring tfs-pandas to be MIT licensed but the LICENSE file does not reflect that. Easy fix

Since we moved away from Travis CI, the build badge should be changed to reflect the github build status.

I suggest to change it to the status of cron tests (which will be on no status until the first cron kicks in on Monday).

@lmalina @JoschD

Not urgent, but if we want to use pathlib in omc3 it should be used here as well so we don't have to convert back to string all the time

FixedTfs does not work with pandas 1.0 and newer. Pandas are getting more flexible. Fix seems to require quite deep digging and redefinition of basic pandas.DataFrame methods. If resolved the behaviour of FixedTfs and of TfsDataFrame will diverge. Different behaviour may be misleading since the use cases are very similar, if not the same.
Now the questions to answer are:

• Is anyone actually using FixedTfs?
• If so, is it worth the effort to make it work with newer pandas?
• Or we just get rid of it?

Make changes for consistency, following pylhc/omc3#263

Empty dataframes might cause the writer to crash. Observed behavior:

a) only index given (columns empty):
Crashes in handler.py:238 as the check in line 234 should read if len(data_frame.columns) == 0:

b) only columns given (index empy):
Writes out empty dataframe, containing columns but also the column names as index

Problem:
When writing a dataframe without specifying the dtype beforehand one often runs into the error
unkown format '%s' for object of type 'int' or similar, because dtype=object, the default if pandas does not know the datatype, maps to '%s' in our writer.

I would not automatically try to solve it, but I would add an easy solution. Talked to Lukas: Have it as a function in tools, then you can also use it on pandas DataFrames. (please share opinion @Mael-Le-Garrec @lmalina @mihofer @awegsche
)

from functools import partial
import pandas as pd
(...)
def auto_dtype(df):
    return df.apply(partial(pd.to_numeric, errors='ignore')) 

because new validation checks if has valid headers, but assumes df always has headers

Comparing our tfs output to madx tfs output shows great disadvantages in readablitity in out tfs files.

Suggestions

• Allow for fixed header widths
• If save_index is used, left-align the first (index) column

Writer crashes on non-string (or non iterable) column names.

ddf = tfs.TfsDataFrame([[1, 2.], [3, 4.]])
tfs.write_tfs('fdfsdf.tfs', ddf)

ddf
   0    1
0  1  2.0
1  3  4.0

leads to

Traceback (most recent call last):
  File "tfs/handler.py", line 187, in write_tfs
    _validate(data_frame, f"to be written in {tfs_file_path.absolute()}")
  File "tfs/handler.py", line 474, in _validate
    if any(" " in c for c in data_frame.columns):
TypeError: argument of type 'int' is not iterable

• Fix
• WRITE TEST

At the moment, if any of TfsDataFrame.append, TfsDataFrame.join, TfsDataFrame.merge or tfs.concat encounter a pandas.DataFrame in their input, they will fail as they try to query the object's headers.

We should make a patch to accept these.

Same as pylhc/omc3#257

Since all necessary 3.9 wheels for our dependencies are out, I will also add 3.9 to the CI configurations: in cron and tests.

Print a nicer header, please

tfs still uses set-env in the coverage workflow.

Various CI build raise the following DeprecationWarning, which should be fairly simple to fix.

DeprecationWarning: `np.str` is a deprecated alias for the builtin `str`. To silence this warning, use `str` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.str_` here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations

Feature Description

As reported by some users, the validation step can be very slow on large dataframes (for instance sliced FCC lattice).

A good first step would be to make the validation step optional and leave the choice to users.

Possible Implementation

A new bool argument in the reader and writer functions, and an if statement before calling validate.

As we mentionned on friday with @lmalina, this could use a few classifiers. I'll include it in #52 too.

Should we add an eol (\n) at the end of the tfs file?

VIM is complaining and if you do cat file.tfs the next prompt starts directly after the values of the last column:

To be tested if this is then still readable by a) us b) madx (which also has no eofeol)

ddf = tfs.TfsDataFrame([[1, 2.], [3, 4.]], columns=['int', 'float'])
tfs.write_tfs('fdfsdf.tfs', ddf)

ddf
   int  float
0    1    2.0
1    3    4.0

crashes with

ValueError: Unknown format code 'd' for object of type 'float'

• WHY????
• fix
• TEST

Make changes for consistency, following pylhc/omc3#263

not really needed and the constant coverage reports are annoying

See the last slide of Alexander's presentation here: MAD-NG introduces %b (boolean) in the twiss headers, which Alexander suggested we support.

Should be a simple case of:

• adding the type to ID_TO_TYPE in the handler.
• give a specific handling in _dtype_to_id_string when pdtypes.is_bool_dtype(type_) is true (currently gives back %d as for an integer)
• add tests for the behavior

Important to check: what does MAD-X do when reading these (unknow for it) types? As @JoschD suggested we could create a flag (compatibility) for write_tfs that determines how to handle this.

@lmalina @JoschD @mihofer

Hello, I like this package and I use it a lot. So thanks for writing it.

I am trying to use it to load output from MADX PTC for the first time and the current implementation of the loader is crashing because of the presence of non-standard lines mixed in with the main table data. A snipped of the data is pasted below (leaving out the header). Obviously it is crashing when trying to load the lines starting with #segment. My understanding of these lines is they mark an observation point with all particles below belonging to that point. The first number on the line is just the index and counts up. Second number I don't know what this is. Third number is the number of lines in that segment. Fourth number is possibly the index of the sampler in the beamline. The name at the end if the name of the observation point.

Maybe this is a "won't fix / not an issue" but if not then it would be nice to consider how to implement this. I don't mind doing it. No doubt if this was treated by default it would drastically slow down the loading. So ideally would be a flag to enable the parsing.

$                    %le                    %le                    %le                    %le                    %le                    %le                    %le                    %le                   
#segment       1       2      11       0 start                                          
  1.000000000000000e+00  0.000000000000000e+00 -7.485050595000000e-03 -3.486651945000000e-03  1.408320528500000e-02 -3.486651945000000e-03  0.000000000000000e+00 -5.000000000000000e-03  0.000000000000000e
  2.000000000000000e+00  0.000000000000000e+00 -5.988040476000000e-03 -2.789321556000000e-03  1.126656422800000e-02 -2.789321556000000e-03  0.000000000000000e+00 -4.000000000000000e-03  0.000000000000000e
  3.000000000000000e+00  0.000000000000000e+00 -4.491030357000000e-03 -2.091991167000000e-03  8.449923171000000e-03 -2.091991167000000e-03  0.000000000000000e+00 -3.000000000000000e-03  0.000000000000000e
  4.000000000000000e+00  0.000000000000000e+00 -2.994020238000000e-03 -1.394660778000000e-03  5.633282114000000e-03 -1.394660778000000e-03  0.000000000000000e+00 -2.000000000000000e-03  0.000000000000000e
  5.000000000000000e+00  0.000000000000000e+00 -1.497010119000000e-03 -6.973303890000001e-04  2.816641057000000e-03 -6.973303890000001e-04  0.000000000000000e+00 -1.000000000000000e-03  0.000000000000000e
  6.000000000000000e+00  0.000000000000000e+00  0.000000000000000e+00  0.000000000000000e+00  0.000000000000000e+00  0.000000000000000e+00  0.000000000000000e+00  0.000000000000000e+00  0.000000000000000e
  7.000000000000000e+00  0.000000000000000e+00  1.497010119000000e-03  6.973303890000001e-04 -2.816641057000000e-03  6.973303890000001e-04  0.000000000000000e+00  1.000000000000000e-03  0.000000000000000e
  8.000000000000000e+00  0.000000000000000e+00  2.994020238000000e-03  1.394660778000000e-03 -5.633282114000000e-03  1.394660778000000e-03  0.000000000000000e+00  2.000000000000000e-03  0.000000000000000e
  9.000000000000000e+00  0.000000000000000e+00  4.491030357000000e-03  2.091991167000000e-03 -8.449923171000000e-03  2.091991167000000e-03  0.000000000000000e+00  3.000000000000000e-03  0.000000000000000e
  1.000000000000000e+01  0.000000000000000e+00  5.988040476000000e-03  2.789321556000000e-03 -1.126656422800000e-02  2.789321556000000e-03  0.000000000000000e+00  4.000000000000000e-03  0.000000000000000e
  1.100000000000000e+01  0.000000000000000e+00  7.485050595000000e-03  3.486651945000000e-03 -1.408320528500000e-02  3.486651945000000e-03  0.000000000000000e+00  5.000000000000000e-03  0.000000000000000e
#segment       2       2      11     104 LUXE.TARGET                                    
  1.000000000000000e+00  1.000000000000000e+00  5.259466027436782e-03 -2.051903873665704e-02 -2.978780358169780e-02 -2.541279540830233e-02  0.000000000000000e+00 -5.000000000000000e-03  7.730849426828061e
  2.000000000000000e+00  1.000000000000000e+00  4.392648890935142e-03 -1.665085832937077e-02 -2.340368089034140e-02 -2.025864365870777e-02  0.000000000000000e+00 -4.000000000000000e-03  7.730849426828061e
  3.000000000000000e+00  1.000000000000000e+00  3.431943743212929e-03 -1.266012177677902e-02 -1.722476714179673e-02 -1.513417573531897e-02  0.000000000000000e+00 -3.000000000000000e-03  7.730849426828061e
  4.000000000000000e+00  1.000000000000000e+00  2.378645362977142e-03 -8.551614443416757e-03 -1.126011892202533e-02 -1.004604018279110e-02  0.000000000000000e+00 -2.000000000000000e-03  7.730849426828061e
  5.000000000000000e+00  1.000000000000000e+00  1.234161014750900e-03 -4.330038920110554e-03 -5.516836135196690e-03 -4.999821487667771e-03  0.000000000000000e+00 -1.000000000000000e-03  7.730849426828061e
  6.000000000000000e+00  1.000000000000000e+00  1.060641785314563e-30 -1.389211385140205e-30  8.408973718086066e-15  1.106205531972171e-14  0.000000000000000e+00  0.000000000000000e+00  7.730849426828061e
  7.000000000000000e+00  1.000000000000000e+00 -1.322236565765053e-03  4.434010064600291e-03  5.287384342390506e-03  4.950085226447743e-03  0.000000000000000e+00  1.000000000000000e-03  7.730849426828061e
  8.000000000000000e+00  1.000000000000000e+00 -2.730865609241517e-03  8.967625924981525e-03  1.034444121494745e-02  9.848270945371038e-03  0.000000000000000e+00  2.000000000000000e-03  7.730849426828061e
  9.000000000000000e+00  1.000000000000000e+00 -4.224131469513009e-03  1.359662305874779e-02  1.517247445916981e-02  1.469360189857558e-02  0.000000000000000e+00  3.000000000000000e-03  7.730849426828061e
  1.000000000000000e+01  1.000000000000000e+00 -5.800214661983473e-03  1.831693117876930e-02  1.977501489057500e-02  1.948636102584803e-02  0.000000000000000e+00  4.000000000000000e-03  7.730849426828061e
  1.100000000000000e+01  1.000000000000000e+00 -7.457239940043075e-03  2.312464709990392e-02  2.415786594644772e-02  2.422809810282515e-02  0.000000000000000e+00  5.000000000000000e-03  7.730849426828061e```

I moved tests to using pathlib.Path inputs in #30, which in hindsight should have been an addition and not a change.

Let's bring string inputs back to tests, in order to test both.
This should be a quite small change: files similar to actual tests, but with the old fixtures inputs using os.path.join shenanigans.

To improve consistency among our packages capitalize the plane keys for "planed" data frames, i.e. ("X", "Y") instead of ("x", "y").
Requires change of _define_property_two_planes(args, kwargs).

One needs to check the use of issubdtype (see https://numpy.org/devdocs/release/1.19.0-notes.html) before bumping numpy dependency to 1.19 (or not).

Related to #48

Sphinx parses docstrings a bit strangely. We may also update the example in TfsCollection (should be at least .tfs files).

See:
https://github.com/Syntaf/travis-sphinx

also the travis.yml
also the /doc folder
also the settings (to define branch for github.io)
also put travis-sphinx in the requirements, so that travis installs it automatically

Obviously not needed anymore

Should be checked if still needed, but most likely not.
Should just be removed from config.py in doc folder.

Feature Description

Provide a tfs definition for reference

Possible Implementation

Write it into the README

• general definition (how to write header, columns, types, content, what does our reader allow)
• MADX specifics (e.g. headers need to be single word, TYPE needs to be in header, all entries are converted to lowercase etc.)

I'm not sure why we have this in the first place, apart from the fact that we later (i.e. omc3) need unique indices. But we might check there ...?

Especially if you save a DataFrame without save_index the check doesn't really make sense.

Metaclass is not working in python3 the way it was working in python2, needs to be fixed.
Also: Include some tests for collection!

Pandas 1.0 are soon to be released and they break things

This happens in standard output of MADX. The types %[num]s specifies string of given length.

Instead of leaving it to the user, we could add our own

• append()
• concat()
• join()
• merge()

functions, that use the pandas one for the data but additionally takes care of the headers.

This was implemeted to keep compatibility with metaclass, but it assumes that there is a "NAME" column in the dataframe.
We are usually loading the frames directly with this column as index read(path, index="NAME") and never used the indx, I think.

Does anyone see a problem with removing that ununsed feature?

@Mael-Le-Garrec @jaimecp89 @rogeliotomas @lmalina @awegsche @mihofer @fsoubelet

When having no headers (headers={}) tfs.write() writes out an empty line before the column definitions.

example: https://github.com/xenon-middleware/xenon/blob/3.0.1/.zenodo.json

Mainly for keywords and authors (rest should be taken from github).

Discuss Authors. CERN OMC-Team?