example: https://github.com/xenon-middleware/xenon/blob/3.0.1/.zenodo.json

Mainly for keywords and authors (rest should be taken from github).

Discuss Authors. CERN OMC-Team?

This is the last package that needs to move to Github Actions.

Will do afte fixing #21

Make changes for consistency, following pylhc/omc3#263

Should be checked if still needed, but most likely not.
Should just be removed from config.py in doc folder.

Hi,

The new format for sdds files often uses llong format. I was able to use your code with the following modifications :
classes.py, line 17-19 : add llong format (>i8, 8, int)
reader.py line 139 : convert num_dims to int -> int(num_dims)

Thank you for the good work! Is there any plan to add ascii compatibility?

Here is an example of the file I was trying to open

LHC_BLM@08_36_18_005@[email protected]

As for the ASCII, this is the file I want to open (I can do it line by line, but that is it for now) :
Ufo_2018-06-09_22-05-42.0_BLMQI.11L8.B2I10_MQ.sdds.gz

Cheers

Feature Description

Would be nice to be able to read directly gzipped files. A minor modification to the read_sdds function would do the trick, a bit of refactoring would allow to add it on top of a common interface for both functions.

Possible Implementation

def read_sdds_gz(file_path: Union[pathlib.Path, str], endianness: str = None) -> SddsFile:
    """
    Reads SDDS file from the specified ``file_path``.

    Args:
        file_path (Union[pathlib.Path, str]): `Path` object to the input SDDS file. Can be a
            `string`, in which case it will be cast to a `Path` object.
        endianness (str): Endianness of the file. Either 'big' or 'little'.
                          If not given, the endianness is either extracted from
                          the comments in the header of the file (if present)
                          or determined by the machine you are running on.
                          Binary files written by this package are all big-endian,
                          and contain a comment in the file.

    Returns:
        An `SddsFile` object containing the loaded data.
    """
    file_path = pathlib.Path(file_path)
    with gzip.open(file_path,'rb') as inbytes:
#     with file_path.open("rb") as inbytes:
        if endianness is None:
            endianness = _get_endianness(inbytes)
        version, definition_list, description, data = _read_header(inbytes)
        data_list = _read_data(data, definition_list, inbytes, endianness)

        return SddsFile(version, description, definition_list, data_list)

Release 0.2 deployed all changes made since August 9, 2019. This means all commits from fca0994 to b0ee5b9 included.

Testing in omc3 started breaking after using this new release, as can be seen here.

The reading does not seem to pose an issue.
However, reading the result of a tbt conversion fails, encountering an invalid start byte, which indicates writing was wrong.

These tests in omc3 fail starting with sdds's commit 5d2a0d0, which is change made to writing behavior.

Should aim to fix this asap, add proper testing, and release 0.2.1.

Same as pylhc/tfs#69

Documentation

• Yes

Question

There is this other sdds package here:
www.aps.anl.gov/
Are there reasons not to use this? What are the main differences? It would be helpful to point that out on the README page.
When using elegant, you have to use their package and someone might confuse the packages, since the other package is also called sdds.

Simple but important addition of pathlib.Path objects support for sdds read / write operations.

Necessary for pylhc/omc3#238

See:
https://github.com/Syntaf/travis-sphinx

also the travis.yml
also the /doc folder
also the settings (to define branch for github.io)
also put travis-sphinx in the requirements, so that travis installs it automatically

Last package to refactor for consistency. Includes

• no bumpversion
• setup.py
• __init__.py
• docs/conf.py
• no setup_requires
• direct use of pytest
• update license to actually be MIT (as said in setup.py)
• update out-of-date parts of README

Documentation

• Yes

Question

I'm encountering an error while using this library to read SDDS5 files, meaning "class Definition does not have a field named format_string." I checked the 1995 documentation (version [1.4]) https://doi.org/10.2172/179270 on page 5, where the Definition only includes format_string without format.
I want to know where the field format in the Definition class comes from.

Recently, while using this library for SDDS5, I found that both reading and writing data were not implemented, so I made some additions. However, due to my limited understanding of the version history of SDDS, I cannot guarantee the completeness of my modifications. Can I submit these relevant changes? Alternatively, is there an updated version of this project available?

sdds still uses set-env in the coverage workflow.

Simple but important addition of pathlib.Path objects support for sdds read / write operations.

Necessary for pylhc/omc3#238

CI config should updated for better maintenance on dependencies and a simpler usage

pytest opts to coverage action. Remainder not needed

Documentation

• Yes

Operating System

Any

Python Version

Any

Package Version

0.3.0

Bug Description

If there is an empty array or an array of empty arrays, the writing function will fail with an index error in:

Steps to Reproduce

No response

Relevant output

No response

Possible Fix Implementation

Possible solution could be

return [len(value)] + get_dimensions_from_array(value[0]) if len(value) else []

but needs to be tested if this would read okay. Or if empty arrays should be allowed in the first place?

Implementation of reading and writing of column data

• Implement writing
• Implement reading

@mulingLHY mentioned they were working on something like this, so maybe we can merge this back into this package later on.
But also @lmalina has been looking into this a few years ago I think. Maybe he can also comment on the status.

Very useful for us to have in this regard would be an example file.
Our sdds files do not use columns and hence this feature has not been neccessary.

Cheers,

Josch

Example:
https://github.com/pylhc/sdds/files/3139431/Ufo_2018-06-09_22-05-42.0_BLMQI.11L8.B2I10_MQ.sdds.gz

I want to use the sdds module to process the output data of ELEGANT,but when I used sdds.read(),it turns out errors.I'm a beginner in PYTHON,could anybody sent me a example script to show how to use sdds module.
Email:[email protected]

++++++++++++++++++++++
import sdds
import os

os.chdir(r'D:\IHEPBox\elegantExamples\PAR\twissCalculation')
sdds_data = sdds.read('parTwiss.twi')
++++++++++++++++++++++++++

I use the commands shown above to load the twiss file(a sdds file),but it turns errors like this:

Traceback (most recent call last):
File "C:\Users\Saike Tian\Desktop\1111111111111111111111111.py", line 5, in
sdds_data = sdds.read('parTwiss.twi')
File "E:\ProgramData\Anaconda3\lib\site-packages\sdds\reader.py", line 25, in read_sdds
version, definition_list, description, data = _read_header(inbytes)
File "E:\ProgramData\Anaconda3\lib\site-packages\sdds\reader.py", line 43, in _read_header
def_dict: Dict[str, str] = _get_def_as_dict(word_gen)
File "E:\ProgramData\Anaconda3\lib\site-packages\sdds\reader.py", line 181, in _get_def_as_dict
[assign.split("=") for assign in parts]}
File "E:\ProgramData\Anaconda3\lib\site-packages\sdds\reader.py", line 180, in
return {key.strip(): value.strip() for (key, value) in
ValueError: not enough values to unpack (expected 2, got 1)

Documentation

• Yes

Operating System

macOS Big Sur 11.6.8

Python Version

Python 3.9.13

Package Version

cloned today

Bug Description

When reading a file, the reader fails to parse the header. The type_hints dictionary in classes.py line 118 appears to only have the type hints from the base class and not this derived class, so it fails with:

Traceback (most recent call last):
  File "/Users/nevay/physics/reps/sdds/sdds/reader.py", line 40, in read_sdds
    version, definition_list, description, data = _read_header(inbytes)
  File "/Users/nevay/physics/reps/sdds/sdds/reader.py", line 61, in _read_header
    definitions.append({
  File "/Users/nevay/physics/reps/sdds/sdds/classes.py", line 118, in __init__
    type_hint = type_hints[argname]
KeyError: 'description'

See attached screenshot of debugging in pycharm.

This doesn't seem to be specific to the data file and happens every time for me.

Cheers,
Laurie

Steps to Reproduce

Load a file as per the documentation.

Relevant output

No response

Possible Fix Implementation

No response

Feature Description

The current test covers the big endian binary file tests/inputs/test_file.sdds, would be nice if little-endian files were supported too.

The current behavior is to crash while trying to read a string of incorrect length:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "$prefix/lib/python3.9/site-packages/sdds/reader.py", line 29, in read_sdds
    data_list = _read_data(data, definition_list, inbytes)
  File "$prefix/lib/python3.9/site-packages/sdds/reader.py", line 89, in _read_data
    return _read_data_binary(definitions, inbytes)
  File "$prefix/lib/python3.9/site-packages/sdds/reader.py", line 107, in _read_data_binary
    return [functs_dict[definition.__class__](inbytes, definition) for definition in definitions]
  File "$prefix/lib/python3.9/site-packages/sdds/reader.py", line 107, in <listcomp>
    return [functs_dict[definition.__class__](inbytes, definition) for definition in definitions]
  File "$prefix/lib/python3.9/site-packages/sdds/reader.py", line 120, in _read_bin_param
    return inbytes.read(str_len).decode(ENCODING)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xee in position 20: invalid continuation byte

Possible Implementation

No response

The write is currently using len(value) as the dimension for arrays, which is wrong:

There can be multiple dimensions, nested lists:

dimensions: 36 and 4452 items
[[-0.26359558 -0.5675639  -0.29767203 ...  0.          0.
   0.        ]
 [-3.2426553  -3.8073838  -3.4528794  ...  0.          0.
   0.        ]
 [ 0.          0.          0.         ...  0.          0.
   0.        ]
 ...
 [ 0.          0.          0.         ...  0.          0.
   0.        ]
 [ 0.          0.          0.         ...  0.          0.
   0.        ]
 [ 0.          0.          0.         ...  0.          0.
   0.        ]]

In this case the writer will only write 36

Feature Description

Upon inspection of the SDDS code, I think I stumbled upon multiple possible bugs.

a) if you have entries with duplicate names, these will overwrite each other, as they are stored by name in the dict. -> this should at least raise an error somewhere (e.g. assert len(names) == len(set(names)))

b) The order of the definitions in the header is sorted (parameter, array, column) upon reading, but not the data. And it is sorted before reading the data. So basically, the data needs to be already sorted in that order. If sorting would actually sort anything, the file could not be read anymore.
Same goes upon writing: here actually the data is always written in that order, but the order of the headers is whatever came into the SDDSFile init. Here sorting the header would make sense, but it is not done!!

We never encountered problems, as we read LHC data which is already sorted and mostly use the turn_by_turn package for writing, which sorts the entries in the same way.
If it wouldn't the order in the header would be different from the order in the data and that order is actually required.

My suggestion is to remove all of that sorting and loop in the writer simply through the lists and write parameter, array, column one at a time, instead of gathering them first.

Possible Implementation

We could add a random hash to the defintions, maybe build from name + some random chars.
Then definitions + values could be combined into a single dict with definitions as keys and the corresponding values as values. (which we could also accept as alternative input to the init, so either two lists, or this kind of dict).
The only problem is, that we then have to rewrite the getter a bit, so that if a name-string is given, we actually return a tuple in case multiple entries are found.

something like that.