Platform : Debian buster.

• If I'm using pydifact 0.1.3 with Python 3.7 everything is fine.
• If using pydifact 0.1.4 with Python 3.9 everything is fine
• If using pydifact 0.1.4 with python 3.7 it crashes at import time with this stacktrace :

    from pydifact.segmentcollection import RawSegmentCollection
/usr/local/lib/python3.7/dist-packages/pydifact/__init__.py:23: in <module>
    from pydifact import segmentcollection, parser, segments, serializer, token, tokenizer
/usr/local/lib/python3.7/dist-packages/pydifact/segmentcollection.py:37: in <module>
    class AbstractSegmentsContainer:
/usr/local/lib/python3.7/dist-packages/pydifact/segmentcollection.py:95: in AbstractSegmentsContainer
    self, name: str, predicate: Callable[[Segment], bool] = None
E   TypeError: 'ABCMeta' object is not subscriptable

Hello
I've recently reinstall pydifact with PyPI.
My programm load a file, do some changes in data and serialize.

All was working with version pydifact-0.0.5. Edifact is parsed ok by my banking software.
But with the last version, when i load an existing edifact file with from_file() and then serialize, my bank software reports error during controls.

UNZ in the original file is : 'UNZ+1+XXXDVINT'
In the serialized file, the function get_footer_segment changes the value by : 'UNZ+17+XXXDVINT'
I not sure (i am not an edifact expert...) but second value should be 1 and not 17.
Maybe the function should only count UNH segment ?

Thanks

Consider an interchange like

UNA:+.? '
UNB+UNOC:1+1234+3333+200102:2212+42'
UNH+42z42+PAORES:93:1:IA'
UNT+2+42z42'
UNZ+1+42'

While the Parser correctly uses the given control characters to convert the tokens to Segments, it does not pass on the UNA Segment to the created Interchange. Therefore, the interchange has no knowledge of any from the default set differing characters:

./tests/test_segmentcollection.py::test_interchange_with_custom_character Failed: [undefined]AssertionError: assert ',' == '.'
  - .
  + ,
def test_interchange_with_custom_character():
        i = Interchange.from_str(
            "UNA:+.? '"
            "UNB+UNOC:1+1234+3333+200102:2212+42'"
            "UNH+42z42+PAORES:93:1:IA'"
            "UNT+2+42z42'"
            "UNZ+1+42'"
        )
>       assert i.characters.decimal_point == "."
E       AssertionError: assert ',' == '.'
E         - .
E         + ,

tests\test_segmentcollection.py:264: AssertionError

or that an UNA segment was parsed:

./tests/test_segmentcollection.py::test_interchange_with_una Failed: [undefined]assert False
 +  where False = <pydifact.segmentcollection.Interchange object at 0x0000010EF8F34B50>.has_una_segment
def test_interchange_with_una():
        i = Interchange.from_str(
            "UNA:+,? '"
            "UNB+UNOC:1+1234+3333+200102:2212+42'"
            "UNH+42z42+PAORES:93:1:IA'"
            "UNT+2+42z42'"
            "UNZ+1+42'"
        )
>       assert i.has_una_segment
E       assert False
E        +  where False = <pydifact.segmentcollection.Interchange object at 0x0000010EF8F34B50>.has_una_segment

tests\test_segmentcollection.py:253: AssertionError

The docstring of convert_tokens_to_segments hints, that passing information of any parsed UNA was intended but has not been implemented yet.

Hi,

I tried to play around a little bit with pydifact but I seem to be unable to get it to run.

I successfully installed it via pip on Windows. When I then try to run the example from the README I get the following error:

Traceback (most recent call last):
  File "g:/test.py", line 1, in <module>
    from pydifact.segmentcollection import SegmentCollection
ModuleNotFoundError: No module named 'pydifact.segmentcollection'

Could you please help me on how to proceed?

All format strings could be changed into f-Strings - but due to @theangryangel's requirements in a project (see PR), compatibility to Python 3.5 will be maintained until this project is upgraded.

This issue can be closed after his requirement is upgraded.
BTW @theangryangel - what would be nice: a git pre-commit hook that checks compatibility with Python 3.5... or a CI with a Python 3.5 environment... Just if you find time.

Hey,
first, thanks for this amazing library, it has helped us a lot. I have noticed, that the in #37 proposed changes introduced more than a mapping functionality. With 23f218d fixes for the PluginMount and the SegmentFactory were introduced, which allow the inheritance based extension of Segment, while correctly evaluating the __omitted__ attribute.

With the current release this code

class TestSegment(Segment):

    tag = "TES"

    __omitted__ = False

    def validate(self) -> bool:
        pass


def test_has_plugin():
    plugins = SegmentProvider.plugins
    assert TestSegment in plugins

fails with

./tests/test_segment.py::test_has_plugin Failed: [undefined]assert TestSegment in []
def test_has_plugin():
        plugins = SegmentProvider.plugins
>       assert TestSegment in plugins
E       assert TestSegment in []

tests\test_segment.py:66: AssertionError

Could we merge those changes prior to the mapping extension? I will provide the necessary commits, but the credits belong to @theangryangel.

Hi,

this is the example, changed sligtly, it shows the input message as output. But the for loop outputs nothing?
Do i maybe need a older python for this?

cat first.py
from pydifact.segmentcollection import Interchange
#interchange = Interchange.from_file("./tests/data/order.edi")
interchange2 = Interchange.from_str(
"UNA:+,? 'UNB+UNOC:1+1234+3333+200102:2212+42'UNH+42z42+PAORES:93:1:IA'UNT+2+42z42'UNZ+2+42'"
)
for message in interchange2.get_messages():
print(message)
print(message.segments)
for segment in message.segments:
print('Segment tag: {}, content: {}'.format(
segment.tag, segment.elements))

I have run the test suit successfully on my machine. CI should support this from now on as well.

I decided to change the license from LGPL to a more "permissive" one like MIT. The "original" license of the php pendant is Apache 2.0 license.
A user asked me if he can use it in a commercial project. MIT helps to spread it more. People that are willing to give something back by creating a PR do that anyways. People who use software illegally do it anyways, even if restrictive, proprietary licenses forbid that.

With this change I hope more people can use this software.

I know pydifact is described as unstable, but a PyPI release would make it easier to adopt and might drive some contributors. As far as I can tell, it's not currently available.

It seems like the UNT element outputs the wrong order here:

https://www.truugo.com/edifact/d96a/unt/

The lib outputs:
UNT+1+272'

For a message with 273 elements. Actually it should be:
UNT+273+1'

Am I wrong? Is D96A a different standard?

In terms of high-level design and rough API, what do you think about :

# Former Message, made abstract class
class AbstractSegmentsContainer:
    @classmethod
    def from_str():
        pass
    @classmethod
    def from_file():
        pass
    @classmethod
    def from_segments():
        pass

    def get_segments():
        pass
    def get_segment():
        pass
    def add_segment():
        pass
    def serialize():
        pass

class Interchange(AbstractSegmentsContainer):
    def __init__(
            delimiters,
            timestamp,
            interchange_control_numbers,
            password_and_application_reference=None
            …
    ):
        pass

    
    def get_messages(msg_type=None):
        """
        Iterate through messages, optionaly filtering by msg_type
        """
    def get_message(msg_type=None):
        """ Same logic as get_segment"""
    

class Message(AbstractSegmentsContainer):
    def __init__(
            message_reference_number,
            message_identifier,
            common_access_reference_number=None,
            status_of_the_transfer=None,
    ):
        pass

It would allow to iterate over the segments as pydifact already does, or to go more semantically, digging Interchange−>Message->Segment.

Il let appart the functional groups for now, as they are optional, but they would fit easily in this model.

I was trying to serialize an Interchange with custom delimiters property, but it did not seem to have any effect. I was always getting UNA:+,? '. Turns out, in Interchange constructor:

    def __init__(
        self,
        sender: str,
        recipient: str,
        control_reference: str,
        syntax_identifier: Tuple[str, int],
        delimiters: Characters = Characters(),
        timestamp: datetime.datetime = None,
        *args,
        **kwargs
    ):
        super().__init__(*args, **kwargs)
        self.sender = sender
        self.recipient = recipient
        self.control_reference = control_reference
        self.syntax_identifier = syntax_identifier
        self.delimiters = delimiters
        self.timestamp = timestamp or datetime.datetime.now()

the self.delimiters property is never actually used. In serialize() method inherited from AbstractSegmentsContainer, in reality self.characters is passed into Serializer class. I believe this is just misnaming and the delimiters property in Interchange class should be renamed to characters.

Hi Christian,

Is this library 'stable' yet? If not, what are the current limitations?

It seems that a lot of changes have been introduced in pydifact since the last release on pypi. Are there plans for a new 1.0 release any time soon?

"import pydifact" failed as module 'collections' does not seem to have an attribute 'Iterable'.

following the error message

File "/Users/mike2/Documents/Python3/EDI1.py", line 3, in
from pydifact.segmentcollection import Interchange
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/pydifact/init.py", line 23, in
from pydifact import segmentcollection, parser, segments, serializer, token, tokenizer
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/pydifact/segmentcollection.py", line 382, in
class Interchange(FileSourcableMixin, UNAHandlingMixin, AbstractSegmentsContainer):
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/pydifact/segmentcollection.py", line 498, in Interchange
cls, segments: Union[list, collections.Iterable]
AttributeError: module 'collections' has no attribute 'Iterable'

I just want to gather a few ideas on how to implement interchange syntax elements like UNB, UNH, UNZ etc.: Messages, Control Header, and other things.
There are syntax implemetation guidelines of the UNECE, and other sources in the internet, like here

Best thing would be to write some sort of "syntax description language" which describes these syntax elements, and parse them. There are a few possibilities for the notation of this "syntax description":

• external files (YAML, TOML, CSV etc.)
• python classes

I had an idea writing these descriptions as classes, like Django does this in it's ORM, describing a syntax element by a class using attributes:

class CharType(Enum):
    ALPHA = 0
    ALPHANUMERIC = 1
    NUMERIC = 2

class SyntaxElement:
    """Common description of an EDIFACT syntax element"""
    def __init__(
        self,
        id: str,
        mandatory: bool = False,
        type: CharType = None,
        length: int = None,
        max_length: int = None,
    ):
        # checks...

        self.mandatory = mandatory
        self.type = type
        self.max = False
        self.length = 0
        if length:
            self.length = length
        elif max_length:
            self.length = max_length
            self.max = True

The first bytes of an Interchange Control Header would be then:

class InterchangeControlHeader:
    syntax_identifier = SyntaxElement(
        id="0001", mandatory=True, type=CharType.ALPHA, length=4
    )
    syntax_version_number = SyntaxElement(
        id="0002", mandatory=True, type=CharType.ALPHANUMERIC, length=1
    )
    service_code_list_directory_version_number = SyntaxElement(
        id="0080", type=CharType.ALPHANUMERIC, max_length=6
    )
    character_encoding = SyntaxElement(
        id="0133", type=CharType.ALPHANUMERIC, max_length=3
    )

@JocelynDelalande what do you think about that?

Hello,

I want to work on this project, and the first thing I'd like to do is improve the documentation. Is this OK? I have a question already, and as I read the existing documentation (and code) carefully, I'm sure I'll have lots more questions, so I'll be asking them here.

First question: Is AbstractSegmentsContainer meant to be used by the user (i.e. the developer that uses pydifact), or only by the developer (i.e. the developer that develops pydifact itself)? My opinion (and standard practice in projects such as Django) is that if it's an internal, we should not bother the user.

Hi

I'm interested in using this module but it needs a basic HOWTO so its possible to see how to use it.

Mike

I am running the following code:

from pydifact.message import Message
from pydifact.segments import Segment

INVOIC = Message()

def UNA():
    INVOIC.add_segment(Segment('UNA', ":+.? '"))

UNA()

result = INVOIC.serialize()

As you can see in the UNA segment, I have :+.? ' and this gets converted to :+,? ' when I run the .serialize() line.

I don't want the '.' to get converted to a ','. Is there a way around to stop this from happening?

The German energy sector uses a EDIFACT subset for market communication (edi@energy), which implements a set of control characters differing from the current defaults (see the decimal separator on p. 49 Allgemeine Festlegungen zu den EDIFACT- und XML-Nachrichten).

Currently, an UNA segment is only mandatory if any control character differs from the defaults. Therefore, parsing an edi@energy interchange without a UNA-segment is possible and results in errors.

I suggest we allow the injection of a preconfigured Parser in the creation methods FileSourcableMixin.from_file and AbstractSegmentsContainer.from_str, as well as modify the Parser to respect the following order of control characters:

1. UNA-segment of interchange
2. characters passed during call-time of parse
3. preconfigured character set
4. pydifact defaults

This would have different benefits:

• the above-mentioned modular control characters
• reuse of Parser objects
• as well as enabling new parsers, e.g.:
  • While dealing with large interchanges like load profiles; a Parser which only pareses a subset of segments is especially useful, because converting all segments might take some time, but only a specific segment per message is actually required at this processing step.

I try to draft a PR for the latter one as soon as possible, but as this would be a new feature, I think we should keep it separately, for now. :)

As we do have test, let's enforce them better :-)

Btw: It would be nice if each segment had a reference to the exact input line (byte position start/end) to ease debugging. That might also help handling malformed files. (I just swapped our self-written edifact parser against pydifact and it was a pretty pleasant experience. However exact byte positions was the one feature we lost in that process.)

Originally posted by @FelixSchwarz in #24 (comment)

I saw that the PHP counterpart had introduced a test for multibyte support.
Pydifact should test that too.

Hello again :)

I've continued to play a bit with pydifact and I some unexpected behaviour when feeding malformatted EDIFACT files to pydifact
I'm using the following code snippet for testing:

for segment in collection.segments:
    print('{}: {}'.format(segment.tag, segment.elements))

For example, when feeding
IMD+F++:::Item description 'for normals'
This results in the following, which is somewhat ok, although you'd have to check for the validity of each tag:

IMD: ['F', '', ['', '', '', 'Item description ']]
for normals: []

But when I feed the following, pydifact doesn't handle the input gracefully and I'm presented a TypeError:
IMD+F++:::Item description 'for :normals'
Result:

Traceback (most recent call last):
  File "g:/GitHub/file.py", line 2, in <module>
    collection = SegmentCollection.from_file("G:\\GitHub\\DESADV_line_break.txt")
  File "C:\<...>Python\Python38-32\lib\site-packages\pydifact\segmentcollection.py", line 60, in from_file
    return cls.from_str(collection)
  File "C:\<...>Python\Python38-32\lib\site-packages\pydifact\segmentcollection.py", line 70, in from_str
    return cls.from_segments(segments)
  File "C:\<...>Python\Python38-32\lib\site-packages\pydifact\segmentcollection.py", line 85, in from_segments
    return cls().add_segments(segments)
  File "C:\<...>Python\Python38-32\lib\site-packages\pydifact\segmentcollection.py", line 115, in add_segments
    for segment in segments:
  File "C:\<...>Python\Python38-32\lib\site-packages\pydifact\parser.py", line 212, in convert_tokens_to_segments
    yield self.factory.create_segment(name, *segment)
  File "C:\<...>Python\Python38-32\lib\site-packages\pydifact\segments.py", line 78, in create_segment
    return Segment(name, *elements)
  File "C:\<...>Python\Python38-32\lib\site-packages\pydifact\segments.py", line 36, in __init__
    raise TypeError("'tag' argument must be a str")
TypeError: 'tag' argument must be a str

My question is, whether this is considered a bug or rather outside of the scope of pydifact (check for syntax validity somewhere else). And following up on that, what would you consider to be the best way of checking syntax validity (with or without pydifact)?

What is called Message in pydifact is not a message in EDIFACT sense of the term.

This is realy confusing.

As pydifact does not currently handle Message Interchange and Functional group objects.

Here are some ways how pydifact.message.Message could be renamed IMHO :

• pydifact.document.Document : "document" word as no meaning in EDIFACT
• pydifact.SegmentCollection would be the more accurate

I know this is painful (because breaking retro-compatibility).

I could be interested (I have my own interest into doing that) into implementing those containers into pydifact, but this bug is a blocker for that.

I kinda struggle with edifact encoding, but here what I came up to :

data:

# https://blog.sandro-pereira.com/2009/08/15/edifact-encoding-edi-character-set-support/
# https://www.truugo.com/edifact/d09a/cl0001/
# A bit unsure of how 10646-1 maps exactly to utf-8
EDIFACT_ENCODINGS = {
    "UNOA": "ascii",  # iso-"646",
    "UNOB": "ascii",  # iso-"646",
    "UNOC": "iso-8859-1",
    "UNOD": "iso-8859-2",
    "UNOE": "iso-8859-5",
    "UNOF": "iso-8859-7",
    "UNOG": "iso-8859-3",
    "UNOH": "iso-8859-4",
    "UNOI": "iso-8859-6",
    "UNOJ": "iso-8859-8",
    "UNOK": "iso-8859-9",
    "UNOW": "utf-8",  # "10646-1",
    "UNOX": "iso-2022-jp",  # "2022 2375",
    "UNOY": "utf-8",  # "10646-1",
}

deserializing helper:

def guess_edifact_encoding(stream):
    unb_line = b"\n"
    eof_marker = b""
    while not unb_line.startswith(b"UNB") and unb_line != eof_marker:
        unb_line = stream.readline()

    if not unb_line.startswith(b"UNB"):
        raise ParseError("Missing UNB segment: ")

    else:
        # Must be ASCII-only
        unb_line_s = unb_line.decode()
        parser = Parser()
        unb_segment = list(parser.parse(unb_line_s))[0]
        try:
            # Ignore version, always v1…
            encoding_element = unb_segment.elements[0][0]
            return EDIFACT_ENCODINGS[encoding_element]
        except KeyError:
            raise ParseError(f"Wrong encoding spec : {encoding_element}")

I wonder what pydifact could embed in its scope in terms of :

• helper (data)
• serialization helper (like having a Interchange.serialize_to_bytes() helper with automatic encoding selection based on syntax identifier ?)
• deserialization from bytes handling decoding with a guesser like the one I wrote

Any thought appreciated :-).

Hey,

unfortunately, b1215db is raising an issue in case of unspecified characters :

observed behaviour

add.segement(Segment("UNA"))
...

acess the empty elements list in the Message class:

self.characters = Characters.from_str(segment.elements[0]

suggestion

            if segment.elements:
                self.characters = Characters.from_str(segment.elements[0])

I hope that helps 😃

Having files more than a few kb slows down the performance drastically.

I am planning to generate EDIFACT file using PYDIFACT. As suggested in the usage am using PIP install, but i am unable to import pydifact.segmentcollection to generate a EDIFACT file.
Please advise