Inability to extend message with both repeating groups and vardata about fix-simple-binary-encoding HOT 7 CLOSED

If messages and repeating groups had fields in the block header to indicate the number of repeating groups and var data items then the protocol would have been much more flexible without adding significant overhead. Better code reuse would have more than offset performance cost of the extra fields. OTF code would also have been more simple.

This would have allowed for full extension capabilities, more generic and cleaner code in decoders, plus useful facilities like skipping to the end of the message without requiring a framing protocol.

These are lessons learned and cannot be predicted upfront without building a full implementation. Now the lessons have been learned further revisions of the protocol would be remiss to ignore these lessons.

SBE could be a codec that is suitable for most financial needs rather than supporting more disparate "standards".

New templates have been suggested a number of times as an alternative. This is not practical for many organisations and technology practices. Forcing upgrades and rework is not a community focused or considerate view point. Why do we keep seeing suggestions like this in the finance world? ;-) Finance needs to evolve to be more customer focused and collaborative.

from fix-simple-binary-encoding.

I do not see this as a significant limitation when using SBE for high performance trading which was the main driver for the design. Fixed length of messages was seen as a key feature. Repeating groups cannot be avoided completely, e.g. to report back multiple partial fills of an order in a single message. Variable length fields are only needed for verbose error or other free text fields used for display purposes and should be avoided as much as possible. What are the use cases on a business level that suffer from this rule that you perceive as a significant limitation?

There are other binary encodings for FIX that support variable length messages (e.g. ASN.1, FAST) and may fit better for an interface that depends heavily on variable-length data. The different encodings all have their pros and cons and SBE is about small, fixed length messages. Any encoding for FIX needs to be able to support the complete FIX Repository. Some of the FIX messages contain heavily nested structures, hence the existence of repeating groups as a concept in SBE. In practice, nested structures should be avoided as much as possible when designing FIX messages together with an SBE encoding. Backward compatibility is not a key concern for firms active in the high speed trading arena.

Last not least, SBE is now a Draft Standard which means that changes to the wire format of Version 1.0 are only possible if something fundamental arises during the implementations undertaken by the community. I suggest to keep your request for a future version of SBE.

from fix-simple-binary-encoding.

@kleihan -- thanks for your response.

I understand that this is a draft standard, and that my comments are almost certainly way too late to be useful. I decided to report the issue so that it was publicly documented, at least.

It's obvious from the result that a key tension in the design of SBE was the desire to support the FIX application layer on one hand, and a desire to "compete" with OUCH-like protocols on the other (fixed length, easy FPGA parsing, etc).

Fixed-structure protocols evolve by (a) adding fields on the end of existing messages, or (b) adding new messages. SBE has chosen to support a more complex model of evolution but that's then compromised, and the result is that neither the evolution model nor variable length data and groups is "nice": both have limitations and inconsistencies.

Variable-length data and groups introduce the need to specify the length and group dimensions in the wire message (unlike the rest of the protocol where the layout is fixed by the schema). Having made that concession, it seems unfortunate not to have added an extra octet that would avoid the restriction from s5.1.1, and allow new vardata and groups to be added as desired.

from fix-simple-binary-encoding.

I do see another evolvement option for fixed-length messages. It is the use of templates where (e.g. as an exchange) one issues a new template and supports the old one for a limited time. That allows to insert fields at any point for the new template and convert messages provided under the old template as a convenience. These days, the new stuff is often of regulatory nature and users have no choice than to support/populate the fields as soon as the exchange makes them available.

I do not want to discard your issue in general and am looking for more people joining in this discussion which I deem to be important in terms of positioning SBE and its strengths/weaknesses correctly.

from fix-simple-binary-encoding.

Duplicate of issue #26. Keeping open for comments.

from fix-simple-binary-encoding.

Note: this enhancement changes wire format and therefore is a breaking change with version 1.0.

from fix-simple-binary-encoding.

The proposed implementation was not accepted but the goal was met in this release.

from fix-simple-binary-encoding.