fixtradingcommunity / fix-simple-binary-encoding Goto Github PK

The example for business message reject in section 7 shows the encodings but is missing the message template.

It would be useful to provide the examples as complete XML file in resources rather than just snippets in the document. Then developers could test schema parsing.

Section 2 is vague about the member names of variable-length data composite type. The example does not match names used elsewhere and in the conformance test suite. The names should be "length" and "varData".

All standardized composite types should have explicit member names. For example, the decimal type has members "mantissa" and "exponent".

https://github.com/FIXTradingCommunity/fix-simple-binary-encoding/blob/master/v1-0-RC4/doc/07Examples.md

1. In XML schema there is an extra character '`' in the first line:

```

1. xmllint and other tools complain:

$ xmllint flat-fixed.xml
flat-fixed.xml:6: namespace error : Namespace prefix xsi for schemaLocation on messageSchema is not defined
xsi:schemaLocation="http://fixprotocol.io/sbe/rc4SimpleBinary1-0.xsd">
^

To fix it either xsi:schemaLocation should be changed to "xs:schemaLocation" either "xmlns:xs="http://www.w3.org/2001/XMLSchema-instance" should be changed to "xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance".

Hi Don and others,

Is there any SBE guidance/ standard what's the best binary type to use for Days to Options expiration (DTE)? DTE used as whole number (annualized), but on the day of option expiration - DTE used for options theoretical pricing will decay by hour. So the same field that is communicated as INT must also support decimal on the day of expiration.
I was thinking just to use decimal composite with exponent uInt32 and -3 mantissa.
Given Today is May 16 and option expiration is May 18 then DTE=2.000
Given Today is May 18 2 PM and option expires in 2 hrs then DTE= 0.083 (2/24)

Are there any other standard SBE options we can use here?

How can a constant be provided on a field like line 292 in the following?

https://github.com/real-logic/simple-binary-encoding/blob/master/sbe-benchmarks/src/main/resources/fix-message-samples.xml

How is one supposed to provide a default null value like in the following message schema?

https://github.com/real-logic/simple-binary-encoding/blob/master/sbe-benchmarks/src/main/resources/fix-message-samples.xml

Line 318 is a good example. This schema works with the reference implementation but only if schema validation is turned off.

I believe all qualifiers like nullValue, presence, etc. should be available on types and fields. nullValue seems to only work on encodedDataType which is insufficient given restricted types can be created plus we have Enums and Bitsets.

Hello,
What is the reason for Data binary type 'Variable-length data' part changing it primitive type from char to int between RC2 and RC4?
Is this change backward compatible for RC2 implementations?

The following example is in the spec and it does not pass schema validation as presence and nullValue are not allowed on enum:

<enum name="optionalBoolean" encodingType="uint8" presence="optional"
    nullValue="255" semanticType="Boolean">
    <validValue name="false">0</validValue>
    <validValue name="true">1</validValue>
</enum>

https://github.com/FIXTradingCommunity/fix-simple-binary-encoding/blob/master/v1-0-STANDARD/doc/02FieldEncoding.md

It would be helpful if all examples where tested for schema validation and with a reference implementation.

Hi Don and company,
Which Semantic type should I use for a field that requires a decimal encoding, but is not a Price, Qty, Amt or Percentage type?
It looks to me that SBE should allow for generic Decimal semantic type which isn't limiting the fields to be of any of the above FIX types.

Hi,

May I know if variable length field is properly supported in repeating group, especially in terms of extending the schema by adding a new variable length field in the group?

I think it is currently impossible to implement a consumer with older version of decoder to decode a message encoded with new schema version and var-length field added in repeating group.

The reason is the blockLength of the group consists only the length of fixed-length fields.
Older version of decoder can't determine the length of the new var-length field, thus not able to advance the offset correctly to the next group entry.

Thanks.

Potential enhancements to consider for SBE 2.0:

#54
#53
#47
#26
#37
#35
#31

Clarifications requested for SBE 2.0:

#51
#48
#44
#40
#39
#36
#30

In early drafts, XML attributes epoch and timeUnit were proposed. However, no other epoch was ever defined besides the UNIX attribute. TimeUnit is handled is handled in the wire format rather than just in metadata. Therefore, the two XML attributes may be removed in version 2.0 of the schema.

Do we have a URL for the 1.0 final XSD for SBE?

Hi Don,
Which way is the right way to specify null-able decimal with not fixed exponent?
Do we specify only mantissa null value or the exponent also?

Will this work?
composite name="decimal" >
type name="mantissa" primitiveType="int64" presence="optional" nullValue="9223372036854775807" />
type name="exponent" primitiveType="int8" />

Or
composite name="decimal" >
type name="mantissa" primitiveType="int64" presence="optional" nullValue="9223372036854775807" />
type name="exponent" primitiveType="int8" presence="optional" nullValue="127" />

If there is no need to provide null value for exponent, then what should be the exponent value on the wire with null mantissa?

I looked up the spec and couldn't find examples of null-able decimals.

The schema extension chapter says "Compatibility is only ensured under these conditions:"

What does "compatibility" mean? Certainly, an old decoder can parse a message generated by a newer compatible schema. But what happens if a new decoder parses a message generated by an older schema? What are the semantics?

(It's worth noting that the C++ reference implementation is unlikely to work well in this case: real-logic/simple-binary-encoding#300)

If I define a reusable type such as a Boolean enum then how can I reuse that across multiple composites?

Where I have a constant field in a message I want to be able to more simply set a constant value for it. As an example I currently need to do something like this:

<types>
<enum name="constval" encodingType="uint8">
    <validValue name="one">1</validValue>
</enum>
</types>

<sbe:message name="example" id="1" description="value setting">
    <field name="constant" type="uint8" id="2" presence="constant" valueRef="constval.one"/>
</sbe:message>

and I'd like to be able to do this:

<sbe:message name="example" id="1" description="value setting">
    <field name="constant" type="uint8" id="2" presence="constant" value="1"/>
</sbe:message>

Essentially I think we could support the value= tag on a constant field in a backward compatible way to aid usability and clarity.

For payload like Mass Quote and Market Data Incremental Refresh, I would like to align price and quantity fields at 8-byte (word) boundaries to maximize my (and my counterparties') theoretical encoding/decoding speed. SBE Message Header is 8 bytes, and I have crafted my root block and repeating group so that MDEntryPx, etc. are word-aligned.

However if I prepend the 6-byte SOFH, all these efforts are thwarted.

Have you considered any "standard" solutions to this?

I propose to add a uint16 "blockOffset" field my Message Header, causing the Message Header to become 10 bytes, so the SOFH + Message Header becomes 16 bytes, and my SBE Message payload is again word-aligned. I encode the value 10 (0x0a) as the blockOffset and will instruct my counterparties to decode the SBE Message starting at that offset (relative to the Message Header). The "blockOffset" could equivalently be interpreted as "messageHeaderLength".

This has an added benefit of allowing me to specify blockOffset=16, and pad with six zeros between the Message Header and SBE Message payload, for cases where we choose not to use SOFH for certain transports. Or if alignment is not needed (small / non-numeric payload) then use blockOffset=10 (no padding) regardless of whether the SOFH is present.

My "messageHeader" now looks different from the recommendation.

<composite name="messageHeader" description="Template ID and length of message root">
    <type name="blockOffset" primitiveType="uint16"/>
    <type name="blockLength" primitiveType="uint16"/>
    <type name="templateId" primitiveType="uint16"/>
    <type name="schemaId" primitiveType="uint16"/>
    <type name="version" primitiveType="uint16"/>
</composite>

Any other ideas?

Hello!
I have question about difference between composite type, group and message.
I asked already here and Martin recommended me recreate issue in this project.

Can I make shorter a schema contains an entity which I want to be able to send either
as a message or as a part of another message or as a group?
For example I have an asset. In java classes it will be something like this

public class Asset {
    char[] isin;
    char[] name;
}

Obviously I should able to send asset as message.
But asset can be part of quotation

public class Quotation {
    double quotation;
    Asset asset;
}

and a clients portfolio contains a list of assets

public class Portfolio {
    char[] portrolioName;
    Collection<Asset> assets;
}

if I right understood my schema will be something like this:

    <types>
        <composite name="groupSizeEncoding" description="Repeating group dimensions">
            <type name="blockLength" primitiveType="uint16"/>
            <type name="numInGroup" primitiveType="uint16"/>
        </composite>
    </types>
    <types>
        <type name="str" primitiveType="char" length="12"/>
        <type name="quote" primitiveType="double"/>
        <composite name="AssetType">
            <type name="isin" primitiveType="char" length="3"/>
            <type name="name" primitiveType="char" length="3"/>
        </composite>
    </types>


    <sbe:message name="Asset" id="1">
        <field name="isin" id="1" type="str"/>
        <field name="name" id="2" type="str"/>
    </sbe:message>
    <sbe:message name="Quotation" id="2">
        <field name="quotation" id="1" type="quote"/>
        <field name="asset" id="2" type="AssetType"/>
    </sbe:message>
    <sbe:message name="Portfolio" id="3">
        <field name="portrolioName" id="1" type="str"/>
        <group name="assets" id="2" dimensionType="groupSizeEncoding">
            <field name="isin" id="1" type="str"/>
            <field name="name" id="2" type="str"/>
        </group>
    </sbe:message>

In this schema fields isin and name repeats 3 times. Can I create a schema a little bit shorter?

Note that this is only example I know there is better way for this task.

Hi,
Can you please clarify the relationship between SBE specifications and FIX Simple Binary Encoding ?
I was initially under the impression SBE and Fix Simple Binary Encoding are the same thing, but reading the License section in the readme they seem have different licensing.

SBE specifications are ©Copyright 2014-2016 FIX Protocol Limited.

FIX Simple Binary Encoding by FIX Trading Community is licensed under a Creative Commons
Attribution-NoDerivatives 4.0 International License.

Hi,

In master branch, the v1-0-STANDARD sbe.xsd says it's Version: 2.0 Release Candidate 1

But there's also a separate folder for v2RC1 at https://github.com/FIXTradingCommunity/fix-simple-binary-encoding/tree/master/v2-0-RC1/resources

For now I've used tag'd release v1.0 to get Draft Standard https://github.com/FIXTradingCommunity/fix-simple-binary-encoding/blob/v1.0/v1-0-STANDARD/resources/sbe.xsd

This is a little confusing..

In v1.0 draft 2016/06/16, section 2.3 lists a set of common schema attributes for the field element, including minValue, maxValue, and nullValue.

However, in the schema (XSD), fieldType defined starting line 272, can have only name, id, type, epoch, timeUnit, semanticType, description, sinceVersion, deprecated, offset, and presence attributes.

Attributes minValue, maxValue, and nullValue are defined as legal for encodedDataType only (see starting line 137 in the XSD).

My reading of s2.3 (and reinforced by s2.3.1), leads me think you should be able to specify minValue and maxValue directly on a field element.

Is the XSD correct? If so, I think it'd be worthwhile to tidy up the text in s2.3.

Update the XML schema namespace to point to the location of the XSD in this GitHub repository.

We have a requirement in which a wrapper message is composed of a number of sub messages or templates with each of these representing specific sections of data, for example:

|--------------message-----------------------------------------------|
|---common---||---specific message---||---metadata--|

Common template represents data which is common across all messages, specific message refers to NewOrderSingle, OrderCancelReplaceRequest etc and metadata refers to internal system data

This way if changes are made to a specific template such as metadata then the impact could be isolated to consumers of that section only

Currently each of these sub messages or templates is required to be present in its own schema with a separate schema for the wrapper message but instead of having to maintain multiple schemas is it possible to include the wrapper template and all of these sub templates in a single schema?

Looking at v1.0 Draft Standard dated June 16, 2016.

Section 2.2 contains a table of fix semantic types with references to the section in which they are defined. The FIX semantic type 'Boolean' on page 14 references section 2.12.6 which should be 2.13.6.

The PDF properties suggest the document was produced using Microsoft Word 2016 and I'm wondering if the indexing needs to be rerun before pdf generation or whether these references are hand done and need a careful check.

I haven't done any further checking other than find this mismatch sorry.

The XSD for 1.0 allows for the sinceVersion attribute to be set on nested types within a composite. This is contrary to the spec as it does not state how composites can be extended.

What is the correct way (if any) to limit the number of occurrences allowed for a given repeating group ?

I can't find anything related in the specification, the only limit seems to be related to the primitiveType used for numInGroup in the dimension composite.

I just had a quick question regarding the use of nested repeating groups. Does the NumInGroup (number of entries) of the outer repeating group include an entry for the nested repeating group? I know that the blockLength of the outer group excludes it but I was not sure about the NumInGroup.

Also while encoding do we have to first encode all entries of the outer repeating group followed by the nested repeating group? What does this statement in the specification mean?

"Nested repeating groups are encoded on the wire by a depth-first walk of the data hierarchy. For example, all inner entries under the first outer entry must be encoded before encoding outer entry 2. (This is the same element order as FIX tag=value encoding.)"

Wouldn't the nested repeating group be at the end of the outer repeating group so this would be encoded at the end after all of the preceding outer group entries have been encoded?

Example of nested repeating group specification:

group name="ListOrdGrp" id="2030"
field name="ClOrdID" id="11" type="string14" semanticType="String"/
field name="ListSeqNo" id="67" type="uint32" semanticType="int"/
field name="Symbol" id="55" type="string8" semanticType="String"/
field name="Side" id="54" type="char" semanticType="char"/
field name="OrderQty" id="38" type="intQty32" semanticType="Qty"/
group name="Parties" id="1012"
field name="PartyID" id="448" type="string14" semanticType="String"/
field name="PartyRole" id="452" type="int" semanticType="int"/
/group
/group

How can I deal with repeating groups when occurrences differ in length ?
I have no guarantee that all occurrence of a repeating group have the same length:

• repeating group has a variable length field defined: length of variable field can vary
• repeating group has a nested repeating group: number of occurrences of nested group can vary

But in the dimension of the repeating group block, I need to specify a unique value for the length.

So I guess the implied constraint is I MUST ensure that all occurrences have the same length ?

In v1.0 draft 2016/06/16, section 2.13.4:

Side field specification references the enumeration type
<field type="Side" name="SideEnum" id="54" />

The type and name values are reversed: the name should be "Side", and the type should be "SideEnum", so that it matches the declaration of the enum type earlier in the section.

Use the fixprotocol.io domain as the root of XML namespaces. That domain is specific to FIXTradingCommunity organization in GitHub. The suggested format is to append version, for example 'http://fixprotocol.io/sbe/rc3' for Release Candidate 3.

As currently stated in the spec, namespace URI is only for unique identification, not intended as a URL to be accessed at application run-time. However, convenient links will be provided for developers to download the XSD file at development time.

Is the element 'epoch' defined at

'epoch' is not mentioned in https://github.com/FIXTradingCommunity/fix-simple-binary-encoding/blob/v1.0/v1-0-STANDARD/doc/04MessageSchema.md

Also message/data elements are not described in the MessageSchema.md file. Because these are expected to be variable length data types, should there be some kind of validation rules for data elements listed in section https://github.com/FIXTradingCommunity/fix-simple-binary-encoding/blob/v1.0/v1-0-STANDARD/doc/04MessageSchema.md#schema-validation

e.g. is this an error? <data name="bad" id="523" type="uint8" />

Copy pasting from a different page as a new issue:

RFrenkel commented:
Hi Don,
Would you recommend schema version change when need to correct semantic type from String to Int for a field using Int32 binary type (SecurityID in this case)? We didn't find any rules describing rule regarding this type of correction. We don;t see how mismatch of Semantic type in this case may create an issue with decoding.
@donmendelson
FIXTradingCommunity member
donmendelson:
We discussed a closely related use case in the FIX Orchestra working group ("machine readable rules of engagement"): the FIX datatype (application layer) of SecurityID may be dependent on SecurityIDSource, string in one case or numeric in another. We are making a provision for that in the Orchestra schema.

In my view, change of application layer datatype is a significant change, even if the wire format doesn't change, and therefore, should result in a version bump. However, we should get others' views on this. Rosa, please enter this as a new issue to be considered for any future SBE versions.

In v1.0 draft 2016/06/16, section 2.13.5:

<enum name="PartyIDSourceEnum" primitiveType="char">
<validValue name="BIC">B</validValue>
<validValue name="GeneralIdentifier">C</validValue>
<validValue name="Proprietary">D</validValue>
</enum%gt;

Note that the <enum> element has a primitiveType attribute.

However, section 2.13.3 states:

An <enum> element must have an encodingType attribute to specify the type of its values.

So, I think that the example in 2.13.5 should use encodingType in place of primitiveType? That's consistent with other examples of enum as well.

encodedDataType in this xsd https://github.com/FIXTradingCommunity/fix-simple-binary-encoding/blob/master/v1-0-RC4/resources/SimpleBinary1-0.xsd
lists characterEncoding as an optional attribute

But characterEncoding is not mentioned in the documentation for Type in https://github.com/FIXTradingCommunity/fix-simple-binary-encoding/blob/master/v1-0-RC4/doc/04MessageSchema.md#type-element-content

The README.md file at real logic SBE on github states

This repository contains the reference implementations in Java and C++

but I could not find any reference to that project in the draft FIX SBE specification.

Is there such a thing as a reference implementation of FIX SBE and if that is the case, to what extent is that implementation of FIX SBE certified by the FIX Trading community ?

It would be useful to have a description available for a refType. For example, there could be two or more refs to the same type in a composite and each could need different descriptions.

We just had a quick question about the extensibility of the template ID. This is the particular section in the specification which outlines backward compatibility:

Compatibility is only ensured under these conditions:

• Fields may be added to either the root of a message or to a repeating group, but in each case, they must be appended to end of a block.
• Existing fields cannot change data type or move within a message.
• A repeating group may be added, but only after existing groups and if there are no subsequent variable data elements at the end of the message.
• A variable data element may be added, but only after existing groups and data.
• Message header encoding cannot change.

Changes that break those constraints require consumers to update to the current schema used by publishers. In general, metadata changes such as name or description corrections do not break compatibility so long as wire format does not change.

If an enumeration is being used in a particular template ID then do we need to assign a new template ID if additional values are being added to the enumeration? Does this qualify as changing data type of an existing field?

For example NewOrderSingle has template ID 99 with sideEnum containing values 1 & 2 and if we were to add an additional value to this enumeration for cross (8) then will a new template ID be required or it is possible to just update the schema version and utilize the sinceVersion attribute?

    <enum name="sideEnum" encodingType="enumEncoding">
        <validValue name="Buy" description="Buy">1</validValue>
        <validValue name="Sell" description="Sell">2</validValue>
    </enum>

<sbe:message name="NewOrderSingle" id="99" blockLength="54" semanticType="D">
    <field name="ClOrdID" id="11" type="idString" description="Customer Order ID"
           offset="0" semanticType="String"/>
    <field name="Account" id="1" type="idString" description="Account mnemonic"
           offset="8" semanticType="String"/>
    <field name="Symbol" id="55" type="idString" description="Security ID" 
          offset="16" semanticType="String"/>
    <field name="Side" id="54" type="sideEnum" description="Side" offset="24" 
          semanticType="char"/>
    <field name="TransactTime" id="60" type="timestampEncoding" 
          description="Order entry time" offset="25" semanticType="UTCTimestamp"/>
    <field name="OrderQty" id="38" type="qtyEncoding" description="Order quantity"
           offset="33" semanticType="Qty"/>
    <field name="OrdType" id="40" type="ordTypeEnum" description="Order type" 
          offset="37" semanticType="char"/>
    <field name="Price" id="44" type="optionalDecimalEncoding" 
          description="Limit price" offset="38" semanticType="Price"/>
    <field name="StopPx" id="99" type="optionalDecimalEncoding" 
          description="Stop price" offset="46" semanticType="Price"/>
</sbe:message>

Currently, the schema only allows blocks of fields to be defined inline as the root of a message or as a repeating group. However, blocks of fields are often reused in multiple FIX messages. Repeating groups belong to multiple messages. For example, InstrumentLegGrp is used in many message types, including SecurityStatus and MarketDataStatisticsReport. Additionally, the FIX Repository (from which message specs are generated) also has a concept of a common component, even when it is not part of repeating group. For example, Instrument component is group of fields that hold all the attributes of a traded instrument. It is used the majority of FIX message types but only needed to be defined the Repository once.

SBE schema should have equivalent features to define repeating groups and blocks once and reference them in any number of messages.

Hi,

The presence attribute is allowed at the field level and the type level

• am I right assuming the field presence attribute takes precedence over the type presence attribute?
• given nullValue can only be specified at the type level, is it correct to define a type with presence="required" nullValue=XXX like in exemple below ?

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<sbe:messageSchema xmlns:sbe="http://fixprotocol.io/2016/sbe"
                   package="sbe.test"
                   id="1" version="1" semanticVersion="0.1"
                   description="Example schema"
                   byteOrder="littleEndian">
    <types>
        <type name="VehicleCode" primitiveType="char" length="6" characterEncoding="ASCII"/>
        <type name="SeatCount" primitiveType="uint8" presence="required" nullValue="255"/>
    </types>
    <sbe:message name="Car" id="1" description="Description of a basic Car">
        <field name="vehicleCode" id="1" type="VehicleCode"/>
        <field name="seats" id="2" type="SeatCount" presence="optional"/>
    </sbe:message>
    <sbe:message name="SuperCar" id="2" description="Description of a super Car">
        <field name="vehicleCode" id="1" type="VehicleCode"/>
        <field name="seats" id="2" type="SeatCount"/>
    </sbe:message>
</sbe:messageSchema>

• If the above is not allowed, then I guess the other way round would work:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<sbe:messageSchema xmlns:sbe="http://fixprotocol.io/2016/sbe"
                   package="sbe.test"
                   id="1" version="1" semanticVersion="0.1"
                   description="Example schema"
                   byteOrder="littleEndian">
    <types>
        <type name="VehicleCode" primitiveType="char" length="6" characterEncoding="ASCII"/>
        <type name="SeatCount" primitiveType="uint8" presence="optional" nullValue="255"/>
    </types>
    <sbe:message name="Car" id="1" description="Description of a basic Car">
        <field name="vehicleCode" id="1" type="VehicleCode"/>
        <field name="seats" id="2" type="SeatCount" />
    </sbe:message>
    <sbe:message name="SuperCar" id="2" description="Description of a super Car">
        <field name="vehicleCode" id="1" type="VehicleCode"/>
        <field name="seats" id="2" type="SeatCount" presence="required"/>
    </sbe:message>
</sbe:messageSchema>

• Or should we just be defining different types for optional and required ? I noticed CME is doing that.

Thank you.

SBE depends on ISO 4217 for currency codes, among other standards. ISO 4217:2008 mentioned in the spec has been revised by ISO 4217:2015.

Hi,

I am trying to validate https://github.com/FIXTradingCommunity/fix-simple-binary-encoding/blob/master/v1-0-RC4/resources/Examples.xml

Using https://github.com/FIXTradingCommunity/fix-simple-binary-encoding/blob/master/v1-0-RC4/resources/SimpleBinary1-0.xsd

it fails to validate

xmlstarlet val -s tests/data/sbe_rc4.xsd -e tests/data/Examples_rc4.xml 
tests/data/Examples_rc4.xml:6.72: Element 'messageSchema': No matching global declaration available for the validation root.
tests/data/Examples_rc4.xml - invalid

Is the xsd only there to guide protocol developers, or is it intended to be used to validate protocol spec xml files?

I can get this to validate if I change the Example.xml <messageSchema> element to <sbe:messageSchema>

Simple Open Framing Header provides registration of constants for different payload encodings. Constants have already been assigned for SBE 1.0 big and little endian payloads. Equivalent constants are needed for SBE version 2.0.

How composites have now been specified I'm unsure about the implementation. How does one specific the offset of an enum, set, or composite within a composite? Only encoded data types seem to carry the offset attribute.

In v1.0 draft 2016/06/16, section 5.1.1, it states:

A repeating group may be added, but only after existing groups and if there are no subsequent variable data elements at the end of the message.

This seems like a significant limitation of the protocol, and essentially prevents the use of both variable-length data and repeating groups in a message where schema extensions might be required.

If both the variable-length data "header" (s2.8.2), and the repeating group "header" (aka group dimensions, s3.4.10), mandated a leading byte that distinguished between vardata and a repeating group, this restriction could be lifted:

Variable data would then be preceded by

• uint8 blockType
  • Bits 0-1: power of two for number of octets in size value (0 = uint8, 1 = uint16, 2 = uint32, 3 = uint64)
  • Bits 2-3: reserved, must be encoded as zero, and ignored on reception
  • Bits 4-5: a standard value VARDATA_BLOCK = 1
  • Bits 6-7: reserved, must be encoded as zero, and ignored on reception
• unsigned integer type length The number of octets of data following, encoded as an unsigned integer

Repeating groups would be preceded by

• uint8 blockType
  • Bits 0-1: power of two for number of octets in block length value (0 = uint8, 1 = uint16, 2 = uint32, 3 = uint64)
  • Bits 2-3: power of two for number of octets in number in group value (0 = uint8, 1 = uint16, 2 = uint32, 3 = uint64)
  • Bits 4-5: a standard value GROUP_BLOCK = 2
  • Bits 6-7: reserved, must be encoded as zero, and ignored on reception
• unsigned integer blockLength Size in octets of group entry
• unsigned integer numInGroup Number of group entries

(These could of course be expanded to 2 (vardata) and 3 (repeating groups) uint8 octets to avoid bit-twiddling, if that's seen as a better approach).

I appreciate that this scheme would compromise the "all description is in the stubs, not on the wire" nature of the protocol as it is today, but as a tradeoff for enabling schema evolution with both vardata and repeating groups, it seems a small price to pay.

This relates to version 1.0 - draft standard - June 16 2016 with the relevant section being 2.7.1.2 and 2.7.2.

My issue is that I'm unable to read a clear interpretation of how I should represent constant char encodings of length 1. e.g.:

<type name="CharConstant" primitiveType="char" presence="constant">P

It's clear that a char encoding without a presence="constant" is a single character and that if presence="constant" with a constant value of length > 1, then we are to use an array.

If the constant value is of length 1, then it would be possible to represent that as an array of length 1 or as a single char.

We might compare this to a string of length 1:

<type name="ascii1" primitiveType="char" semanticType="String" length="1" CharacterEncoding="ASCII"/>

In sbe-tool's Java generator, the character constant of length 1 is represented as a byte array of length 1, whereas the String of length 1 is represented as a single character. I'm happy with this interpretation but I'd like to see the spec be explicit.

It might be nice to be able to have a constant encoding of a character (rather than a String) but I can see you could accomplish that with an enum if it were not possible. Was this the intent?

The message dumps in section 7 Examples need to be updated for the new header structures. See issue #26

We have a requirement in which a wrapper message is composed of a number of sub messages or templates with each of these representing specific sections of data, for example:

|--------------message-----------------------------------------------|
|---common---||---specific message---||---metadata--|

Common template represents data which is common across all messages, specific message refers to NewOrderSingle, OrderCancelReplaceRequest etc and metadata refers to internal system data

This way if changes are made to a specific template such as metadata then the impact could be isolated to consumers of that section only

If we could put offset fields in the “wrapper message” that point to the start of the individual sub-messages then consumers could jump straight to that particular sub-message. We could also perhaps include the total length as well of each sub message

Could this information be placed in the root block of the schema not in the message header necessarily but in a separate index like construct?

It seems the spec does not provide an option for alignment for the starting of repeating groups or var data. For example, the var data header could be a length field of type uint32 that is followed by a string of 13 bytes, then the following repeating group or var data field would have it header unaligned for the blockLength/length member.

It possible solution would be to have an alignment attribute for the schema that would be used to align fields such as repeating group headers and var data header on alignment boundaries.

As per the SBE specification:

To maximize deterministic field positions, message schemas must be specified with this sequence of message body elements:

1. Fixed-length fields that reside at the root level of the message (that is, not members of repeating groups), including any of the following, in the order specified by the message schema::
   a. Fixed-length scalar fields, such as integers
   b. Fixed-length character arrays
   c. Fixed-length composite types, such as MonthYear
2. Repeating groups, if any.
3. Data fields, including raw data and variable-length strings, if any.

Do the fixed length fields refer to both required fields as well as optional fields?