Would need to specify message field names for values such as:

Symbol
SecurityExchange (MIC code)
TransactTime (for manufacturing period structures)

The MessageHandler DatacastField subclasses need JSON Schema field support for newly manufactured fields, to bring up to the existing Avro and BQ levels of output schema support.

Ideas around revising console logging of JSON messages.

Currently, if neither --quiet nor output_type is specified, the transcoder will default to writing Avro files to --output_path or avroOut in the local directory. In parallel, JSON representations of the transcoded messages are written to standard output.

The behavior should be adjusted to:

• If no output_type is specified to the CLI, default to streaming JSON to standard out. This means that writing Avro files no longer is the default and must be explicitly chosen, like bigquery and pubsub are now.
• The design should now treat stdout as the equivalent of an output path
• The default encoding if stdout is chosen should be JSON vs. any binary format (enforce this?)
• If the minimum parameters to run the transcoder now are passed (schema file, source file, factory), the result would be JSON streamed to console. This would allow one to use the transcoder even without a disk to write to

Current pcap source input format type only supports classic PCAP and not PCAP-NG. This may entail a move to scapy from dpkt within the PcapFileMessageSource class.

Similarly to how --message_skip_bytes is used for each message found in a LengthDelimitedFileMessageSource to ignore things like UDP header information, --message_skip_chars would trim the first n characters from messages arriving via LineDelimitedFileMessageSource.

One could run the transcoder with debug options, but this slows down processing since the output is a function of a file. In some instances, one might desire seeing where a txcode process is with it's current batch non-destructively. One way to do this might be trapping kill -s INFO signal in main thread and then dump a snapshot of the progress up to that point. The dd command has a similar feature:

Sending an `INFO' signal to a running `dd' process makes it print
I/O statistics to standard error and then resume copying.  In the
example below, `dd' is run in the background to copy 10 million blocks.
The `kill' command makes it output intermediate I/O statistics, and
when `dd' completes, it outputs the final statistics.

     $ dd if=/dev/zero of=/dev/null count=10MB & pid=$!
     $ kill -s INFO $pid; wait $pid
     3385223+0 records in
     3385223+0 records out
     1733234176 bytes (1.7 GB) copied, 6.42173 seconds, 270 MB/s
     10000000+0 records in
     10000000+0 records out
     5120000000 bytes (5.1 GB) copied, 18.913 seconds, 271 MB/s

MulticastUdpNetworkSource should be configurable with the multicast group to join.

This would enable the transcoder to be run in streaming mode for POSIX inputs, for example:

http -pb 'https://www.cmegroup.com/ftp/SBEFix/Production/secdef.dat.gz' \ 
| gunzip - \
| txcode --schema_file ../datacast/transcoder/test/FIX50SP2.CME.xml --factory fix --source_file_format_type line_delimited

ApplID: '310'
Asset: ES
CFICode: FFIXSX
Currency: USD
DisplayFactor: 0.01
HighLimitPrice: 456775.0
LastUpdateTime: '20221120170356000585'
LowLimitPrice: 400425.0
MarketSegmentID: '64'
MatchAlgorithm: F
MatchEventIndicator: '00000000'
MaturityMonthYear: '202409'
MaxPriceVariation: 600.0
MaxTradeVol: 3000.0
MinPriceIncrement: 25.0
MinPriceIncrementAmount: 12.5
MinTradeVol: 1.0
MsgType: SECURITYDEFINITION
NoEvents:
- EventTime: '20220617133000000000'
  EventType: ACTIVATION
- EventTime: '20240920133000000000'
  EventType: LAST_ELIGIBLE_TRADE_DATE
NoInstrAttrib:
- InstrAttribType: TRADE_TYPE_ELIGIBILITY_DETAILS_FOR_SECURITY
  InstrAttribValue: '00000000000001000010000000001111'
NoLotTypeRules: '0'
NoMDFeedTypes:
- MDFeedType: GBX
  MarketDepth: 10
SecurityExchange: XCME
SecurityGroup: ES
SecurityID: '118'
SecurityIDSource: EXCHANGE_SYMBOL
SecurityType: FUTURE
SecurityUpdateAction: ADD
SettlPriceType: Final Actual Settlement at Clearing Tick
Symbol: ESU4
TradingReferenceDate: '20221123'
TradingReferencePrice: 428600.0
UnderlyingProduct: 5
UnitOfMeasure: Index Points
UnitOfMeasureQty: 50.0
UserDefinedInstrument: N

JsonOutputManager would need to keep track of required fields as defined in the source / dest schema and transcode appropriately. For JSON Schema, each object supports an array property named required with the values being the defined required properties in the current object definition.

https://www.codeproject.com/Articles/85336/Exploiting-Data-Parallelism-in-Ordered-Data-Stream

LengthDelimitedOutputManager takes --prefix_length to determine the size in bytes of the length prefix to output, but endianness is currently hard-coded. Additionally, the prefix length being read by LengthDelimitedFileMessageSource should be separately optioned from the prefix length written out. Hence --source_prefix_length should be an option for LengthDelimitedFileMessageSource and --output_prefix_length should be come an option instrumenting LengthDelimitedOutputManager.

When processing FIX messages, if the transcoder cannot find a tag present in the data, it fails with a KeyError. Instead, the transcoder should possibly issue a warning, but instead of halting processing, it should use the numeric tag number as they key instead and continue processing.

Would require similar mapping that's been done for output Avro encodings. This should enable us to support all available output transports and upgrade to the BigQuery Storage Write API for enhanced performance.

Some binary feed specifications have field lengths that do not correspond to the existing SBE primitive types - for example, a 6 byte integer with a field definition resembling:

<type name="timestamp_nanos_midnight" primitiveType="uint8" length="6"/>

To handle these, one can use a uint8 with a length of 6 as above, but the codec doesn't know how to interpret this properly as an integer, and I've not seen a way to instruct it to do so. It seems as if these are interpreted as UTF-8 by the codec.

For field definitions of this kind`, there should be a way to override or send a hint to the codec on how the field should be interpreted.

https://developers.google.com/protocol-buffers/docs/proto3
https://cloud.google.com/pubsub/docs/schemas#protocol-buffer

The current --base64 decoding capability doesn't automatically handle b64url decoding of payloads. Some systems, such as Pub/Sub, return message payloads as base64url decoded strings (which are safe for use in URLs and filenames). The transcoder should detect from the payload contents and use the proper codec.

Currently, in order to properly decode a Pub/Sub message payload delivered by gcloud CLI, the b64url encoding needs to be explicitly replaced with b64 classic equivalents, for example:

gcloud pubsub subscriptions pull ${SUBSCRIPTION_NAME}  --format=json | json -a .message.data | tr '_' '/' | tr '-' '+'

Without the tr on the message.data string returned from Pub/Sub, the transcoder will fail to base64 decode the original message bytes.

Default to chr(1) - SOH char

Add label to resources created by transcoder: goog-packaged-solution = datacast

This would necessarily have to bypass schema-awareness perhaps with the exception of message typing and sequencing.

--create_schemas_only option would ignore any input source and just create output schemas

I believe there is an option in the SBE decoding library that snake cases all of the decoded field names, this should be suppressed and default to verbatim transcoding of the field name as specified in the schema.

          @t8dogg I agree with that but would suggest maybe tracking that under another issue. I think we''d do a FilterHandler or similar then apply the sequencing capability. Will open under separate issue.

Originally posted by @salsferrazza in #114 (comment)

In lieu of sharding output destinations by message type, support interleaved output of all message types to a single destination file, topic, etc.

This has some implications to schema generation and most obviously would not work for BigQuery output types. It would also require that Pub/Sub topic schemas or Avro outputs be handled differently in this mode, perhaps using the Union option in Avro.

Capture abstractions and design decisions:

schema/encoding/transport abstractions in the object hierarchy
message handler framework

Some feeds only provide timestamps as duration past midnight, assuming prior context of what day the processed messages are from. Since the messages themselves don't have this context, it would be useful to be able to append a statically specified date, convert that to Unix milliseconds, and append it as a manufactured int field to outbound messages. This would reduce friction for downstream SQL analytics on messages ingested from these feeds.

e.g --message_handlers AppendDateHandler:date=20291102

would turn that date into it's UNIX seconds equivalent, and append that as column to all outbound messages.

When running with the following set of options:

wget -q -O - ftp://ftp.cmegroup.com/SBEFix/Production/secdef.dat.gz | gunzip -  | txcode \ 
 --schema_file  ~/src/datacast/transcoder/test/FIX50SP2.CME.xml  \
 --factory fix  \
 --source_file_format_type line_delimited  \
 --message_type_inclusions=SecurityDefinition \
 --fix_header_tags 8,9,35,1128,49,56,34,52,10 \
 --destination_project_id $(gcloud config get-value project)  \ 
 --output_type pubsub \
 --output_encoding json \
  --lazy_create_resources 

The following error is thrown:

google.api_core.exceptions.InvalidArgument: 400 AVRO schema definition is not valid: sbeMessage.NoLotTypeRules exists twice in schema. [detail: "[ORIGINAL ERROR] generic::invalid_argument: AVRO schema definition is not valid: sbeMessage.NoLotTypeRules exists twice in schema. [google.rpc.error_details_ext] { message: \"AVRO schema definition is not valid: sbeMessage.NoLotTypeRules exists twice in schema.\" }"

This is likely because of the complex schema defined in FIX50SP2.CME.xml, where the same field name may be present at different levels of the entity hierarchy. A graph such as Object.Property1 and Object.Property2.Property1 appears to be incompatible with Avro, but is commonly encountered within legitimate FIX schema definitions.

It's notable that BigQuery output types do not exhibit this behavior, but the fastavro output type to local POSIX files does as well:

fastavro._schema_common.SchemaParseException: redefined named type: sbeMessage.NoLotTypeRules

Similar to how avro in transcoder/output has it's own code namespace, we should similarly add google_cloud as a higher-level output category for output specific code.

The manufactured field name for sequence numbers in SequencerHandler is hard-coded to sequence_number. For FIX presentation this is not optimal. User should be able to pass this value through either via CLI (meta?)option or otherwise through the environment.

This is related to issue #76 and should probably be the illustrative use case for that implementation

Feature would output either the specified output schema with the potential ability to wrap it in a Terraform block, IE: google_bigquery_table

Decide whether to:

• Remove the notion of any threshold whatsoever and handle all byte arrays without filtering on size.
• Use a heuristic like "larger than the number of bytes in the message header definition for the particular encoding" (e.g. sum of the bytes in the fields comprising the messageHeader component for SBE). This seems like the best case long term.
• Allow for length_threshold to flow through from the user surface

Filing this as a bug since 50 bytes is likely too high of a fixed default.

an option named --human or similar would be used to visually check transcoding from sources by writing YAML to the console.

no_md_entries:
- md_entry_px:
    exponent: -9
    mantissa: 2191000000000
  md_entry_size: 4
  md_entry_type: Offer
  md_price_level: 1
  md_update_action: New
  number_of_orders: 1
  rpt_seq: 2328011
  security_id: 123890
  tradeable_size: 0
no_order_id_entries:
- md_display_qty: 4
  md_order_priority: 77606680730
  order_id: 8040595458543
  order_update_action: New
  reference_id: 1
transact_time: 1663632017199755749

Tranacoder would only crack messages open to determine their message type to add to the rolling count, then output the statistics observed:

INFO:root:Source record count: 27798842
INFO:root:Summary of message counts: {'InstrumentDirectory': 42, 'RegSHORestriction': 27798800}

Currently, the transcoder emits messages to output sources based on the message type specified in the schema. This would add a capability to further filter messages based on input dimensions observed, sharding them to separate outputs. For example, if you specify a list of --shard_keys fully qualified with the parent message type, it would manufacture a file/table/topic for each distinct value found in that dimension. So

--shard_keys SnapshotFullRefreshTcpLongQty.SecurityID

would produce and populate a separate persistent resource for each unique value of that dimension found in the input data.

Currently, MessageHandler specific configurations must be hard-coded within the handler module itself. There should be a syntax for specifying user parameters at runtime through the command line syntax. Something like:

message_handlers=TimestampPullForwardHandler:timestamp_field=transact_time

Or potentially a <MessageHandlerSubclassName>.json can be automatically checked and the dictionary passed into the MessageHandler constructor? This might give the most flexibility without rendering the CLI syntax too unweildy.

With the --replace option, the transcoder will not try to use existing cloud resources that correspond to message types in the schema (schemas, topics, BigQuery tables), rather it will delete and replace them with new ones derived from the schema represented in --schema_file. This would be a destructive action, so would have to be explicit and used with care.

Currently, custom handlers need to be authored and put into the source tree under transcoder/message/handler, in addition to modifying the __init__.py in the directory to register the new handler. It would be convenient to be able to store handler modules in arbitrary places addressed by a CLI option in order to dynamically load handler componentry

The --source_file and --schema_file options currently imply file:// if using fully qualified URL syntax. Other protocols, such as http://, ftp:// and ws:// should be possible to resolve in this way, although ws:// would only be applicable as a protocol under --source_file and not --schema_file and would need to be implemented in a manner similar to the recent stdin support addition.

Example:

txcode --schema_file https://raw.githubusercontent.com/SunGard-Labs/fix2json/master/dict/FIX50SP2.CME.xml --source_file ftp://ftp.cmegroup.com/SBEFix/Production/TradingSessionList.dat --factory fix --source_file_format_type line_delimited

Default to utf-8

The transcoder's SBE schema interpretation does not support ref attributes for embedding composite types or enums as fields in a message type definition.