Recommendation to Set Application Headers on AMQP 1.0 Messages about cfx HOT 3 OPEN

Excellent suggestion. I had to configure a AMPQ server and set different Exchanges for different publishers in order to have a kind of additional property for the message.
Your solution will be a much better way to do it.
Thank you

from cfx.

Hi Tim,

I totally agree with your analysis. I have made some tests by using the "routing key" property. I assume "application properties" and "routing key" are the same thing but I am not an AMQP expert.
Can you confirm?.

Anyway my test was to automatically set the Topic of the CFX message as the routing key and it worked, so I can say that I validate the feasibility.

I will discuss this topic at the next meeting (19th March).

from cfx.

I have made some tests by using the "routing key" property

Hi @alexisfouqueteuroplacer, thanks for taking the time to test this and verify it.

I think this is a topic that will require a bit of discussion and background to find the best option so I'm going to use this comment to add some AMQP context for the broader audience that can hopefully help advance the discussion because the distinction between routing key and target vs application properties has changed between AMQP 0.9.1 and AMQP 1.0 and is a bit nuanced.

IPC-CFX mandates AMQP 1.0, however, I think most users probably use RabbitMQ as their broker (which the CFX working group recommends in their Getting Started guide). RabbitMQ is a native 0.9.1 broker and supports AMQP 1.0 via a plugin that maps the AMQP 1.0 concepts down to AMQP 0.9.1 concepts so a lot of things that "don't make sense" generally in AMQP 1.0 do make sense if you're careful and are using RabbitMQ as your broker.

Background - target vs routing key

AMQP 1.0 vs AMQP 0.9.1

It's tempting to think of AMQP 1.0 as a "minor version bump" over AMQP 0.9.1 but its actually not. It's a fairly radical rethinking of AMQP concepts.

In particular, in AMQP 0.9.1 a message has a routing key, which is a string that indicates how the message should be routed to receivers. In AMQP 0.9.1 a message is published to an exchange with a routing key. The interpretation of the routing key is 100% determined by the exchange and you can implement a custom exchange to do whatever you want but generally people don't and one of 3 policies is applied to the routing key:

• topic: typical pub/sub style message routing where subscribers put filters on parts of the routing key that they care about and parts that they don't and then they receive all messages that match their wildcards. This is more-or-less directly equivalent to how MQTT works.
• direct: receivers receive only those messages whose routing key exactly matches a fixed string that they define.
• fanout: receivers receive all messages.

AMQP 1.0 is different. There are no exchanges and (as far as I can tell based on reading the standard), no routing keys either. I believe this change was made to allow AMQP 1.0 to work in point-to-point contexts without a broker in the middle so it couldn't mandate the concept of an exchange as part of the standard because that's inherently a "broker" concept and AMQP 1.0 was intended to also support situations where there are no brokers.

Instead of publishing with a target exchange and routing key, AMQP 1.0 just includes a source and target string that are opaque values whose interpretation can be whatever the source and receiver agree on.

IPC-CFX Use of AMQP 1.0

IPC-CFX mandates that it works on top of AMQP 1.0, which means that it does not have routing keys. It does have target strings though. However, all IPC-CFX machine vendor implementations that I have seen set target as follows:

• target is a fixed value hardcoded into the factory machine and the same for all messages (set by the machine owner during setup time and not dynamic in any way)
• source is a fixed value that is basically a unique identifier for the factory machine

So, while it's possible to use the target string to automatically set the routing key to include topic information on RabbitMQ brokers, this is not compatible with the way IPC-CFX machines currently set the target since it would require that the message target be different for each message.

AMQP 1.0 on RabbitMQ

I think one of the points of confusion here is due to the specific way that the RabbitMQ message broker supports AMQP 1.0 via a standard plugin. RabbitMQ is not inherently an AMQP 1.0 broker. It is an AMQP 0.9.1 broker (with exchanges and routing keys). So, it needs to map all of the AMQP 1.0 concepts (like target) to AMQP 0.9.1 concepts so it can deal with them.

The way RabbitMQ chose to interpret the AMQP 1.0 concept of target is as /<exchange>/<routing key> so, if you know you are sending an AMQP 1.0 message to a RabbitMQ broker, you can construct the target string in a such a way as to recover the exchange and routing key concepts from AMQP 0.9.1.

See [1].

Header Exchanges

As a workaround for the fact that IPC-CFX machines already have a defined use of target and hence routing key that doesn't include filtering by topic or other properties, I was looking for a backwards compatible option that would allow for opting-in to complex message routing without requiring a reinterpretation of any current behavior.

This is why I thought to use the headers instead of routing key (though @alexisfouqueteuroplacer I agree with you that both could be used to accomplish the same purpose).

AMQP 1.0 defines key/value headers that you can use to route messages and RabbitMQ maps those to a "Headers Exchange" concept that lets you do that routing even if you use RabbitMQ as your broker implementation.

A header exchange lets you filter messages based on rules that apply to their header fields independent of what the routing key is, which is why it can be backwards compatible with the way IPC-CFX implementations currently set target to a fixed value and hence routing key also to a fixed value.

See [2].

Options

1. We could repurpose target and include additional dynamic information in it so that it forms a routing key that includes (for example) topic information about the message and then use a topic exchange to filter messages.
2. We could define an add additional application properties while leaving the target field as is.

In either case, I think the choices that need to be made are:

1. Are we going to support use cases that require message filtering / routing?
2. What are the message features that are going to be surfaced to filter/route based on?

I think the answer to question 1 is very important and believe it should be YES since I see a lot of challenges scaling IPC-CFX implementations to large numbers of machines without this ability.

I think the answer to question 2 requires a lot of engagement and thought to make sure the filtering ends up being useful.

I propose (based on our own use cases, so others will definitely have other ideas) that the most useful features to filter on are:

• message topic
• isa95 machine logical location (site, area, line, work cell, etc).

My reasoning is based on the following considerations:

1. Filtering on message topic is important to allow for different applications that consume different kinds of information (so local line displays can consume performance messages while MES systems consume detailed traceability messages).

2. Filtering based on isa95 or similar logical factory hierarchy, I also see as important in reducing the factory implementation cost and complexity of adopting IPC-CFX. Without it, I think it will be tough to have many IPC-CFX lines connected to a centralized data exchange (say one per factory). Every subscriber to that exchange would need to memorize and keep updated the list of which machines are on which line in which area and filter out all of the other machines.

   For example, consider a performance tracking system that just shows a simple OEE calculation. Using IPC-CFX this could be an entirely general and automatic thing. You subscribe to the performance messages and group them per line to show machine performance per line. However, right now there's no reliable way to know which line a given machine belonged to without hardcoding that information in your application or pulling it from a separate database.

References

[1] https://github.com/rabbitmq/rabbitmq-amqp1.0#routing-and-addressing
[2] https://www.rabbitmq.com/tutorials/amqp-concepts.html#exchange-headers

from cfx.