Support Apache Kafka as a complement to ZeroMQ for an example Tracker Server about barefoot HOT 5 CLOSED

It's not on the roadmap in the sense that it's one of the next implementation steps. However, it's in the scope of the project and I can support you as much as possible and it would be a great fit. (In the past, we have implemented a similar solution with Apache Kafka, Apache Storm and Apache Cassandra. However, this implementation was too specific to publish it as open source.)

An important aspect of a stream solution is that real-time map matching is stateful which means that for each tracked object, one must maintain a map matching state (see MatcherKState). Hence, a major design question is how you plan to store, retrieve, and manage the state. I know Kafka Streams supports basic stateful stream processing, which makes it easy to store and retrieve state information by object ids, but at least as important as this is the question of what your application expects to access state information (spatial search, stream, ...). What does your application look like, technically speaking?

from barefoot.

Our application has two purposes:

• Real time map matching to visualize vehicles in real time.
• Real time map matching to identify segments being travelled and augment the live road network with collected information as well as with planned traffic maintenance

As of now it is only on paper but we are thinking of substituting RocksDB for local postgres/postgis databases. We don't know yet if this is feasible or desired at all.
One solution is of course to stream the output of the matcher to Kafka and use a secondary index in the form of PostGIS or something else.
On another note we foresee that we will update the underlying road network in fairly short intervals to account for real time traffic information. This is a separate flow from our intended map matching use case so the question is how "easy" is it to substitute the routing part of Barefoot since we already have that component in the form of OSRM?

from barefoot.

1. What's the purpose of the "live road network"? Real-time traffic information is not necessarily needed for map matching. Hence, I would actually decouple a "live road network", e.g. for route planning, from the road network for map matching. Of course, road network updates must also be available for map matching but this is another granularity as for real-time traffic information.

Note: A road network that can be updated without down-time is indeed on our roadmap (but only planned so far).

2. A scalable visualization may be a problem if you send lots of information for every vehicle and for every position update as Barefoot's monitor does it currently. (It sends lots of information per vehicle with every position update including probability distribution over possible positions and routes.) If you only visualize vehicle position and some context information, it should scale to several hundreds or even thousands of vehicles. I'm assuming you want to track many vehicles, otherwise a scalable solution like Kafka streams wouldn't make any sense.

Why is visualization a problem? Because a browser-based visualization would either need to manage position information of all vehicles (not very scalable, but as said would work for several hundreds/thousands of vehicles if only position information is sent) or must be capable to request vehicle information in a certain region (screen region) and must subscribe to updates in that region. The latter requires a more comprehensive spatial database/index.

In summary:

(1) I wouldn't replace the routing component of Barefoot to handle real-time traffic information and (2) visualization requires some analysis of scalability issues and only parts of Barefoot's monitor can be reused since it visualizes map matching states of objects with lots of details.

from barefoot.

I think my description was a bit vague. There are multiple use cases involved.
I'll add a picture and try to clarify.

We want to:

• build historical data of link speeds by using Barefoot as a matcher to link (we'll split OSM links that are too long)
• create predictive / near realtime link speeds given planned traffic events
• visualize vehicle positions in realtime
• use raw location events to enhance the road network both manually and semi automatically. Think Strava Slide. Barefoot is not used here.

All of this to support near real time route optimization and scheduling.

I hope this makes it more clear what we want to do.

As for point 2 we won't be using barefoot monitor. We'll be using a combination of Elixir/Phoenix framework and their channel concept. And rely heavily on Canvas/WebGL in the client.
We are talking about roughly 4000-10000 assets initially which will be sending location events every 2-5 seconds.
In addition to this we'll try to create vector tiles on the fly for a particular region where we display the actual routes.

from barefoot.

This looks very interesting and well planned. Let me focus on the question "'Barefoot Matching' uses 'Road Network"', where I have the following concern/thoughts:

Map matching relies heavily on spatial search for road segments to identify matching candidates and on routing between those candidates. For that reason, I think that the coupling strategy of matching and road network is a relevant performance issue. (1) If we assume that the road network is stored on some central server and every matching iteration must query roads and routes from that server, then performance of map matching may be horrible. To the best of my knowledge/experience, I would choose (2) each matching instance has an instance of the road network (or some tiles/regions) locally available. The consequence is that making this consistent with a "real-time" road network may be extremely complicated. Recall, real-time map matching is stateful and, hence, changes in the map on the fly may cause unforeseen problems.

For that reason, I would think about the temporal design of your system. Let's say your application requires changes to the road network (with regard to topology and geometry - so to say construction measures) on a daily base, but real-time traffic information updates (like speed parameters, etc.) every minute or second. In this case, one could exploit this temporal differentiation to decouple (a) the road network for map matching/tracking from (b) the road network for routing and route optimization with real-time traffic information.

This should be valid since - as said - map matching is not subject to real-time traffic information, because map matching follows the vehicles in real-time and not the "real-time map", even if a road is completely blocked it doesn't care and follows the objects. Further, routing/route optimization with real-time traffic information can indeed consider such incidents with updates every minute or second.

The gain is that updates of (a) the road network for map matching (within Kafka Streams) can be done daily and (b) real-time traffic information indepentenly every minute or second; and ALSO all road networks, (a) and (b), are consistent (with regard to topology and geometry) over the whole system for the daily intervals which is necessary for passing references to roads in your system based on messages. The update of (a) the road networks for map matching in Kafka Streams can be done as a daily extract from (b) the real-time map. This way, the road network of (a) would be your reference system for all messages, processing, and visualization (real-time traffic information could be an overlay, or so).

from barefoot.