Streaming and WebSockets APIs about spec HOT 14 CLOSED

Hey @jkarneges! Thanks for the suggestion. It is a very interesting point.

If I properly understood, the key point behind this issue is the following:

... the APIs may not necessarily discuss brokers, topics, subscriptions, etc. ...

I think it could be interesting to modify AsyncAPI, so it doesn't require the topics section, and we could just jump into defining what the API is expecting you to publish or what the API is probably gonna send you.

I'm focusing on the topics section on purpose because brokers are already abstracted as servers following the same naming convention as OpenAPI.

So, just to confirm that we're aligned, it could be something like this:

Streaming API (i.e. Twitter Streaming API)

servers:
  - url: streaming.api.twitter.com
    scheme: http

stream:
  read: # ...or receive, subscribe, etc.
    - $ref: '#/components/messages/status'

Websockets API (i.e. Slack RTM)

servers:
  - url: rtm.api.slack.com
    scheme: ws

events:
  publish:
    - $ref: '#/components/messages/message'
    - $ref: '#/components/messages/pong'
    - $ref: '#/components/messages/typing'
  subscribe:
    - $ref: '#/components/messages/message'
    - $ref: '#/components/messages/ping'
    - $ref: '#/components/messages/typing'

Will something like this define what you mean?

from spec.

It looks like webhooks are supported by OpenAPI lately:
https://github.com/OAI/OpenAPI-Specification/blob/master/versions/3.0.0.md#callback-object

So maybe nothing needs to be done in AsyncAPI about webhooks. It is a little strange to me how the specs are divided on what async transports to support, so I include it here for completeness.

I would agree that webhooks should be left to OpenAPI's callbacks feature to describe. The re-use of the operation object makes it clear that it is a two-way REST-ish exchange rather than a true asynchronous protocol.

AsyncAPI would therefore concern itself with pub/sub and streaming APIs.

from spec.

This is now on version 1.2.0 🎉

from spec.

Good start :)

Regarding HTTP streaming, there should be a way to specify the message framing. Typically this is lines of text (like Twitter) or Server-Sent Events. Maybe add a framing field which could be lines or sse. In the case of SSE, there are some fields baked into the protocol like ID and event type that could be used. Should these be added to the Message object?

WebSocket is tricky because there could be messages that I wouldn't consider to be part of a pubsub session. For example, an API could require a subscription request/response to listen for certain messages, but the server might still send other unsolicited messages like keep alives. Also, an API might allow the client to send messages to the server to cause things to happen, for example Slack lets the client send chat messages over the WebSocket. Should that really be called a "publish" ? I'm not so sure.

One solution could be for WebSocket APIs to be able to specify both a topics section and an events section. The events section could then contain read/write (or maybe receive/send?) sections to specify messages that happen outside the context of pubsub.

from spec.

Regarding HTTP streaming, there should be a way to specify the message framing. Typically this is lines of text (like Twitter) or Server-Sent Events. Maybe add a framing field which could be lines or sse. In the case of SSE, there are some fields baked into the protocol like ID and event type that could be used. Should these be added to the Message object?

This is a great observation. Not sure where it should be specified because it's not part of the message but more of the protocol used to actually obtain the messages. Ideally, messages should contain no protocol-related information. I'll investigate more on HTTP streaming. Would appreciate links to articles/documentation I can read for a good reference 😊

WebSocket is tricky because there could be messages that I wouldn't consider to be part of a pubsub session. For example, an API could require a subscription request/response to listen for certain messages, but the server might still send other unsolicited messages like keep alives.

I understand, but it's up to who is documenting the API to specify all the possible messages or not. Like HTTP APIs, you might not document all endpoints. The only problem here is that you'll receive those messages even if you don't want.

Also, an API might allow the client to send messages to the server to cause things to happen, for example Slack lets the client send chat messages over the WebSocket.

The spec should be used to document the message exchange. The semantic meaning of those messages (events or actions) behind the scenes should not be part of this spec IMHO.

Should that really be called a "publish" ? I'm not so sure.

I think publish/subscribe are the right terms here. You publish/subscribe to a channel, where the channel is the event name. Others could be emit/receive, send/receive, etc.

One solution could be for WebSocket APIs to be able to specify both a topics section and an events section. The events section could then contain read/write (or maybe receive/send?) sections to specify messages that happen outside the context of pubsub.

Makes total sense. I'm wondering if an API could contain the 3 sections: topics, events and stream. With streaming happening on a different server/port. Or maybe they should be split into 2 different files if that was the case.

Can I ask you for other streaming and WebSockets APIs you know would be worth look at?

from spec.

There's some good SSE background here: https://www.html5rocks.com/en/tutorials/eventsource/basics/

It might be nice if AsyncAPI didn't need special awareness of different HTTP streaming formats, and just let the API designer explicitly specify the details, but this would mean AsyncAPI would need to be quite flexible about payload format definitions.

Another thing: HTTP streaming APIs tend to make use of headers. E.g. Authorization for auth, and also Content-Type (in the case of SSE, the type is text/event-stream). So those should be specifiable somewhere. WebSockets have headers too although I'm not sure of any public APIs that use them.

Some more API definitions:

https://firebase.google.com/docs/reference/rest/database/#section-streaming
https://developer.gitter.im/docs/streaming-api
https://blockchain.info/api/api_websocket
https://docs.gdax.com/#websocket-feed
https://www.cbix.ca/api-streaming
https://www.cbix.ca/api-websocket

from spec.

It might be nice if AsyncAPI didn't need special awareness of different HTTP streaming formats, and just let the API designer explicitly specify the details, but this would mean AsyncAPI would need to be quite flexible about payload format definitions.

By "payload format definitions", you mean JSON and others? If that's what you mean, I don't think it would be easy to represent other formats (except XML) using JSON Schema. I've been thinking about that regarding giving support to Google's protobuf and it's a hell 😄

Another thing: HTTP streaming APIs tend to make use of headers. E.g. Authorization for auth, and also Content-Type (in the case of SSE, the type is text/event-stream). So those should be specifiable somewhere.

Good point. Totally agree.

Thanks for all the resources @jkarneges. Will have a deep look at these APIs and some others, so I can try to come up with a starting point.

from spec.

By "payload format definitions", you mean JSON and others?

I mean even lower level details like framing bytes, since HTTP streaming could theoretically send anything. But that's probably too crazy. :) Since there are not so many framing types in the wild, I think it should be enough for AsyncAPI to maintain a list of known types (e.g. SSE, carriage returns, or newlines) and the API could specify the type. If new framing types appear in the future, they could be added to the spec.

Inner payload always being JSON seems fine. I think all the APIs I linked use JSON.

from spec.

Oh ok! makes total sense. Yes, I also agree on following an iterative process. We can just add the most used ones first and see how it fits since we don't have much information about what people are using.

from spec.

@fmvilas @jkarneges This discussion is great! But some questions seems to have stayed on hold since.

The major difficulty, IMO, is how to document Server-Sent Events and Client-Sens Events.
Documentation events without there source, in a websocket API don't make things clear.

If this is already handled, may you please provide a minimalistic example?
If this not, may you elaborate on current workarounds / future improvements?

Thanks a lot for your effort in providing a documentation standard.

from spec.

@yamsellem my general advice is that you document the whole API from a client/consumer perspective. It means that, if an event is under "receive", the event would be sent by the server. Conversely, if they are under "send", it means the client can "send" the messages.

Please, take into account that this is an advice, not a rule.

from spec.

@fmvilas thanks for the advice and the quick answer.

May you provide a minimal sample as a convenience startup kit (even in this thread)?
It had taken me some time to bootstrap a ws asyncapi documentation; a minimal startup kit may help others, and surely will help me to compare / improve what I've already done lately.

Thanks again.

from spec.

Sorry, I forgot to give you an example. We're working on documentation right now, in the meantime, you might want to have a look at Slack Events API AsyncAPI definition.

It's a good example. You also have some examples in the v2 branch of this repo.

from spec.

official example on websocket is finally here https://github.com/asyncapi/spec/blob/master/examples/2.0.0/websocket-gemini.yml

for more details have a look at #253 (comment)

from spec.