Should we add further testing to address #32

Expected Behavior

Automatic releases should be setup for this repo, including publishing when pushing to master. Use tooling where possible.

Actual Behavior

Release tooling not setup. There's a release.sh file, but I'm not sure it it's automated.

Can we use GitHub Actions?

• https://github.com/googleapis/release-please
• https://github.com/pypa/gh-action-pypi-publish

See how the Google python client uses it:

• https://github.com/googleapis/google-api-python-client/blob/master/.github/release-please.yml

Rough steps (Github Actions):

• Create .github/workflows folder.
• Create .github/workflows/release-please.yml
  • Use releaseType: python and python in other places in the yaml
• Push to master to trigger the bot.

Contact @busunkim96 from Yoshi, @grant, go/python-chat for possible help.
Read this: https://github.com/actions/setup-python, possibly setup on a personal repo

I got an error with the pattern:

m = marshaller.NewDefaultHTTPMarshaller()

@app.route("/")
def handle():
    event = m.FromRequest(....)
    print(event)

Invoking this a second time, I got the following exception:

  File "env/lib/python3.5/site-packages/cloudevents/sdk/marshaller.py", line 71, in FromRequest
    content_type, self.__converters))
cloudevents.sdk.exceptions.UnsupportedEventConverter: Unable to identify valid event converter for content-type: 'No registered marshaller for application/json; charset=utf-8 in <generator object HTTPMarshaller.__init__.<locals>.<genexpr> at 0x7fbb573c6db0>'

I can look into this later this week.

Expected Behavior

event = from_http(request.get_data(), request.headers)
event1 = from_http(request.get_data(), request.headers)
assert event == event1

I would expect this to pass as we are passing in the same fields into both objects

Actual Behavior

Assertion error because cloudevent does not support equality overload

It feels a little heavy to install sphin alongside this package. Maybe this could be only installed in a different target (dev maybe) ?

Expected Behavior

Following crashes as CloudEvent attempts to unmarshall this as a binary event

from cloudevents.sdk.http_events import CloudEvent
import json
data = json.dumps({"id": "123", "source": "<source url>", "type": "com.sample.type", "specversion": "1.0"})
headers = {}
event = CloudEvent.from_http(data, headers)

Actual Behavior

from cloudevents.sdk.http_events import CloudEvent
import json
data = json.dumps({"id": "123", "source": "<source url>", "type": "com.sample.type", "specversion": "1.0"})
headers = {"Content-Type": "application/cloudevents+json"}
event = CloudEvent.from_http(data, headers)

We have not registered this content type with the IANA yet as seen here therefore we shouldn't expect all requests to have this header. We should instead check the http headers to determine whether event is structured or binary.

Error at

I get:

cloudevents/sdk/event/base.py", line 157, in MarshalBinary
    for key, value in props.get("extensions"):
ValueError: too many values to unpack (expected 2)

I think it should be props.get("extensions").items().

I think I'm hitting this because I'm creating one event from another and specifically passing on the extensions using SetExtensions rather adding each as an item:

            revent = (
                v02.Event().
                    SetContentType("application/json").
                    SetData(responseStr).
                    SetEventID(resp_event_id).
                    SetSource(self.model.event_source()).
                    SetEventType(self.event_type).
                    SetExtensions(event.Extensions())
            )

Expected Behavior

from cloudevents.http import to_binary, to_structured

Because these functions are within the http module, we don't need to have the suffix _http

Actual Behavior

from cloudevents.http import to_binary_http, to_structured_http

Expected Behavior

Some standardized auto formatting pre-commit such that developers can spend less time manually modifying code format and more time implementing logic.

Actual Behavior

Tox provides warnings but doesn't exactly auto-format. There were several complaints against tox, and developers having to create several additional commits to fix lint issues in the CI.

Possible tools

https://pypi.org/project/black/ for code formatting
https://pypi.org/project/isort/ for import path ordering
https://pypi.org/project/autoflake/ to remove unused imports
https://pypi.org/project/flake8/ for pep8 conformance

Expected Behavior

Following code should work using the pypi package.

pip install cloudevents
python
>>> from cloudevents.sdk.http import CloudEvent

Actual Behavior

pip install cloudevents
python
>>> from cloudevents.sdk.http import CloudEvent

Throws error because module http does not exist due to pypi being outdated.

Ultimately i'm asking if everyone thinks branch v1.0.0-dev should be considered stable now, and if not what else can be done.

At this moment ToRequest(event, "structured", lambda x: x) will return the data as io.StringIO, but ToRequest(event, "binary", lambda x: x) will return with respect to the returning datatype of a data_unmarshaller.

There's not suppose to be any kinds of differences. That's said, data must be returned as io.BytesIO in both cases.

Assuming we need the breadth of data encapsulation as seen in the events folder, we should rewrite all of the setter/getter methods into python properties using the @properties decorator

As mentioned here in the spec the data must be serialize/deserialize to/from data_base64 when the data is determined to be binary.

For serialization examples, see the go implementation, or the C# implementation
For deserialization examples, see the go implementation, the java implementation, or the C# implementation

Need discussion on how to define the namespaces for http methods (e.g. #49 (comment)).

Expected Behavior

from cloudevents.sdk import http
http.binary.to_json(event)

Actual Behavior

from cloudevents.sdk import converters
headers, body = event.to_http(converters.TypeBinary)

The above is just an example naming schema we could use.

We may want to add the following to pypi_release.yml:

with:
    skip_existing: true

to enable readme adjustments pushed to master. Unsure if this is the best way of handling this, or if the workflow should be changed to only allowing tagged commits

Using a very minimalistic event results in a non-worky cloud events:

vaikas-a01:sdk-python-master vaikas$ python3 ./app2.py
{'content-type': 'application/cloudevents+json'}
b'{"specversion": "1.0", "id": "my-id", "source": "from-vaikas", "type": "vaikas.test", "datacontenttype": "application/json", "data": "{\\"HELLO\\": \\"FROM VILLE\\"}", "extensions": {}}'

Workaround:
https://github.com/cloudevents/sdk-python/blob/master/cloudevents/sdk/event/v1.py#L31

Removing this line makes it work (but then you can't use extensions):
in v1.py

self.ce__extensions = opt.Option("extensions", dict(), False)

Repro code:

event = (
    v1.Event().
    SetContentType("application/json").
    SetEventID("my-id").
    SetSource("from-vaikas").
    SetEventType("vaikas.test").
    SetData('{"HELLO": "FROM VILLE"}')
)

m = marshaller.NewHTTPMarshaller(
    [
        structured.NewJSONHTTPCloudEventConverter()
    ]
)

#broker url

#url = 'http://localhost:8080/'

get headers and body

headers, body = m.ToRequest(event, converters.TypeStructured, lambda x: x)
print(headers)
print(body.getvalue())

#send the request
response = requests.post(url, headers=headers, data=body)

We probably want to test version bumps using test pypi in pypi-release.yml

May also want to add stable branch creation to pypi_packaging.py and RELEASING.md

Actual Behavior

from cloudevents.sdk.http_events import CloudEvent
attributes = {"source": "<source-url>", "type": "com.issue.extensions", "example-extension": "ext1"}
data = 'Hello'
event = CloudEvent(attributes, data)
headers, body = event.to_http()
print(body)

The above code will produce the following:

b'{"specversion": "1.0", "id": "fc713795-93f6-44a8-b67e-7b8bd7071e2a", "source": "<source-url>", "type": "com.issue.extensions", "time": "2020-07-16T22:41:48.222788+00:00", "extensions": {"example-extension": "ext1"}, "data": "Hello"}'

When in fact we expectthis to be outputted:

b'{"specversion": "1.0", "id": "fc713795-93f6-44a8-b67e-7b8bd7071e2a", "source": "<source-url>", "type": "com.issue.extensions", "time": "2020-07-16T22:41:48.222788+00:00", "example-extension": "ext1", "data": "Hello"}'

Current output is readable by python CloudEvents, but possibly not by other cloudevent systems unless I'm misunderstanding extensions.

Hi all, first: thanks for providing this library. It's going to be really useful to have a standard way to work with CloudEvents from Python.

I took a preliminary look at it and my first impression was that the interface provided by this library isn't very "Pythonic", meaning that it doesn't provide an interface in a way that (I believe) would be expected by most Python developers. There are some small things (like using method names like FromRequest instead of from_request, using New in class/method names) but also larger things (like the use of method chaining, setters/getters, etc.)

I'm hoping that since this project is still in alpha, there's still time to improve the interface here (and that y'all are open to suggestions). I'll try to provide some examples below of what I'm talking about but I'm happy to clarify if anything's unclear. I'm also not intimately familiar with the CloudEvents spec, so I may have some misunderstandings as well.

Parsing upstream Event from HTTP Request

The current example for turning an HTTP request into an event:

import io

from cloudevents.sdk.event import v02
from cloudevents.sdk import marshaller

m = marshaller.NewDefaultHTTPMarshaller()
event = m.FromRequest(
    v02.Event(),
    {
        "content-type": "application/cloudevents+json",
        "ce-specversion": "0.2",
        "ce-time": "2018-10-23T12:28:22.4579346Z",
        "ce-id": "96fb5f0b-001e-0108-6dfe-da6e2806f124",
        "ce-source": "<source-url>",
        "ce-type": "word.found.name",
    },
    io.BytesIO(b"this is where your CloudEvent data"), 
    lambda x: x.read()
)

• This makes the assumption that the user is only interested in supporting a single spec, but I think it's possible that either they would want to support multiple versions (and automatically detect which one is being used), or be able to upgrade to newer versions of the spec (and this module) without having to change their code.
• There's only one type of marshaller provided by the library right now, HTTPMarshaller, and if it's the default maybe we should just get it by default when creating an Event, instead of making the user initialize it every time?
• The data_unmarshaller is maybe not necessary? We're taking a bytestring, turning it into a BytesIO object to pass as data, and then using data_unmarshaller to turn it back into a bytestring. Why not just have the user do any data unmarshalling themselves before constructing the event?

Instead, consider:

from cloudevents.sdk import Event

event = Event(
    headers={
        "content-type": "application/cloudevents+json",
        "ce-specversion": "0.2",
        "ce-time": "2018-10-23T12:28:22.4579346Z",
        "ce-id": "96fb5f0b-001e-0108-6dfe-da6e2806f124",
        "ce-source": "<source-url>",
        "ce-type": "word.found.name",
    },
    body=b"this is where your CloudEvent data",
)

• This initializes a generic Event, detects which spec it matches, and gives back a corresponding Event subclass. (If the user knew which spec they were supporting, instead of constructing an Event they could construct a v02.Event and skip the event detection)
• The HTTPMarshaller is used by default. This also helps avoid issues like #12.
• The data_unmarshaller field is removed (or at the very least, optional)

Creating a minimal CloudEvent

The provided example is:

from cloudevents.sdk.event import v01

event = (
    v01.Event().
    SetContentType("application/json").
    SetData('{"name":"john"}').
    SetEventID("my-id").
    SetSource("from-galaxy-far-far-away").
    SetEventTime("tomorrow").
    SetEventType("cloudevent.greet.you")
)

• Similar to above, the user has to specify the exact event type they're expecting
• Method chaining / fluent API: while this pattern is used by some Python libraries (pandas, various ORMs) it's not typical for the majority of Python libraries and generally not recommended.

Instead, consider the following based on the same Event class proposed above:

from cloudevents.sdk import Event

event = Event(
    data=b'{"name":"john"}',
    content_type="application/json",
    event_id="my-id",
    event_time="tomorrow",
    event_type="cloudevent.greet.you",
    source="from-galaxy-far-far-away",
)

• The spec is automatically determined, but the user can also use a specific subclass if they prefer.
• The constructor uses keyword arguments, so that an Event object can be created with a single method call. If the user needs to modify the event after initialization, they can act on the attributes directly (e.g. event.event_time = "yesterday").

Getting event attributes

Currently, after creating an event, if the user wanted to get one of the fields such as the event time, the user would have to do something like:

event = ...
data = event.ce__eventTime.value

• The ce-prefix is a bit unexpected and the double-underscore is unconventional
• Having to call .value to get the value is also unexpected

Instead, consider something like:

event = ...
data = event.event_time

• No prefix or double underscore
• Access the attribute value directly

Final thoughts

I realize this is proposing a pretty big overhaul of the current interface, but I think doing so would probably go a long way to lend towards the usability and maintainability of this library, and it'd be better to do it sooner than later. I'm happy to help out here, including designing/implementing these changes and helping maintain them afterwards, if it makes sense. Thanks!

Expected Behavior

from_http(data="", headers={})

This should throw CloudEventMissingFields("Invalid data field") or similar descriptive error

Actual Behavior

First, from_http attempts to read this event as a structured event.

Because there is nothing inside data, it fails to unmarshal'. Therefore it runs json.loads("") because it expects data to be a json string with cloud event extensions.

Finally you get a json decoder error which is difficult to debug

Expected Behavior

from cloudevents.sdk.http import CloudEvent, to_binary_http
event = CloudEvent({"source": "<my-url>", "type": "issue.example"})
headers, body = to_binary_http(event)

Actual Behavior

Raises TypeError object of type NoneType has no len() when trying to marshall a None data type

Hi,

I tried the sample cloudevent_to_request script and I got this:

binary CloudEvent
content-type: application/json

ce-specversion: 0.2

ce-type: cloudevent.event.type

ce-source: <event-source

ce-id: my-id

Traceback (most recent call last):
  File "./cloudevent_to_request.py", line 74, in <module>
    run_binary(event, url)
  File "./cloudevent_to_request.py", line 32, in run_binary
    print(binary_data.getvalue())
AttributeError: 'str' object has no attribute 'getvalue'

structured format worked well

Expected Behavior

Changelog should explain why versions v02 and v01 were removed

Actual Behavior

Changelog wasn't updated in that PR

Steps to Reproduce the Problem

View the changelog

Specifications

• Platform: any

to_json and from_json in cloudevents.sdk.http.event.py raises notimplemented error

Expected Behavior

from cloudevents.http import from_http
from_http({"ce-id":"1", "ce-source":"s", "ce-type": "t"}, "123")

should raise MissingRequiredHeader hinting at specversion

Actual Behavior

raises int object has no attribute get because we don't check instance of raw_ce after json loading here on line 54

Might want an if statement checking the value of raw_ce, and error handlinlg appropriately

Expected Behavior

from cloudevents.http import from_http
headers = {
    "Ce-Source": "<my-source>",
    "Ce-Specversion": "1.0",
    "Ce-Type": "my-type",
    "Ce-Id": str(uuid4())
}
from_http("", headers)

should pass despite upper-cased attributes names.

from cloudevents.http import from_http
headers = {
    "ce-source": "<my-source>",
    "ce-specversion": "1.0",
    "ce-type": "my-type",
    "ce-id": str(uuid4())
}
from_http(None, headers)

likewise, this should return an event with data set to none.

Actual Behavior

None data causes _json_or_string to fail, Recommend addding a None if statement in that function.

Capitalized ce-specversion isn't read by from_http. Instead, from_http should make all keys lower case before passing it into future functionality.

MultiDict (https://github.com/aio-libs/multidict) is a better option to work with HTTP headers

Expected Behavior

from cloudevents.http import from_http
headers = {"ce-specversion": "1.0", "type": "me.type", "source": "<my-source>"}
from_http(headers, None)

I would expect the above code to raise cloudevents.exceptions.MissingRequiredFields("...")

Actual Behavior

Instead it does raise cloudevents.exceptions.InvalidStructuredJSON("...")

I believe it does this because line 48 we check whether headers is binary and if not we assume this is a structured request and try to read as json.

Perhaps instead of raising InvalidStructuredJSON we should set specversion to None and outside the if statement raise MissingRequiredFields('can't find specversion and couldn't decode json').

Expected Behavior

from cloudevents.sdk.converters.binary import NewBinaryHTTPCloudEventConverter
cnvtr = NewBinaryHTTPCloudEventConverter()
assert not cnvtr.can_read({"ce-specversion": "1.0"})

Actual Behavior

cnvtr.can_read({"ce-specversion": "1.0"})

will return true because it only checks for ce-specversion and not other required versions.

Expected Behavior

http_events.CloudEvent.from_http method currently only marshalls v1 events. This should probably be more dynamic in order to support multiple event versions. Perhaps we could restore specversion detection as seen in #36 or implement some other workaround.

Actual Behavior

Line 70 in https://github.com/cloudevents/sdk-python/blob/v1.0.0-dev/cloudevents/sdk/http_events.py

        event = marshaller.NewDefaultHTTPMarshaller().FromRequest(
            v1.Event(), headers, data, data_unmarshaller)

Expected Behavior

Merge v1.0.0-stable branch into master updates pypi

Actual Behavior

Must run release.sh with proper permissions.

We should setup a github action using:
https://github.com/pypa/gh-action-pypi-publish

An approximate hook in .github/workflows/main.yml could be:

- name: Publish package
  if: github.event_name == 'push' && startsWith(github.ref, 'refs/tags')
  uses: pypa/gh-action-pypi-publish@master
  with:
    user: __token__
    password: ${{ secrets.pypi_password }}

This assumes we have a github secret from pypi. We need 2 things for this automated step to work:

1. Need someone with authorization for the pypi package to set the github secret pypi_password here
2. Need a PR adding the above job to workflows and merging it either into cloudevents/master or cloudevents/v1.0.0-dev

We should probably create sample code showing both non-json data and base64 encoded data

Expected Behavior

README badges for quick status of the repo

• Build/Tests are green
• pypi link

Actual Behavior

No badges

Example

https://github.com/GoogleCloudPlatform/functions-framework-python

Functions Framework for Python

The SDK spec states we don't have to support v0.2 v0.1 versions. We should remove support and document current support here.

Hi! Is there any release soon for V1 implementation at PyPi?

Expected Behavior

event = CloudEvent(headers, data)
event.specversion

Actual Behavior

event = CloudEvent(headers, data)
event['specversion']

Steps to Reproduce the Problem

from cloudevents.sdk import converters
from cloudevents.sdk.http_events import CloudEvent
import requests


attributes = {
    "Content-Type": "application/json",
    "type": "README.sample.binary",
    "id": "binary-event",
    "source": "README",
}
data = {"message": "Hello World!"}

event = CloudEvent(attributes, data)
event['specversion']

Expected Behavior

from cloudevents.http import CloudEvent
Simple path to import stuff

Actual Behavior

from cloudevents.sdk.http import CloudEvent
sdk path doesn't actually hold anything therefore we would want to either sub-package http folder or possibly expose it top level by importing it within a cloudevents.__init__.py

Should the sdk be handling datacontenttype CloudEvent headers?

Seems like you guys have a lot going here for HTTP, I've got a need to include the AMQP bindings for processing cloud events in Python.

I'll be trying to resolve this issue myself, is that alright?

Should we unit test the sample code and if so how should we standardize this? Unsure if we should make http samples modular such that we can directly test the logic, or if tests should simply reflect at a high level what you expect CloudEvents to do.

Right now there is a poorly defined PR template and no issue template.
This can make it hard to clearly communicate issues (as seen in some issues in this repo).

We should improve this.

Example:
https://github.com/google/new-project/tree/master/.github

Expected Behavior

def from_http(
    headers: typing.Dict[str, str],
    data: typing.Union[str, bytes],
    data_unmarshaller: types.UnmarshallerType = None,
):

This order of parameters would make cloudevents.http.from_http more consistent with the other methods which either expect some header like object first, or return headers followed by data

Actual Behavior

def from_http(
    data: typing.Union[str, bytes],
    headers: typing.Dict[str, str],
    data_unmarshaller: types.UnmarshallerType = None,
)

A json CE without required attributes can still pass the FromRequest, e.g. without id

data = { "specversion": "string", "datacontenttype": "string", "data": "Unknown Type: object,string", "data_base64": "string", "time": "2020-04-24T10:58:14.315Z", "dataschema": "string", "subject": "string", "type": "string", "source": "string" } marshaller.FromRequest( v1.Event(), dict(request.headers), io.StringIO(json.dumps(data)), lambda x: x )
meanwhile,
the v1.Event() doens't enforce the value of specversion in structured mode.
the data_base64 is put into extensions instead of treating as an optional attribute.

We should add further testing to have 100% test coverage

Recommend adding the following into .tox:

    PYTESTARGS = -v -s --tb=long --cov=cloudevents --cov-report term-missing --cov-fail-under=100

Expected Behavior

from_http call on an invalid request raises CloudEvent.InvalidAttributesError

Actual Behavior

from_http call on an invalid request raises ValueError which is vague

Need to validate headers in structured CloudEvents calls and automatically toggle between Binary and Structured CloudEvents

samples/http-json-cloudevents and samples/python-requests need tests. Additionally, samples/http-image-cloudevents needs to test with a flask test server

I prefer using Travis but any CI pipeline integration with pull requests would be helpful. Is Travis the right place for this? Want me to put in a config?

Expected Behavior

from cloudevents.http import is_binary, is_structured

Actual Behavior

from cloudevents.sdk.converters import is_binary, is_structured

These methods at the moment comform to the http spec but aren't generalized to other specs. Therefore we would want to move these methods into the http module