EventTime Format doesn't match spec about hermes HOT 14 CLOSED

Afaik, the ISO 8601 standard allows both +/-00:00 and +/-0000 So I wouldn't say this is a bug.

from hermes.

The eventtime in Kibana is the current one, so the problem is before hermes. I can check for it and return it, but looping over all the events to grab the field and check it seems silly if we can just change how it's created. Going to see if I can sort out what it looks like in the middleware step.

from hermes.

It's in the Elasticsearch mapping. there are several different date formats, the mapping file that connects them is in the logstash. Trying to sort out if there's one that matches what we want, or if we'll need a custom mapping for it.

from hermes.

Okay, the logstash date filter plugin for elasticsearch is using the Z timezone offset structure. The request is for it to be the ZZ structure. So i've sort out the what, now to sort out the how.

Z | time zone offset or identity Z Timezone offset structured as HHmm (hour and minutes offset from Zulu/UTC). Example: -0700. ZZ Timezone offset structured as HH:mm (colon in between hour and minute offsets). Example: -07:00.

A side argument is, we should probably hold the mappings file and other config specific stuff in this repo?.. seems logical. at least an example.

from hermes.

And after a long goose chase, I finally found it. Man there are a lot of time modules.

The actual problem is the time generated in the pycadf package

https://github.com/openstack/pycadf/blob/master/pycadf/timestamp.py

test that validates -

#!/usr/bin/python

import datetime
import pytz

TIME_FORMAT = "%Y-%m-%dT%H:%M:%S.%f%z"

utc_datetime = pytz.utc.localize(datetime.datetime.utcnow())
print utc_datetime.strftime(TIME_FORMAT)

which prints

2017-12-16T00:28:08.615425+0000

The utc_datetime.strftime method does not have an option current to give timezone offset in the dd:dd format.

from hermes.

@jobrs given that the issue is in pycadf, what do you think is the best approach?

from hermes.

pattern = "(+\d\d)(\d\d)"
string = re.sub(pattern, r"\1:\2", string)

does the trick

from hermes.

@notque thanks for pinpointing the issue. I file bug https://bugs.launchpad.net/pycadf/+bug/1738737 on pycadf and will add a workaround into openstack-audit-middleware, tracked via sapcc/openstack-audit-middleware#19

therefore this ticket can be closed

from hermes.

still, we should use the right data type for the timestamp, so that possible further/future format issues are revealed

from hermes.

There is a test in the API for valid time formats that doesn't include the future format. Need to add that and look for other potential issues like that.

Edit - It doesn't evaluate the Timezone offset at all in the queryparam, so it works fine. Fair enough

from hermes.

@edda CADF is more specific on this and so is the Golang marshaller for JSON. It's no valid timestamp value in JSON. Readability is also limited IMHO ...

@notque If the timezone offset is ignored that would mean that only UTC queries will work properly, right?

from hermes.

@jobrs I see you've switched to Time.time for eventTime and reverted it. I had a branch I was working on trying to sort out issues. Where are you at with this, is it still a concern or not?

from hermes.

@notque yeah, I gave up. ISO8601 is not built-in, so to me effort vs. benefit are no longer balanced.

from hermes.

Fair enough, let's call this issue closed and move on.

from hermes.