Panic currently for storage when it's unavailable. Needs better handling and remove the panic.

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Awaiting Schedule

These updates are awaiting their schedule. Click on a checkbox to get an update now.

• Renovate: Update github.com/sapcc (github.com/sapcc/go-api-declarations, github.com/sapcc/go-bits)
• Renovate: Update module github.com/gophercloud/gophercloud to v1.14.0

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• Renovate: Update module github.com/gophercloud/gophercloud to v2

Vulnerabilities

Renovate has not found any CVEs on osv.dev.

Detected dependencies

• Check this box to trigger a request for Renovate to run again on this repository

In order to support the growth in audit logs, we have to migrate existing Hermes ES from single instance to cluster.

In the cluster, we want to enable sharding and increase the data retention to at least 6 months better 1 year if possible (as requested by Team Core API and Services).

There must be a possibility to search for events inside every project using a system scope. This feature was removed in 2269120, but it would be great to have it back with proper policy rules logic.
cc @rajivmucheli

As for now most of the resource types contain only ID (e.g. server id, security group id, etc), but there are no names available.

How expensive is it to resolve names on the hermes server side or does it make sense to do this on the client side?

GetEvent seems to fail in a variety of ways due to the mapping used.

Need to see if there is some mapping setup that can avoid this issue. using .raw was the initially working attempt, tried using .keyword instead.

This ultimately involves UUIDs and their tokenization

We want to enable the events api to add additional details if requested. This is partially fleshed out and in the code base, but not functioning in practice. Need to find the bug and correct.

Document metrics in Ops Guide after instrumentation is complete.

Created a branch for implementing prom on metrics endpoint. Still working on it, but first pieces are together with some small reorganization.

Support specifying a target project/domain in all API operations, so that a cloud-admin view, accessing multiple projects with the same Keystone token, is possible.

Bug submitted that userid filtering is not working on partial filter. Need to test and solve.

The current max limit for ES Queries configured for Hermes and the Elasticsearch instances are 10k (the default).

Increase the number will increase memory usage in the Elasticsearch, but many projects/domains contain over 10k events.

The long term solution will be to change the API. In the short term we can attempt increasing that limit to 20k for example (or more), and validate the stability.

Our current situation of Dummy testing does not catch most bugs, as there isn't much happening but testing if the mock returns properly.

Move over to GoMock and unit test each individual query param. Use Maia as a guide, which has already implemented GoMock.

Several values within the audit events are hirearchical. Ex. update/add/interface

For getting Unique attributes, there should be the option to add a max_depth parameter of "1-3" to return all unique events that contain update/

This will also require that listing events will take globs, so that if you query unique attributes for update, you can then send action=update* to return the events that match the update query.

New parameter, new unique accepting parameter, new elastic call for handling that.
New events list handling, and elastic call for the glob.

current - 2017-07-20T13:43:38.368933+0000
desired - 2017-07-20T13:43:38.368933+00:00

check what the value is in kibana, if it's fine in kiabana, problem is likely in json translation to type.

if it's not fine in kibana, then either it needs changed on openstack middleware side, or just replaced as the return value for the api.

Tests exist, but they are not found when running make

▶ running tests…
? github.com/sapcc/hermes [no test files]

Need to correct. Ideally that will work towards automatic coverage checks working as well.

Olivere will no longer work on the golang client, to upgrade to ES 8 we need to upgrade the client.

There is also a question of moving to opensearch.

Evaluate both golang clients, and determine what includes the features we use, and makes the most sense.

Improve all es error details with full errors es client struct. Example added to attributes storage call.

"attachments": [
{
"content": "["ce0179d6-8a94-4f7c-91c2-f3038e2acbd0"]",
"contentType": "network/security-group",
"name": "security_groups"
}

When the target project/domain has child-projects, the API calls cascade to the child-projects, so that the audit-viewer of the main project does not have to traverse all the child-projects.

When called against the main project, the API act like the child-projects had been merged with the main project. That means that the /events call returns all the events of child-projects. Likewise the /attributes API returns existing unique values in not only the main but also the child projects.

Fix all Golint errors, a lot have built up over time. All of them need to be corrected.

Working on it.

Works for getting attributes, but not for listing events. troubleshooting.

Narrow use case, but reported that if you check for the attributes for a project with no events, it returns "Could not load attributes"

May not even be a Hermes issue, may be with the dashboard.

If the project resource is created/modified/deleted from the system scope (admin), then this event won't be visible within the project scope.

The Gorilla project has been archived, and is no longer under active maintenance. You can read more here: https://github.com/gorilla#gorilla-toolkit

I'm having a look at the event list and a single event detail of the same event id
and I notice: the observer.name is different.

In the list it takes the initiator name

Which may be a copy&paste error in df270d7?

As a user I'd like to get events for the network/loadbalancer OR network/loadbalancer/listener target types. At the moment there is only AND range combination possible for dates (https://github.com/sapcc/hermes/blob/cf15906738a0c8e2c9fb32b1ecf982c82f4d8a0d/pkg/api/events.go#L100..L139)

Added the param so far, and hooked it up. There are a few different levels of size, and I don't think it's working exactly right.

Will continue to futz with tomorrow.

I believe this should be possible, I will have to check, but it would add value to customers and internal stakeholders to implement a search across the entire event to do things like search within attachments.

2019/11/20 13:34:14 [DEBUG] OpenStack Request URL: GET https://hermes/v1/attributes/initiator_name
2019/11/20 13:34:14 [DEBUG] OpenStack Request Headers:
Accept: application/json
User-Agent: gophercloud/2.0.0
X-Auth-Token: ***
2019/11/20 13:34:15 [DEBUG] OpenStack Response Code: 502
2019/11/20 13:34:15 [DEBUG] OpenStack Response Headers:
Content-Length: 173
Content-Type: text/html
Date: Wed, 20 Nov 2019 12:34:15 GMT
Server: nginx/1.15.3
Strict-Transport-Security: max-age=15724800; includeSubDomains
2019/11/20 13:34:15 [DEBUG] Not logging because OpenStack response body isn't JSON
Error: Failed to list attributes: Expected HTTP response code [200 204 300] when accessing [GET https://hermes/v1/attributes/initiator_name], but got 502 instead
<html>
<head><title>502 Bad Gateway</title></head>
<body bgcolor="white">
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.15.3</center>
</body>
</html>

https://gist.github.com/kayrus/d345f91280f06b1a5966c46cbf1d29cf

/cc @DRichardt

Attributes returning a max of 10 values, shouldn't be restricted.

Considering an amount of auth methods it would be useful to log keystone token details along with the event details.
It would be useful to list an auth method, token roles, when the token was issued, when it will expire, audit IDs, etc.

Would be nice to have prom metrics on the total number of events, and maybe the number of different types of events.

Will need to flesh this out more, and of what would be of value.

Seems like the /audit API is not implemented. We should not expose API that is defunct.

Examples:

$ hermesctl attributes action -l5
update
update/add/security-group
create
delete
create/role_assignment

$ hermesctl attributes action --max-depth 1 -l 3
update
create

$ hermesctl attributes action  --max-depth 2 -l 3
update
update/add
create

$ hermesctl attributes action --max-depth 1 -l 10
update
create
delete
read

$ hermesctl attributes action --max-depth 1
update
create
delete
read
stop
allow
disable
deny
restore
start

I assume that the limit should be applied after the max-depth filter.

We have several go apis that utilize Hermes, and well as the audit middleware for OpenStack Services.

This needs to be documented.

Seems to work on time, but not other query params. This is likely due to tokenization and the use of .raw fields at various points. The sort fields need to consistently use the same name. Different areas add it in different ways, so standardize it, and improve.

ERROR: api.ListEvents: error elastic: Error 400 (Bad Request): all shards failed [type=search_phase_execution_exception]

Adding the new full text search feature, it will have a much higher compute/memory impact. We will need to implement rate limiting to help handle these issues in the future.