kyverno / reports-server Goto Github PK

Alternate reports storage in Kubernetes outside etcd

License: Apache License 2.0

Makefile 7.17% Go 90.52% Smarty 2.09% Dockerfile 0.22%

reports-server's Introduction

Reports Server

Reports server provides a scalable solution for storing policy reports and cluster policy reports. It moves reports out of etcd and stores them in a PostgreSQL database instance.

Why reports server?

It is desirable to move reports out of etcd for several reasons:

Why leave etcd:

The etcd database currently has a maximum size of 8GB. Reports tend to be relatively large objects, but even with very small reports, the etcd capacity can be easily reached in larger clusters with many report producers.
Under heavy report activity (e.g. cluster churn, scanning, analytical processes, etc.), the volume of data being written and retrieved by etcd requires the API server to buffer large amounts of data. This compounds existing cluster issues and can cascade into complete API unavailability.
CAP guarantees are not required for reports, which, at present, are understood to be ephemeral data that will be re-created if deleted.
Philosophically, report data is analytical in nature and should not be stored in the transactional database.

Benefits of reports server:

Alleviation of the etcd + API server load and capacity limitations.
Common report consumer workflows can be more efficient.
- Report consumers are often analytical and/or operate on aggregate data. The API is not designed to efficiently handle, for example, a query for all reports where containing a vulnerability with a CVSS severity of 8.0 or above. To perform such a query, a report consumer must retrieve and parse all of the reports. Retrieving a large volume of reports, especially with multiple simultaneous consumers, leads to the performance issues described previously.
- With reports stored in a relational database, report consumers could instead query the underlying database directly, using more robust query syntax.
- This would improve the implementation of, or even replace the need for, certain exporters, and enable new reporting use cases.

Installation

See INSTALL.md

reports-server's People

Contributors

Stargazers

Watchers

Forkers

realshuting giantswarm mikedougherty anushkamittal2001 nirmata knechtionscoding kalyann567 s0ko1ex

reports-server's Issues

Unable to install the helm chart

helm repo list
reports-server          https://kyverno.github.io/reports-server/

helm repo update
...Successfully got an update from the "reports-server" chart repository

helm search repo reports-server
No results found

conformance testing with kyverno and chainsaw

We need to add tests for:

All policies in kyverno/kyverno and kyverno/policies with reports-server installed.
Checks for all upgrade and downgrade scenarios.

Add post-install hook to install apiservices in migration

add migration step before starting reports-server

DB Conflict: create clusterpolicyreport: pq: duplicate key value violates unique constraint "clusterpolicyreports_pkey"

I am using kyverno-reports-server with external Postgres Cluster using CloudnativePG Operator v16.3
Once restarting the kyverno-reports-server pod, i get this error:

I0807 07:09:42.387073 1 cpolr.go:88] creating entry for key:00269f9b-6f65-424b-8165-e58f2e69dcb4 E0807 07:09:42.389862 1 cpolr.go:98] "failed to crate cpolr" err="pq: duplicate key value violates unique constraint \"clusterpolicyreports_pkey\"" E0807 07:09:42.389896 1 config.go:59] create clusterpolicyreport: pq: duplicate key value violates unique constraint "clusterpolicyreports_pkey" Error: create clusterpolicyreport: pq: duplicate key value violates unique constraint "clusterpolicyreports_pkey" . . . panic: create clusterpolicyreport: pq: duplicate key value violates unique constraint "clusterpolicyreports_pkey" goroutine 1 [running]: main.main() github.com/kyverno/reports-server/main.go:20 +0x8b

it is trying to recreate the reports....but once the name is there, it should update the values instead of insert

Finalise helm docs template

Finalise helm docs template.

Installation via yaml manifests fails

I followed the installation instruction here via yaml manifests

I can see that pod is crashing:

➜ kubectl get pods
NAME                             READY   STATUS             RESTARTS      AGE
reports-server-d5ccb6f79-crhfn   0/1     CrashLoopBackOff   3 (16s ago)   69s
reports-server-postgresql-0      1/1     Running            0             69s

and when I checked the logs:

➜ kubectl logs reports-server-d5ccb6f79-crhfn
Error: unknown flag: --dbsslmode
Usage:
   [flags]

Policy server flags:

      --dbhost string       Host url of postgres instance (default "reportsdb.kyverno")
      --dbname string       Name of the database to store policy reports in (default "reportsdb")
      --dbpassword string   Password to login into postgres (default "password")
      --dbport int          Port of the postgres instance (default 5432)
      --dbuser string       Username to login into postgres (default "postgres")
....
....
....
Features flags:


panic: unknown flag: --dbsslmode

goroutine 1 [running]:
main.main()
	github.com/kyverno/reports-server/main.go:20 +0x88

Fix CT lint job

When repo is made public, finalise CI

Harden workflows permissions.

Add CodeQL.

Enable gh-pages.

Release chart and publish in artifact hub.

Enable SLSA.

[Feature] Add support for high availability deployments

Explaination

Since, the main purpose of reports-server is to remove load from api server and etcd. Having multiple replicas of reports-server will help in processing the requests faster.

Solution

Add support for multiple replicas of reports-server

[Help] How to validate the all reports are stoting in postgreSQL report-server DB

Hi Team,

I have installed report-server on kubernetes cluster and edited the apiservice objects to point report-server svc, it is working fine.

Now I am looking for information that how do we validate Kyverno reports are stored on the report-server DB which is deployed as part of chart.

Also please help me to know how to generate the kyverno reports, because in my current cluster count is showing 0.

np@shell:reports-server ()$ kubectl get --raw=/metrics | grep apiserver_storage_objects | grep kyverno
apiserver_storage_objects{resource="admissionreports.kyverno.io"} 0
apiserver_storage_objects{resource="backgroundscanreports.kyverno.io"} 0
apiserver_storage_objects{resource="cleanuppolicies.kyverno.io"} 0
apiserver_storage_objects{resource="clusteradmissionreports.kyverno.io"} 0
apiserver_storage_objects{resource="clusterbackgroundscanreports.kyverno.io"} 0
apiserver_storage_objects{resource="clustercleanuppolicies.kyverno.io"} 0
apiserver_storage_objects{resource="clusterephemeralreports.reports.kyverno.io"} 0
apiserver_storage_objects{resource="clusterpolicies.kyverno.io"} 0
apiserver_storage_objects{resource="ephemeralreports.reports.kyverno.io"} 0
apiserver_storage_objects{resource="globalcontextentries.kyverno.io"} 0
apiserver_storage_objects{resource="policies.kyverno.io"} 0
apiserver_storage_objects{resource="policyexceptions.kyverno.io"} 0
apiserver_storage_objects{resource="updaterequests.kyverno.io"} 0

Thanks in advance !

setup HA mode in postgres

Helm install fails

here's the steps i followed:

➜ helm repo add reports-server https://kyverno.github.io/reports-server
"reports-server" already exists with the same configuration, skipping

➜ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "kyverno" chart repository
...Successfully got an update from the "concourse" chart repository
...Successfully got an update from the "reports-server" chart repository
Update Complete. ⎈Happy Helming!⎈

➜ helm install reports-server -n reports-server reports-server/reports-server --create-namespace
Error: INSTALLATION FAILED: chart "reports-server" matching  not found in reports-server index. (try 'helm repo update'): no chart version found for reports-server-