replicatedhq / kots-lint Goto Github PK

View Code? Open in Web Editor NEW

6.0 17.0 3.0 4.76 MB

Lint a KOTS application before deploying it

Home Page: https://kots.io

License: Apache License 2.0

Dockerfile 0.27% Makefile 0.39% Go 87.00% Open Policy Agent 12.34%

kots kubeval lint best-practices

kots-lint's Introduction

kots-lint is a service used for linting Replicated KOTS yaml files.

Linting with this service includes:

Detecting YAML syntax errors
Validating with Kubeval
Utilizing OPA to lint for best practices and some special errors and warnings

Using the production API

$ tar cvf - path/to/folder | curl -XPOST --data-binary @- https://lint.replicated.com/v1/lint

To lint our example

$ tar cvf - example/files-to-lint | curl -XPOST --data-binary @- https://lint.replicated.com/v1/lint

Development

Development for the applications in this project is done through Okteto.

Debugging Rego files

Rego supports the use of print() function. To enable printing to stderr, add EnablePrintStatements and PrintHook to initialization code. For example:

	buildersQuery, err := rego.New(
		rego.Query("data.kots.spec.builders.lint"),
		rego.Module("builders-opa.rego", string(buildersRegoContent)),

		// The lines below allow using print() in the rego code to print to stderr
		rego.EnablePrintStatements(true),
		rego.PrintHook(topdown.NewPrintHook(os.Stderr)),
	).PrepareForEval(ctx)

Setup

Install the Okteto CLI (brew install okteto)
Setup Okteto CLI (okteto context create https://replicated.okteto.dev)
Setup Okteto context in kubectl (okteto context update-kubeconfig)
Deploy your current branch. (from the Vandoor root directory: okteto pipeline deploy)

The project can also be run with Skaffold for local development and testing.

$ skaffold dev

Once skaffold runs successfully, the service can be reached at http://localhost:30082/v1/lint

Run tests

Tests can be run manually with

$ make test

Updating specs

Generated specs can be copied from the source project directly. File names should match, but since projects will change independently, care should be taken when copying.

Troubleshoot

cp <troubleshoot root>/schemas/*.json <kots-lint root>/kubernetes_json_schema/schema/v1.23.6-standalone-strict/

KOTS

cp <kots root>/kotskinds/schemas/*.json <kots-lint root>/kubernetes_json_schema/schema/v1.23.6-standalone-strict/

kots-lint's People

Contributors

Stargazers

Watchers

Forkers

mikaelsmith marianbendik alexrogalskiy

kots-lint's Issues

"may-contain-secrets" warning has many false-positives

It seems the check for may-contain-secrets frequently surfaces false positives, adding to a lot of noise in linter output.

I'd like to suggest that we disable this check until the false positive rate is lower.

Example

For example, there are no irresponsible secrets in this bundle of manifests, but the linter comes up with several warnings:

https://github.com/replicatedhq/kotsapps/tree/master/postgres-snapshots/manifests

may-contain-secrets    info    manifests/pg-consumer.yaml                                                                          It looks like there might be secrets in this file    
may-contain-secrets    info    manifests/pg-snapshot.yaml             29      It looks like there might be secrets in this file    
may-contain-secrets    info    manifests/postgres-secret.yaml         24      It looks like there might be secrets in this file    
may-contain-secrets    info    manifests/postgres-secret.yaml         18      It looks like there might be secrets in this file

For example, this file triggers the warning, but is clearly safe

apiVersion: v1
kind: Secret
metadata:
  name: postgres
data:
  DB_HOST: >-
    {{repl if ConfigOptionEquals "postgres_type" "embedded_postgres" }}
      {{repl Base64Encode "postgres" }}
    {{repl else}}
      {{repl ConfigOption "external_postgres_host" | Base64Encode }}
    {{repl end}}
  DB_PORT: >-
    {{repl if ConfigOptionEquals "postgres_type" "embedded_postgres" }}
      {{repl Base64Encode "5432" }}
    {{repl else}}
      {{repl ConfigOption "external_postgres_port" | Base64Encode }}
    {{repl end}}
  DB_USER: >-
    {{repl if ConfigOptionEquals "postgres_type" "embedded_postgres" }}
      {{repl Base64Encode "postgres" }}
    {{repl else}}
      {{repl ConfigOption "external_postgres_user" | Base64Encode }}
    {{repl end}}
  DB_PASSWORD: >-
    {{repl if ConfigOptionEquals "postgres_type" "embedded_postgres" }}
      {{repl ConfigOption "embedded_postgres_password" | Base64Encode }}
    {{repl else}}
      {{repl ConfigOption "external_postgres_password" | Base64Encode }}
    {{repl end}}
  DB_NAME: >-
    {{repl if ConfigOptionEquals "postgres_type" "embedded_postgres" }}
      {{repl Base64Encode "postgres" }}
    {{repl else}}
      {{repl ConfigOption "external_postgres_db" | Base64Encode }}
    {{repl end}}
  SCHEMAHERO_URI: >-
    {{repl if ConfigOptionEquals "postgres_type" "embedded_postgres" }}
      {{repl Base64Encode (printf "postgresql://postgres:%s@postgres:5432/postgres?connect_timeout=10&sslmode=disable" (ConfigOption "embedded_postgres_password")) }}
    {{repl else}}
      {{repl Base64Encode (printf "postgresql://%s:%s@%s:%s/%s?connect_timeout=10&sslmode=disable" (ConfigOption "external_postgres_user") (ConfigOption "external_postgres_password") (ConfigOption "external_postgres_password") (ConfigOption "external_postgres_host") (ConfigOption "external_postgres_port") (ConfigOption "external_postgres_db")) }}
    {{repl end}}

Linter fails to catch invalid template function

value: "{{repl RandomssString 32}}"

Linter incorrectly shows "Missing `kind` field" and "Missing `apiVersion` field" line numbers for valid multidoc yaml with empty docs.

We've recently switched on the new linter. This has resulted in some situations where previously valid yaml is showing incorrect errors.

To Repro:

Create a new application, using the default yamls.
Combine all deployment and service yamls together. Verify no errors.
Append the following to the multidoc yaml:

---
# Random Comment

---
#  Random other Comment

Try to save the release.

Expected: No errors are returned. Or if an error is returned, it points to the line numbers for the empty file.
Actual: Errors are returned. Error shows a line number for valid yaml, citing that "kind" and "apiVersion" is missing. This is misleading. It could be argued that it's invalid to have an empty file, but given that it was previously passing validating, I'd argue that we should continue to treat it as valid. That is continue to accept empty docs in a multidoc stream.

Missing lints for invalid DNS names

This may be a problem downstream, but I noticed a couple of deployment failures on some YAML files that passed linting successfully:

Invalid name on Secret. YAML that passed linting:

apiVersion: v1
kind: Secret
metadata:
  name: object_store

value should be object-store, not object_store

Error from attempted deploy:

Error from server (Invalid): error when creating "/Users/iristyle/source/holodeck/holodeck-manifests/output/cd4pe/overlays/midstream": Secret "object_store" is invalid: metadata.name: Invalid value: "object_store": a DNS-1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*')

Related, invalid reference to this name from a Deployment (only relevant YAML shown):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: cd4pe
  namespace: 'cd4pe'
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: cd4pe
  template:
    metadata:
      labels:
        app.kubernetes.io/name: cd4pe
    spec:
      initContainers:
      - name: create-bucket
        image: minio/mc
        env:
        - name: MINIO_ACCESS_KEY
          valueFrom:
            secretKeyRef:
              name: object_store
              key: accessKey

value should be object-store, not object_store

Error message

Resource: "apps/v1, Resource=deployments", GroupVersionKind: "apps/v1, Kind=Deployment"
Name: "cd4pe", Namespace: "cd4pe"
for: "/Users/iristyle/source/holodeck/holodeck-manifests/output/cd4pe/overlays/midstream": Deployment.apps "cd4pe" is invalid: [spec.template.spec.containers[0].env[13].valueFrom.secretKeyRef.name: Invalid value: "object_store": a DNS-1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*'),

Same basic problem with a preflight check

apiVersion: troubleshoot.replicated.com/v1beta1
kind: Preflight
metadata:
  name: cd4pe-preflight
spec:
  collectors:
  - run:
      name: object_store
      collectorName: object_store_credentials_valid
      image: minio/mc

object_store_credentials_valid should be object-store-credentials-valid

having duplicate keys for any object breaks the linter

for example

apiVersion: apps/v1
kind: Deployment
metadata:
  name: example-nginx
  labels:
    app: example
    component: nginx
spec:
  replicas: 1
  replicas: 1

Update Kubernetes schemas to 1.18.0, and drop 1.15.0

We should add v1.18.x schemas to kubernetes-json-schema and drop v1.15.7-standalone-strict schemas

Feature Request: Url parameter to force processing of all files

Add url parameter to force processing of all uploaded files before returning results, regardless of errors. This would be very helpful for CI use cases to avoid ugly repetition of builds to work through multiple files with issues. Currently, the workaround requires uploading each file individually, and compiling the results for each, which takes an unacceptable amount of time to process.