The handler should validate the incoming ScanRequest, enqueue it for processing, and return an identifier for fetching the corresponding ScanReport.

Do not bother with private registries. We have a dedicated issue for that #12

Hello,

At OVH we plan to deploy Trivy for our Managed private reigstry service based on Harbor.

Context I would like to deploy Trivy for hundreds of customers.
Environment Harbor on Kubernetes clusters thanks to Harbor Operator.

1. Is it ok to deploy *multiple Trivy (HA) with a shared volume for /home/scanner/.cache for a single customer?
2. What kind of data are stored in /home/scanner/.cache? Is there personal data?
   Is it possible to share single Trivy instance with multiple harbor?
   Or probably safer, distinct Trivy but with a shared /home/scanner/.cache directory?
   Or even safer, distinct Trivy but with a shared /home/scanner/.cache/trivy/db/trivy.db file?

Any help appreciated.
Cheers

So far Trivy 0.1.6 is used by default. We should be able to easily upgrade to newer version.

Version should be also reflected in the /api/v1/matadata response. As of now it's hardcoded.

This is mainly to avoid GitHub rate limit errors in Harbor CI/CD environments

When the harbor installed by helm in k8s cluster, the interal url is like http://harbor-harbor-core. Then scan the imagelibrary/redis@sha256:86edeb058fb97b7e9441515a5f1bb57f1547432fedbaf3513117ba521d99ef9 by trivy, the trivy adapter will process the imags as harbor-harborr-core/library/redis@sha256:86edeb058fb97b7e9441515a5f1bb57f1547432fedbaf3513117ba521d99ef9, the trivy command will recognite the registry is index.docker.io and the scanning failed.

Trivy adapter should append port (80 for http, 443 for https) to the image before call the trivy command.

This is the issue in harbor-helm goharbor/harbor-helm#547 , harbor-helm will include port in core url as workaround to fix this issue, but we think that fix this issue in trivy adapter will be better.

Current container only support redis:// protocol.

When connecting to redis using SSL/TLS, the rediss:// protocol is used.

Would it be worthwhile adding support for the rediss:// protocol from Trivy ?

This would streamline release management and upgrades to Trivy. @knqyf263

https://redis.io/topics/sentinel

Harbor can be deployed in HA environment where Redis is also available in Sentinel mode. We should be able to support such setup.

The task here is to check what are the requirements to support sentinel mode and whether the https://github.com/gomodule/redigo lib is suitable. Then, implement the changes to the connection pool and config.

This is relevant for the integration with https://github.com/goharbor/harbor-operator and https://github.com/goharbor/harbor-cluster-operator

Trivy requires read/write access to the locations configured via SCANNER_TRIVY_CACHE_DIR and SCANNER_TRIVY_REPORTS_DIR environment variables.

The adapter service should do a sanity check on init and fail fast, if it dosn't have enough permissions.

Currently in value.yaml what I see is the below,

persistence:
  enabled: true
  storageClass: ""
  accessMode: ReadWriteOnce
  size: 5Gi

Is there a way to add the existing pvc? I tried the below, but it doesn't seem to be working.

persistence:
  enabled: false
  resourcePolicy: "keep"
  persistentVolumeClaim:
    existingClaim: "trivy-claim"
    storageClass: ""
    accessMode: ReadWriteOnce
    size: 2Gi

For Clair Harbor was able to display the upstream vulnerabilities DB update timestamp. If possible we should support the same, i.e. add the harbor.scanner-adapter/vulnerability-database-updated-at property to the metadata response.

curl -k https://harbor-scanner-trivy:8443/api/v1/metadata

{
  "scanner": {
    "name": "Trivy",
    "vendor": "Aqua Security",
    "version": "0.1.7",
  },
  "capabilities": [
    {
      "consumes_mime_types": [
        "application/vnd.oci.image.manifest.v1+json",
        "application/vnd.docker.distribution.manifest.v2+json"
      ],
      "produces_mime_types": [
        "application/vnd.scanner.adapter.vuln.report.harbor+json; version=1.0",
        "application/vnd.scanner.adapter.vuln.report.raw"
      ]
    }
  ],
  "properties": {
    "harbor.scanner-adapter/scanner-type": "os-package-vulnerability",
    "harbor.scanner-adapter/vulnerability-database-updated-at": "2019-08-13T08:16:33.345Z",
    "harbor.scanner-adapter/vulnerability-database-next-update-at": "2019-08-14T08:16:33.345Z"
  }
}

Harbor will send robot account name and token as the value of registry authorization property of ScanRequest. Those values should be set as TRIVY_USERNAME and TRIVY_PASSWORD envs.

The handler should check whether the corresponding ScanJob completed and return appropriate HTTP status codes. If the ScanJob is completed, a ScanReport should be returned based on the Accept MIME type request by the client.

We only set TTL for Finished or Failed scan jobs. However, it might happen that a scan job fails and is never retried. Therefore we'll have stale scan jobs stored in Redis.

Currently this Scanner Adapter exposes the REST API for Harbor and uses Trivy standalone (mode) to perform scans.

We could extend it and expose also Trivy server's twirp endpoint to allow Trivy clients to connect to the harbor-scanner-trivy deployment as depicted below. We could actually rename harbor-scanner-trivy to a more generic scanner-trivy deployment suitable for integrations with Harbor and other clients.

                       　　　　　　(Twirp request)
Harbor core ---→ scanner-trivy  -----------------> trivy-server 
 　　　　　　　 (HTTP REST Endpoint)                (Twirp Endpoint)
                      ↑                                 ↑
                      ↑                                 ↑
(deployed by Helm)    ↑                                 ↑
---------------------------------------------------------------------------
(prepared by a user)  ↑                                 ↑
                      ↑ (HTTP request)                  ↑ (Twirp request)
                   SDK/CLI                              ↑    
　　　　　　　for Pluggable Scanner API               trivy-client

//cc @knqyf263 @lizrice

Send request:

{"registry_url":"10.16*.*.*","registry_token":"<MY-TOKEN>","repository":"scanners/mongo","tag":"3.4-xenial","digest":"sha256:762277c0de87e5499e418c40cfe7352f7f976802b019e7ac0948d39e5f4a7413"}

The following error occurred:

2019/08/13 11:27:57 Starting harbor-scanner-trivy with config &{:8080     }
2019/08/13 11:29:46 RegistryURL: http://10.16*.*.*
2019/08/13 11:29:46 Repository: scanners/mongo
2019/08/13 11:29:46 Tag: 3.4-xenial
2019/08/13 11:29:46 Digest: sha256:762277c0de87e5499e418c40cfe7352f7f976802b019e7ac0948d39e5f4a7413
2019/08/13 11:29:46 Scan request: 46b3e41a-b096-44ad-b8c0-c014e10eb8c0
2019/08/13 11:29:46 Started scanning http://10.16*.*.*/scanners/mongo:3.4-xenial ...
2019-08-13T11:29:46.612Z        DEBUG   cache dir:  /root/.cache/trivy
2019-08-13T11:29:46.613Z        DEBUG   db path: /root/.cache/trivy/db/trivy.db
2019-08-13T11:29:46.623Z        INFO    Updating vulnerability database...
2019-08-13T11:29:46.623Z        DEBUG   git pull
2019-08-13T11:29:47.444Z        DEBUG   total updated files: 1
2019-08-13T11:29:47.475Z        FATAL   invalid image:
    github.com/knqyf263/trivy/pkg.Run
        /root/project/pkg/run.go:153
  - parsing image "http://10.16*.*.*/scanners/mongo:3.4-xenial" failed: invalid reference format
2019/08/13 11:29:47 ERROR: running trivy: exit status 1

The goal here is to:

• Release as github-actions bot
• Do not encrypt credentials in .travis.yaml if it's not absolutely necessary

I noticed that the API sever returns status code 302 in case the scan is not ready yet but I don’t think 302 is a good status code for that. For one, 302 means the requested resource resides temporarily under a different URI which isn’t true by all means (the scan definitely resides under this URI). Yet, the bigger problem is that when the server returns 302, the http client automatically redirects to the url in the Location header and in this case, it is exactly the same URL. What I observed is that the HTTP client makes 10-20 redirects (depending on what I am using) at which point it returns something like maximum redirections reached. Does the 302 status code come from some kind of specification or you chose it specifically for some reason? I think a 2xx status code can be better suited for such kind of response. Maybe 204 (no content)?

P.S. I'm talking about this part of the code.

So far I have to run docker compose, wait for all service to become ready, and only then execute the out of process component tests. We'd like to have a single make goal that does all.

Before:

docker-compose -f test/component/docker-compose.yaml up -d
make test-component
docker-compose -f test/component/docker-compose.yaml down

After:

make container && make test-component

Implement logic for processing enqueued ScanJobs, i.e. exec trivy, parse the output report, map it to Harbor vulnerability report, and finally persist.

Based on kube/harbor-scanner-trivy.yaml we should provide service and deployment templates and define configuration values to install adapter with helm install ./charts/harbor-scanner-trivy

Currently I am building a test instance of Harbor and wanting to test the use of harbor-scanner-trivy, but I don't (yet) want to set up the whole kittten caboodle as a HA cluster (using Helm). I'm able to get a simple container set up and can access the metadata (see below) but nothing else appears to be working.

I've tried various environment variables to be passed to the scanner-trivy container but there is no log output to work with.

Suggestions would be appreciated. The dump from api/v1/metadata is:

{
"scanner":{
"name":"Trivy",
"vendor":"Aqua Security",
"version":"0.4.2"
},
"capabilities":[
{
"consumes_mime_types":[
"application/vnd.oci.image.manifest.v1+json",
"application/vnd.docker.distribution.manifest.v2+json"
],
"produces_mime_types":[
"application/vnd.scanner.adapter.vuln.report.harbor+json; version=1.0"
]
}
],
"properties":{
"harbor.scanner-adapter/scanner-type":"os-package-vulnerability",
"org.label-schema.build-date":"unknown",
"org.label-schema.vcs":"https://github.com/aquasecurity/harbor-scanner-trivy",
"org.label-schema.vcs-ref":"none",
"org.label-schema.version":"dev"
}
}

With Harbor 2.0 things are much easier when it comes to deploying Trivy scanner adapter.
For online/offline installer it's just:

$ ./install.sh --with-trivy

For HA deployment with Harbor Helm chart, Trivy is enabled by default

We're using Cilium with default-closed egress, so we have to explicitly allow all egress traffic. A quick way to get started is usually by looking at logs and seeing what gets rejected. Trivy first politely complains about not being able to talk to api.github.com (via kubectl -n harbor logs pod/harbor-harbor-trivy-0 trivy and some re-formatting):

{
  "error": "running trivy wrapper: running trivy: exit status 1: 
2020-06-09T10:02:18.244Z\t\u001b[34mINFO\u001b[0m\tNeed to update DB\n
2020-06-09T10:02:18.245Z\t\u001b[34mINFO\u001b[0m\tDownloading DB...\n
2020-06-09T10:02:48.245Z\t\u001b[31mFATAL\u001b[0m\tfailed to download vulnerability DB: failed to download vulnerability DB: failed to list releases: Get https://api.github.com/repos/aquasecurity/trivy-db/releases: dial tcp 140.82.118.6:443: i/o timeout\n",
  "level": "error",
  "msg": "Scan job failed",
  "scan_job_id": "4f0dd8b163f6ea192e88d3b0",
  "time": "2020-06-09T10:02:48Z"
}

But after opening for api.github.com it still doesn't work, and the complaints are getting way less informative:

{
  "error": "running trivy wrapper: running trivy: exit status 1: 
2020-06-09T10:23:31.853Z\t\u001b[34mINFO\u001b[0m\tNeed to update DB\n
2020-06-09T10:23:31.853Z\t\u001b[34mINFO\u001b[0m\tDownloading DB...\n
2020-06-09T10:25:02.422Z\t\u001b[31mFATAL\u001b[0m\tfailed to download vulnerability DB: failed to download vulnerability DB: DB file not found\n",
  "level": "error",
  "msg": "Scan failed",
  "time": "2020-06-09T10:25:02Z"
}

From looking at package-drops in cilium (kubectl -n kube-system -it exec cilium-qbhbq -- cilium monitor -t drop) and some reverse-IP'ing, I'm stranded somewhere in S3:

xx drop (Policy denied) flow 0x285d9398 to endpoint 0, identity 25037->2: 10.217.5.243:40494 -> 52.216.160.91:443 tcp SYN
xx drop (Policy denied) flow 0x543eaa25 to endpoint 0, identity 25037->2: 10.217.5.243:40494 -> 52.216.160.91:443 tcp SYN
...

(10.217.5.243 is my trivy-container; opening http://52.216.160.91 redirects you to a S3-page)

Practically speaking, I guess downloads from api.github.com are probably re-directed to come from S3, which you obviously can't do anything about.

But getting a better error message would definitely be helpful here.

Currently harbor-scanner-trivy scans only for os vulnerabilities, even Trivy support for some application dependency checking.

1. Install trivy with skip_update
2. Copy trivy.db from offline to /data/trivy-adapter/trivy/db/trivy.db
3. Scan image
4. Got error error in vulnerability DB initialize: failed to open db: open /home/scanner/.cache/trivy/db/trivy.db: permission denied but the trivy.db permission is -rwxr-xr-x 1 root root 128770048 Apr 10 01:00 trivy.db

Hi.

I am attempting to start up the trivy container.
It doesn't seem to fail, but has many error messages.

Snippet from pod logs

ERROR: requeuer.process - EOF
ERROR: worker.fetch - EOF
ERROR: requeuer.process - EOF
{"level":"trace","msg":"Connecting to Redis","time":"2021-01-12T08:36:55Z","url":"redis://[user]:[pass]@[host]:25061"}
ERROR: worker.fetch - EOF
{"level":"trace","msg":"Connecting to Redis","time":"2021-01-12T08:36:55Z","url":"redis://[user]:[pass]@[host]:25061"}
ERROR: worker.fetch - EOF
{"level":"trace","msg":"Connecting to Redis","time":"2021-01-12T08:36:55Z","url":"redis://[user]:[pass]@[host]:25061"}
ERROR: worker.fetch - EOF
{"level":"trace","msg":"Connecting to Redis","time":"2021-01-12T08:36:55Z","url":"redis://[user]:[pass]@[host]:25061"}
ERROR: worker.fetch - EOF
{"level":"trace","msg":"Connecting to Redis","time":"2021-01-12T08:36:55Z","url":"redis://[user]:[pass]@[host]:25061"}
ERROR: worker.fetch - EOF
{"level":"trace","msg":"Connecting to Redis","time":"2021-01-12T08:36:55Z","url":"redis://[user]:[pass]@[host]:25061"}
ERROR: worker.fetch - EOF
{"level":"trace","msg":"Connecting to Redis","time":"2021-01-12T08:36:55Z","url":"redis://[user]:[pass]@[host]:25061"}
ERROR: worker.fetch - EOF
{"level":"trace","msg":"Connecting to Redis","time":"2021-01-12T08:36:55Z","url":"redis://[user]:[pass]@[host]:25061"}
ERROR: worker.fetch - EOF
{"level":"trace","msg":"Connecting to Redis","time":"2021-01-12T08:36:55Z","url":"redis://[user]:[pass]@[host]:25061"}
ERROR: worker.fetch - EOF
{"level":"trace","msg":"Connecting to Redis","time":"2021-01-12T08:36:55Z","url":"redis://[user]:[pass]@[host]:25061"}
ERROR: worker.fetch - EOF
{"level":"trace","msg":"Connecting to Redis","time":"2021-01-12T08:36:55Z","url":"redis://[user]:[pass]@[host]:25061"}
ERROR: worker.fetch - EOF
{"level":"trace","msg":"Connecting to Redis","time":"2021-01-12T08:36:55Z","url":"redis://[user]:[pass]@[host]:25061"}
ERROR: worker.fetch - EOF
{"level":"trace","msg":"Connecting to Redis","time":"2021-01-12T08:36:55Z","url":"redis://[user]:[pass]@[host]:25061"}
ERROR: worker.fetch - EOF
{"level":"trace","msg":"Connecting to Redis","time":"2021-01-12T08:36:55Z","url":"redis://[user]:[pass]@[host]:25061"}
ERROR: worker.fetch - EOF
{"level":"trace","msg":"Connecting to Redis","time":"2021-01-12T08:36:55Z","url":"redis://[user]:[pass]@[host]:25061"}
ERROR: worker.fetch - EOF
{"level":"trace","msg":"Connecting to Redis","time":"2021-01-12T08:36:55Z","url":"redis://[user]:[pass]@[host]:25061"}
ERROR: worker.fetch - EOF
{"level":"trace","msg":"Connecting to Redis","time":"2021-01-12T08:36:55Z","url":"redis://[user]:[pass]@[host]:25061"}
ERROR: worker.fetch - EOF
{"level":"trace","msg":"Connecting to Redis","time":"2021-01-12T08:36:55Z","url":"redis://[user]:[pass]@[host]:25061"}
ERROR: worker.fetch - EOF
{"level":"trace","msg":"Connecting to Redis","time":"2021-01-12T08:36:55Z","url":"redis://[user]:[pass]@[host]:25061"}
ERROR: worker.fetch - EOF
{"level":"trace","msg":"Connecting to Redis","time":"2021-01-12T08:36:55Z","url":"redis://[user]:[pass]@[host]:25061"}
ERROR: worker.fetch - EOF
{"level":"trace","msg":"Connecting to Redis","time":"2021-01-12T08:36:55Z","url":"redis://[user]:[pass]@[host]:25061"}
ERROR: worker.fetch - EOF
{"level":"trace","msg":"Connecting to Redis","time":"2021-01-12T08:36:55Z","url":"redis://[user]:[pass]@[host]:25061"}
ERROR: worker.fetch - EOF
{"level":"trace","msg":"Connecting to Redis","time":"2021-01-12T08:36:55Z","url":"redis://[user]:[pass]@[host]:25061"}
ERROR: worker.fetch - EOF
{"level":"trace","msg":"Connecting to Redis","time":"2021-01-12T08:36:55Z","url":"redis://[user]:[pass]@[host]:25061"}
ERROR: worker.fetch - EOF
{"level":"trace","msg":"Connecting to Redis","time":"2021-01-12T08:36:55Z","url":"redis://[user]:[pass]@[host]:25061"}

Debug and Trace Level Logs switched on

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: trivy
  labels:
    app: trivy
spec:
  replicas: 1
  serviceName: trivy
  selector:
    matchLabels:
      app: trivy
  template:
    metadata:
      labels:
        app: trivy
    spec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 10000
        fsGroup: 10000
      automountServiceAccountToken: false
      containers:
        - name: trivy
          image: goharbor/trivy-adapter-photon:v2.1.2
          imagePullPolicy: IfNotPresent
          securityContext:
            privileged: false
            allowPrivilegeEscalation: false
          env:
            - name: "SCANNER_LOG_LEVEL"
              value: "trace"
            - name: "SCANNER_TRIVY_CACHE_DIR"
              value: "/home/scanner/.cache/trivy"
            - name: "SCANNER_TRIVY_REPORTS_DIR"
              value: "/home/scanner/.cache/reports"
            - name: "SCANNER_TRIVY_DEBUG_MODE"
              value: "true"
            - name: "SCANNER_TRIVY_VULN_TYPE"
              value: "os,library"
            - name: "SCANNER_TRIVY_GITHUB_TOKEN"
              valueFrom:
                secretKeyRef:
                  name: trivy
                  key: gitHub.token
            - name: "SCANNER_TRIVY_SEVERITY"
              value: "UNKNOWN,LOW,MEDIUM,HIGH,CRITICAL"
            - name: "SCANNER_TRIVY_IGNORE_UNFIXED"
              value: "false"
            - name: "SCANNER_TRIVY_SKIP_UPDATE"
              value: "false"
            - name: "SCANNER_TRIVY_INSECURE"
              value: "false"
            - name: SCANNER_API_SERVER_ADDR
              value: ":8080"
            - name: "SCANNER_REDIS_URL"
              valueFrom:
                secretKeyRef:
                  name: trivy
                  key: redis.url
            - name: "SCANNER_STORE_REDIS_URL"
              valueFrom:
                secretKeyRef:
                  name: trivy
                  key: redis.url
            - name: "SCANNER_JOB_QUEUE_REDIS_URL"
              valueFrom:
                secretKeyRef:
                  name: trivy
                  key: redis.url
          ports:
            - name: api-server
              containerPort: 8080
          volumeMounts:
          - name: data
            mountPath: /home/scanner/.cache
            subPath: 
            readOnly: false
          livenessProbe:
            httpGet:
              scheme: HTTP
              path: /probe/healthy
              port: api-server
            initialDelaySeconds: 5
            periodSeconds: 10
            successThreshold: 1
            failureThreshold: 10
          readinessProbe:
            httpGet:
              scheme: HTTP
              path: /probe/ready
              port: api-server
            initialDelaySeconds: 5
            periodSeconds: 10
            successThreshold: 1
            failureThreshold: 3
          resources:
            limits:
              cpu: 1
              memory: 1Gi
            requests:
              cpu: 200m
              memory: 512Mi
  volumeClaimTemplates:
  - metadata:
      name: data
      labels:
        app: trivy
    spec:
      accessModes: ["ReadWriteOnce"]
      resources:
        requests:
          storage: "5Gi"

New version of Trivy has some cool features, including support for more linux distributions. We should have it. @knqyf263

I'd love to have linter configured that can be run locally withmake lint as well as integrated with our CI pipeline. Even though our CI is using Travis CI, the configuration should be pipeline agnostic in case we want to switch to something else.

I keep getting this from a fresh harbor installation I don't unfortunately know what its complaining about though.. is it pulling the image or ?

2020-04-09T05:15:30Z [INFO] [/pkg/scan/job.go:287]: {
  "uuid": "348a8354-7a20-11ea-ad51-7aa0e75b9f43",
  "name": "Trivy",
  "description": "The Trivy scanner adapter",
  "url": "http://harbor-harbor-trivy:8080",
  "disabled": false,
  "is_default": true,
  "health": "healthy",
  "auth": "",
  "skip_certVerify": false,
  "use_internal_addr": true,
  "adapter": "Trivy",
  "vendor": "Aqua Security",
  "version": "v0.5.3",
  "create_time": "2020-04-09T05:08:52.72567Z",
  "update_time": "2020-04-09T05:08:52.725672Z"
}
2020-04-09T05:15:30Z [INFO] [/pkg/scan/job.go:276]: scanRequest:
2020-04-09T05:15:30Z [INFO] [/pkg/scan/job.go:287]: {
  "registry": {
    "url": "http://harbor-harbor-core",
    "authorization": "[HIDDEN]"
  },
  "artifact": {
    "namespace_id": 2,
    "repository": "devops/alert",
    "tag": "",
    "digest": "sha256:7717b452552995583e2eb8563d8d40a310bbf6ec50344709c15d767b206196ac",
    "mime_type": "application/vnd.docker.distribution.manifest.v2+json"
  }
}
2020-04-09T05:15:30Z [INFO] [/pkg/scan/job.go:126]: Report mime types: [application/vnd.scanner.adapter.vuln.report.harbor+json; version=1.0]
2020-04-09T05:15:30Z [INFO] [/pkg/scan/job.go:153]: Get report for mime type: application/vnd.scanner.adapter.vuln.report.harbor+json; version=1.0
2020-04-09T05:15:32Z [INFO] [/pkg/scan/job.go:170]: Report with mime type application/vnd.scanner.adapter.vuln.report.harbor+json; version=1.0 is not ready yet, retry after 5 seconds
2020-04-09T05:15:37Z [ERROR] [/pkg/scan/job.go:235]: check scan report with mime type application/vnd.scanner.adapter.vuln.report.harbor+json; version=1.0: running trivy wrapper: running trivy: exit status 1: 2020-04-09T05:15:34.272Z	�[34mINFO�[0m	Use your github token
2020-04-09T05:15:34.272Z	�[34mINFO�[0m	Need to update DB
2020-04-09T05:15:34.272Z	�[34mINFO�[0m	Downloading DB...
2020-04-09T05:15:36.122Z	�[34mINFO�[0m	Reopening DB...
2020-04-09T05:15:36.207Z	�[31mFATAL�[0m	unable to initialize the docker scanner: unable to initialize a image struct: failed to initialize source: unable to retrieve auth token: invalid username/password: unauthorized: incorrect username or password
: general response handler: unexpected status code: 500, expected: 200

Hi @danielpacak
For issue #5
I think you can consider using something like below command to get the latest release on github:

curl -sL -o /dev/null -w %{url_effective} https://github.com/aquasecurity/harbor-scanner-trivy/releases/latest

This is actually referenced from goreleaser script

It might happen that the adapter cannot connect to redis, even if accroding to redis logs it's up and running:

component_test.go:130: adatper logs
[{"built_at":"unknown","commit":"none","level":"info","msg":"Starting harbor-scanner-trivy","time":"2020-03-25T14:03:00Z","version":"dev"}
ERROR: write_concurrency_controls_max_concurrency - dial tcp: lookup redis on 192.168.65.1:53: no such host
{"addr":":8080","level":"warning","msg":"Starting API server without TLS","time":"2020-03-25T14:03:00Z"}
ERROR: worker.fetch - dial tcp: lookup redis on 192.168.65.1:53: no such host
ERROR: heartbeat - dial tcp: lookup redis on 192.168.65.1:53: no such host
ERROR: write_known_jobs - dial tcp: lookup redis on 192.168.65.1:53: no such host
ERROR: periodic_enqueuer.should_enqueue - dial tcp: lookup redis on 192.168.65.1:53: no such host
ERROR: periodic_enqueuer.loop.enqueue - dial tcp: lookup redis on 192.168.65.1:53: no such host
ERROR: worker.fetch - dial tcp: lookup redis on 192.168.65.1:53: no such host
ERROR: worker.fetch - dial tcp: lookup redis on 192.168.65.1:53: no such host
ERROR: worker.fetch - dial tcp: lookup redis on 192.168.65.1:53: no such host
ERROR: worker.fetch - dial tcp: lookup redis on 192.168.65.1:53: no such host
ERROR: worker.fetch - dial tcp: lookup redis on 192.168.65.1:53: no such host

/cc @simar7

When deploying in Kubernetes using the Helm chart, there are some issues with the configuration of persistent volumes.

If you're using persistent volumes, the .cache directory is not writable by uid 1000. If you disable persistent volumes, it's also not writable due to it not being a separate volume and the root filesystem being read only.

It would also be nice to use the default storage class by default like most Helm charts do.

Hi,

We are seeing below duplications in the scan results. And also a discrepancy between scans done short intervals apart. These two were done less than a minute apart.
First scan:

Second scan:

scanner version 0.10.0

CC: @svisagan83

Regards.

Hi,
After updating to harbor-scanner-trivy v0.10.0, we are seeing a sharp difference in the CVE report and count. Earlier we were running on v0.9.0.
Does this scanner internally use trivy?
From the commit history i could make out 0.9.0 was using trivy 0.6.0 and now it's using 0.7.0.
Some tests i ran:
trivy 0.6.0:

trivy 0.7.0:

As you can see there is a difference in the individual counts, although vuln DB is the same.

CC: @svisagan83

So far we have a persistent keys for scan jobs, i.e. harbor.scanner.trivy:data-store:scan-job:<scan_job_id>. However it's not required by Scanner Adapter API to store scan reports forever. The clients, in particular Harbor, are supposed to cache the latest scan report.

We could use volatile key instead with configurable TTL. See https://redis.io/commands/expire

When Trivy fails for some reason, the logs accessible from the Harbor web console sometimes do not contain all the information required for troubleshooting. Typically the logs of the scanner adapter have more info. For example, when we do misconfigure GITHUB_TOKEN, the following meaningless error is shown in Harbor

2020-03-13T15:54:22Z [INFO] [/pkg/scan/job.go:126]: Report mime types: [application/vnd.scanner.adapter.vuln.report.harbor+json; version=1.0]
2020-03-13T15:54:22Z [INFO] [/pkg/scan/job.go:153]: Get report for mime type: application/vnd.scanner.adapter.vuln.report.harbor+json; version=1.0
2020-03-13T15:54:24Z [ERROR] [/pkg/scan/job.go:235]: check scan report with mime type application/vnd.scanner.adapter.vuln.report.harbor+json; version=1.0: running trivy wrapper: running trivy: exit status 1: : general response handler: unexpected status code: 500, expected: 200

Whereas in the adapter's logs we have:

{"exit_code":1,"image_ref":"core:8080/library/alpine@sha256:acd3ca9941a85e8ed16515bfc5328e4e2f8c128caa72959a58a127b7801ee01f","level":"error","msg":"Running trivy failed","std_err":"","std_out":"2020-03-13T15:56:36.134Z\t\u001b[34mINFO\u001b[0m\tUse your github token\n2020-03-13T15:56:36.134Z\t\u001b[34mINFO\u001b[0m\tNeed to update DB\n2020-03-13T15:56:36.134Z\t\u001b[34mINFO\u001b[0m\tDownloading DB...\n2020-03-13T15:56:36.392Z\t\u001b[31mFATAL\u001b[0m\tfailed to download vulnerability DB: failed to download vulnerability DB: failed to list releases: GET https://api.github.com/repos/aquasecurity/trivy-db/releases: 401 Bad credentials []\n","time":"2020-03-13T15:56:36Z"}

I just deployed Trivy with my harbor environment and everything seems to be working fine in the portal, except for one thing.

I'm trying to scan a local image using the trivy server that was deployed in my harbor server.

# trivy client --remote https://harbor-trivy.mydomain:8443 image:tag

2020-02-19T18:00:00.291Z	FATAL	error in image scan: failed to scan the image:
failed to detect vulnerabilities: failed to detect vulnerabilities via RPC:
twirp error bad_route: Error from intermediary with HTTP status code 404 "Not Found"

The address harbor-trivy.mydomain is a cname I created that points to the load balancer helm created.

I installed with the following params:

helm install harbor-scanner-trivy ./helm/harbor-scanner-trivy \
               --namespace harbor-test \
               --set service.port=8443 \
               --set scanner.api.tlsEnabled=true \
               --set scanner.api.tlsCertificate="`cat tls.crt`" \
               --set scanner.api.tlsKey="`cat tls.key`"

Can you pls tell me if I am doing something wrong or if this function (trivy client --remote) is not available for this kind of deployment?

It could be nice if we can customized some parameters of the trivy scan :

1. severities of vulnerabilities to be displayed
2. ability to display only fixed vulnerabilities

My use case is to provide

• a "light" scanner configure with CRITICAL and HIGH severities and only with fixed vulns
• a "hard" scanner which reports all vulns

As scanner wraps trivy under the hood, we could simply use trivy env variables to do this :

• TRIVY_SEVERITY to set severities
• TRIVY_IGNORE_UNFIXED to display fixed vulnerabilities

Or introducing new env variables for scanner like for example ?

• SCANNER_TRIVY_SEVERITY
• SCANNER_TRIVY_IGNORE_UNFIXED

What do you think about this proposition ?

Currently we return a list of URLs, which are displayed in Harbor in a tool tip. This is a bit confusing. We could return the primary URL and navigate users to AVD, and from there they could find more details.

IIRC If we return only 1 URL, Harbor will display it without the tool tip.

When we enable policy check in Harbor Portal Trivy cannot exchange robot account credentials to JWT access token.

We don't have much details in the logs as well

{"exit_code":1,"image_ref":"core.harbor.domain/library/jenkins/jenkins@sha256:c3b6fa9216336022067be94adb9d4696459903bf3e67e01a8d0009994f5ab78d","level":"error","msg":"Running trivy failed","std_err":"","std_out":"","time":"2019-11-07T09:38:29Z"}
{"error":"running trivy wrapper: running trivy: exit status 1: ","level":"error","msg":"Scan failed","time":"2019-11-07T09:38:29Z"}
{"error":"running trivy wrapper: running trivy: exit status 1: ","level":"error","msg":"Scan job failed","scan_job_id":"b53336ec9e8028250ee78d75","time":"2019-11-07T09:38:31Z"}