google / osv.dev Goto Github PK

To support package ecosystems.

OSV's infra should:

• Support enumerating package versions to provide fast version querying.
• Deducing commit ranges where possible based on repo version tags to support commit queries as well, if there was a reliable way to get the repo URL.

When we have a large number of versions in the Bug, we may exceed the entity index limit.

spec: http://maven.apache.org/pom.html#Dependency_Version_Requirement_Specification

I don't believe it is currently possible to re-evaluate an existing entry. It would be useful to re-run analysis over existing entries when we add enhancements to the code to better classify things, etc. It would also be useful for cases where there is no fixed version at the time an advisory is published, but a fixed version is later released. Currently I just have some python scripts I run manually to try and check some of this and manually update when needed.

I thought it could be useful to add GHSA data as an input source for the pypi vulnfeed. We can combine it with the matching CVE source data and then extract the best information from the combination. That could help prevent bad matches like the one that happened with pypa/advisory-database#34 as we could have evaluated the GHSA record and realised that it was not a match for that cve and pypi package combination. Also, there are many instances where there is a GHSA record, but no CVE is ever requested for it, but we may still want a PYSEC and advisory created.

We'd have to make some decisions around which source to prefer when both exist. I suspect that the GHSA record will usually be more trustworthy for identifying the affected pypi package since it will be created by the project itself, and probably the version range info is more reliable as well? We could aggregate all of the reference links from both sources. For the details field, I think the GHSA ones tend to be more verbose and contain more markdown syntax than the nvd ones, so unsure which should be preferred there.

@oliverchang , any thoughts on this? And are you already collecting all of the GHSA JSON entries somewhere where we could just download a zip of them or would we need something new for that? I know there is a process in aquasecurity/vuln-list-update which creates the JSON entries at aquasecurity/vuln-list which is used by trivy.

some query responses may be quite large, so we should return a paginated view.

I was wondering if you are planning to also release a dump of the data, perhaps storing each vulnerability object in a folder that can be accessed (e.g. GitHub) or downloaded. I think it would be a great addition to have.

I am trying to get the OSV for gopkg.in/yaml.v2 and here is curl command
curl -X POST -d '{"version":"2.4.0", "package": {"name": "gopkg.in/yaml.v2", "ecosystem": "Go"}}' "https://api.osv.dev/v1/query"

Here is the result of the call.

But as per result, it has been fixed in 2.2.8 and in 2.2.3 , so 2.4.0 should return clean. It isn't. This is correct results I am seeing in https://deps.dev/go/gopkg.in%2Fyaml.v2/v2.4.0

What am I doing wrong?

Thanks

{
  "vulns": [
    {
      "id": "GO-2020-0036",
      "package": {
        "name": "gopkg.in/yaml.v2",
        "ecosystem": "Go"
      },
      "details": "Due to unbounded aliasing, a crafted YAML file can cause consumption\nof significant system resources. If parsing user supplied input, this\nmay be used as a denial of service vector.\n",
      "affects": {
        "ranges": [
          {
            "type": "SEMVER",
            "fixed": "2.2.8"
          }
        ]
      },
      "aliases": [
        "CVE-2019-11254"
      ],
      "modified": "2021-04-14T12:00:00Z",
      "published": "2021-04-14T12:00:00Z",
      "ecosystem_specific": {
        "symbols": [
          "yaml_parser_fetch_more_tokens"
        ]
      },
      "database_specific": {
        "source": "https://storage.googleapis.com/go-vulndb/gopkg.in/yaml.v2.json",
        "url": "https://go.googlesource.com/vulndb/+/refs/heads/master/reports/GO-2020-0036.yaml"
      },
      "references": [
        {
          "type": "FIX",
          "url": "https://github.com/go-yaml/yaml/pull/555"
        },
        {
          "type": "FIX",
          "url": "https://github.com/go-yaml/yaml/commit/53403b58ad1b561927d19068c655246f2db79d48"
        },
        {
          "type": "WEB",
          "url": "https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=18496"
        }
      ],
      "affected": [
        {
          "package": {
            "name": "gopkg.in/yaml.v2",
            "ecosystem": "Go"
          },
          "ranges": [
            {
              "type": "SEMVER",
              "events": [
                {
                  "introduced": "0"
                },
                {
                  "fixed": "2.2.8"
                }
              ]
            }
          ],
          "ecosystem_specific": {
            "symbols": [
              "yaml_parser_fetch_more_tokens"
            ]
          },
          "database_specific": {
            "url": "https://go.googlesource.com/vulndb/+/refs/heads/master/reports/GO-2020-0036.yaml",
            "source": "https://storage.googleapis.com/go-vulndb/gopkg.in/yaml.v2.json"
          }
        },
        {
          "package": {
            "name": "github.com/go-yaml/yaml",
            "ecosystem": "Go"
          },
          "ranges": [
            {
              "type": "SEMVER",
              "events": [
                {
                  "introduced": "0"
                }
              ]
            }
          ],
          "ecosystem_specific": {
            "symbols": [
              "yaml_parser_fetch_more_tokens"
            ]
          },
          "database_specific": {
            "url": "https://go.googlesource.com/vulndb/+/refs/heads/master/reports/GO-2020-0036.yaml",
            "source": "https://storage.googleapis.com/go-vulndb/gopkg.in/yaml.v2.json"
          }
        }
      ]
    },
    {
      "id": "GO-2021-0061",
      "package": {
        "name": "gopkg.in/yaml.v2",
        "ecosystem": "Go"
      },
      "details": "Due to unbounded alias chasing, a maliciously crafted YAML file\ncan cause the system to consume significant system resources. If\nparsing user input, this may be used as a denial of service vector.\n",
      "affects": {
        "ranges": [
          {
            "type": "SEMVER",
            "fixed": "2.2.3"
          }
        ]
      },
      "modified": "2021-04-14T12:00:00Z",
      "published": "2021-04-14T12:00:00Z",
      "ecosystem_specific": {
        "symbols": [
          "decoder.unmarshal"
        ]
      },
      "database_specific": {
        "url": "https://go.googlesource.com/vulndb/+/refs/heads/master/reports/GO-2021-0061.yaml",
        "source": "https://storage.googleapis.com/go-vulndb/gopkg.in/yaml.v2.json"
      },
      "references": [
        {
          "type": "FIX",
          "url": "https://github.com/go-yaml/yaml/pull/375"
        },
        {
          "type": "FIX",
          "url": "https://github.com/go-yaml/yaml/commit/bb4e33bf68bf89cad44d386192cbed201f35b241"
        }
      ],
      "affected": [
        {
          "package": {
            "name": "gopkg.in/yaml.v2",
            "ecosystem": "Go"
          },
          "ranges": [
            {
              "type": "SEMVER",
              "events": [
                {
                  "introduced": "0"
                },
                {
                  "fixed": "2.2.3"
                }
              ]
            }
          ],
          "ecosystem_specific": {
            "symbols": [
              "decoder.unmarshal"
            ]
          },
          "database_specific": {
            "url": "https://go.googlesource.com/vulndb/+/refs/heads/master/reports/GO-2021-0061.yaml",
            "source": "https://storage.googleapis.com/go-vulndb/gopkg.in/yaml.v2.json"
          }
        },
        {
          "package": {
            "name": "github.com/go-yaml/yaml",
            "ecosystem": "Go"
          },
          "ranges": [
            {
              "type": "SEMVER",
              "events": [
                {
                  "introduced": "0"
                }
              ]
            }
          ],
          "ecosystem_specific": {
            "symbols": [
              "decoder.unmarshal"
            ]
          },
          "database_specific": {
            "source": "https://storage.googleapis.com/go-vulndb/gopkg.in/yaml.v2.json",
            "url": "https://go.googlesource.com/vulndb/+/refs/heads/master/reports/GO-2021-0061.yaml"
          }
        }
      ]
    }
  ]
}

I was wondering what is the license of the data you provide?
I could not find something explicit.
This is to possibly import these in nexB/vulnerablecode#341 as we were alredy planning to import oss-fuzz data otherwise.
Thanks

We should display where we obtained a source from on the vulnerability page.

Work with the Dart team to support tracking of Dart Open Source vulnerabilities.

Relates to dart-lang/pub-dev#2160.

Internal clients are all migrated. We can remove and turn this down next week.

It would be nice to have a scanner or analysis tool available that scans dependencies of Maven (pom.xml) or NPM (package.json) projects and checks the components against the API.

Packagist recommends, but does not enforce Semver

if an oss-fuzz bisection resolves to a range of commits, we should move the range to database_specific instead.

the affects.ranges fields should have single commit entries.

Current logic is here

Here are a few I've seen from examining some of my recent manual triages:

• x.x.x up to and including x.x.x
• x.x.x and earlier
• x.x.x or older
• x.x.x and below
• up to and including x.x.x
• prior to x.x.x
• below x.x.x

Support to identify package with package url would be nice and it will be easy to integrate with other tool chain.
https://github.com/package-url/purl-spec

Add another option to the false positive configuration to allow ignoring specific package:vulnerability id combinations when there is just a bad match but the vulnerability still applies to other packages in the ecosystem

Currently bisection only works for OSS-Fuzz vulnerabilities. We need to extend our Vulnerability schema with reproduction steps to support generic automated bisections.

Hi,

I just had a first look at OSV and have to say that it really looks like a great help to me. Thank you very much for starting this.

Unfortunately, I noticed that at least one issue's states are inconsistent in oss-fuzz and OSV:
oss-fuzz 24266 is "Verified" since January 5th, so all versions greater than v6.0.0 should be fixed, but the resulting OSV-2020-902 still reads "Not fixed". Could you please double-check this?

In case it matters: The issue was found in Qt built from branch 5.15. It was marked "Verified" immediately after I moved to building from the dev branch.

Cheers,
Robert

Some times project maintainers leave references to OSS-Fuzz bug or testcase IDs in the commit message. We can use them.

• https://github.com/PyCQA/bandit
• https://bandit.readthedocs.io/

purl is a recommended field within the package object and should be easy enough to create based off of information we already know.

Sometimes bisection fails to resolve to a small enough range and we don't include the result in OSV at all. We should still push this to a separate branch to allow users to manually fix these.

Parts of OSV (impact analysis, bisection) may be run as part of a CI workflow instead.

We're sorry but frontend2 doesn't work properly without JavaScript enabled. Please enable it to continue.

You are not sorry, if you were sorry, this website wouldn't have tried to coerce me into enabling JS.

Also https://osv.dev/docs/ don't work.

At least the following from the pypa advisory set:

• gitlab.com
• bitbucket.org

Currently cherry picks are detected based on the git patch-id.

We can add additional heuristics like searching for the "cherry picked from" string in commit messages in cases where the cherry pick results in different patch-ids.

Should we extract GHSA IDs as an alias entry when it is present on an NVD CVE entry?

Counterpart to pypi/warehouse#9552

In some cases, IDs may not be unique across databases.

There are about 6000 OSS-Fuzz issues tagged as bug-security: https://bugs.chromium.org/p/oss-fuzz/issues/list?q=Type%3DBug-Security%20-status%3AWontFix%2CDuplicate&can=1

However, the OSS Fuzz Vuln Data only contains about 1500 Vulnerabilities. Why is that the case? It seems that early issues, like https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=9&q=Type%3DBug-Security%20-status%3AWontFix%2CDuplicate&can=1, are not present?

If a vulnerability has multiple commit ranges, it may be useful to provide a mapping of versions to commit ranges.

Extend the current OSS-Fuzz pipeline to support generic data sources.

• Input: Partially filled Vulnerability data from a repository.
• Output: Vulnerability with appended affected commit range and versions info pushed back to the same repository and location.

More requirements:

• Parts of OSV will also be made to run as part of CI workflows (#51)
• Users can also submit PRs to fix up details in Vulnerability entries.
• OSS-Fuzz data will be dumped as well in the same format.

Current sketch (subject to change):

• https://sarifweb.azurewebsites.net/

What are the existing [container] artifact vuln scanners, databases, and specs?

tools, vuln databases

awesome-docker-security >
Container [vuln] scanning:

• @aquasecurity/#Trivy
• @quay/#Claire
• @project_harbor/#Harbor #CNCF
• @anchore/#Syft
• @anchore/#Grype
• #Dagda works w/ Falco
• @falco_org/#Falco #CNCF #sysdig
• @snyksec/#Snyk sends PRs
  https://github.com/myugan/awesome-docker-security#container-scanning

• List of CNCF open source security projects:
  https://landscape.cncf.io/card-mode?category=security-compliance&grouping=category&license=open-source
• https://analysis-tools.dev/tag/security
• https://github.com/TaptuIT/awesome-devsecops#dependency-management
• Trivy data sources:
  https://github.com/aquasecurity/trivy#data-sources
• https://github.com/anchore/grype
  • syft scans and generates a SBOM, grype does lookups etc
  • https://github.com/anchore/grype/tree/main/grype/matcher
• https://github.com/DependencyTrack
• https://github.com/eliasgranderubio/dagda
  • https://github.com/jeremylong/DependencyCheck
    • https://www.owasp.org/index.php/OWASP_Dependency_Check

      Dependency-Check is a Software Composition Analysis (SCA) tool that attempts to detect publicly disclosed vulnerabilities contained within a project’s dependencies. It does this by determining if there is a Common Platform Enumeration (CPE) identifier for a given dependency. If found, it will generate a report linking to the associated CVE entries.

  • ClamAV, Falco

standards

SBOM standards:

• https://cyclonedx.org/

For PyPI vulnfeeds, there are some cases of bad data in the form of:

introduced: 1.0
fixed: 1.0b4

Encoded like so, this means that everything after and including 1.0 is affeted (because 1.0b4 comes before 1.0). This should instead be something like

introduced: 1.0a0
fixed: 1.0b4

We need to detect these cases in the vulnfeeds tool.

should be runnable as part of CI as well.

• For the /vulns/ GET endpoint, there is no need for rate limiting.
• For the query endpoint, we can support anonymous rate limiting.

Here are a few suggestions to improve matching based on links for pypi packages

• strip .git from the end of source control links. For instance, in the pypi_links.json input file, for the notebook package, we have https://github.com/jupyter/notebook.git, but in the actual CVE references it is just https://github.com/jupyter/notebook, so a match is never found

• Convert older-style pypi urls to https://pypi.org/project/ urls. There are at least a couple of older ones: https://pypi.python.org/pypi/, https://upload.pypi.org/legacy/. We could also transform https://pypi.org/simple/

Currently, all of the references on a CVE entry from NVD just get classified with type WEB. We should be able to classify at least some types based on well-known URL patterns, similar to what is being done in the GHSA converter at https://github.com/ossf/osv-schema/blob/d4d2764253eeb002daa738440ae6333c8082d069/tools/ghsa/convert_ghsa.py#L115.

We could also potentially set the FIX type when extracting the git fix range.

NuGet is almost semver, but not exactly: https://docs.microsoft.com/en-us/nuget/concepts/package-versioning#where-nugetversion-diverges-from-semantic-versioning

google/oss-fuzz-vulns#7 (comment)