Remove these allow's and resolve all the warnings that result from doing so.

Seems there is free conversion from two versions of CPE, so no need to maintain versioned tables (yet) for CPE. Instead, we should rename and accommodate whatever superset of columns is required.

Current status

• From the README.md file the period can be written as 60s to express "60" seconds.

http POST localhost:8080/api/v1/importer/redhat-csaf csaf[period]=60s

• The list of importers at http://localhost:8080/api/v1/importer can return period=1m

[
    {
        "name": "redhat-csaf",
        "configuration": {
            "csaf": {
                "disabled": false,
                "period": "1m",
                "source": "https://redhat.com/.well-known/csaf/provider-metadata.json",
                "v3Signatures": false
            }
        },
        "state": "waiting",
        "lastChange": "2024-04-16T06:32:13.941898Z",
        "lastRun": "2024-04-16T06:32:11.428871Z",
        "lastError": "Visitor error: Invalid signature: Invalid key: \"Subkey of 77E79ABE93673533ED09EBE2DCE3823597F5EAC4 not bound: No binding signature at time 2024-04-16T04:20:22Z\""
    }
]

Problem

There is no unique way of expressing period.
period should be a number (seconds, milliseconds or whatever else) as an Input and Output. The Clients (e.g. UI) and Backend should then translate the number to the format of their preference.

          For bonus points, do we want to re-hash to also ensure file contents didn't change during the shuffling?

Originally posted by @bobmcwhirter in #114 (comment)

Navigating through the Graph code, I see the following pattern:

• get_foo -> if Some -> return foo
• ingest parent of foo
• add foo to parent
  • get_foo -> if Some -> return foo
  • insert foo -> return foo

That patterns nests into the parent structures.

Without having a concrete idea, I have the expectation that we could at least eliminate the second get. As for a medium sized SBOM we have ~8000 of those calls, I think it makes sense to improve the performance.

Existing tests are more integration than unit, i.e. they require postgres to run successfully

https://www.sea-ql.org/SeaORM/docs/install-and-config/debug-log/

yes, RHT is CSAF, but... others do otherwise.
hopefully also help us generalize the fetch/ingest/store pipeline across formats.
so we're not having to mirror osv_walker like csaf_walker like ghsa_walker like msftadvisory_walker, like...
Walker<CSAF, Http, S3>::new()...
Walker<OSV, FileSystem, FileSystem>::new()
or whatnot
Walker<FORMAT, SOURCE, STORAGE_DEST>

          I think the answer is TryFrom instead since failure is an option.

Originally posted by @bobmcwhirter in #3 (comment)

Converting to TryFrom has rippling effects that bring into question the existence of both Purl and PackageUrl types, as well as their corresponding PurlErr and packageurl::Error types, not to mention the system::Error type.

Followed the instructions in the README.md and something about webpack failed and nothing is listening on port 3000.

bob@macbox trustify % cd frontend
bob@macbox frontend % ls
Dockerfile		README.md		branding		client			common			entrypoint.sh		package-lock.json	package.json		scripts			server
bob@macbox frontend % npm clean-install --ignore-scripts


npm WARN deprecated [email protected]: Modern JS already guarantees Array#sort() is a stable sort, so this library is deprecated. See the compatibility table on MDN: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/sort#browser_compatibility
npm WARN deprecated [email protected]: Use your platform's native atob() and btoa() methods instead
npm WARN deprecated [email protected]: Use your platform's native DOMException instead

added 1508 packages, and audited 1512 packages in 18s

387 packages are looking for funding
  run `npm fund` for details

3 moderate severity vulnerabilities

Some issues need review, and may require choosing
a different dependency.

Run `npm audit` for details.
npm notice
npm notice New minor version of npm available! 10.2.4 -> 10.5.0
npm notice Changelog: https://github.com/npm/cli/releases/tag/v10.5.0
npm notice Run npm install -g [email protected] to update!
npm notice
bob@macbox frontend %
bob@macbox frontend % npm run start:dev


> @trustification-ui/[email protected] start:dev
> concurrently -n port-forward,common,client -c 'white.bold.inverse,green.bold.inverse,blue.bold.inverse' 'npm:port-forward' 'npm:start:dev:common' 'npm:start:dev:client'

[common]
[common] > @trustification-ui/[email protected] start:dev:common
[common] > npm run start:dev -w common
[common]
[port-forward]
[port-forward] > @trustification-ui/[email protected] port-forward
[port-forward] > concurrently -c auto 'npm:port-forward:*'
[port-forward]
[client]
[client] > @trustification-ui/[email protected] start:dev:client
[client] > npm run start:dev -w client
[client]
[client]
[client] > @trustification-ui/[email protected] start:dev
[client] > NODE_ENV=development webpack serve --config ./config/webpack.dev.ts
[client]
[common]
[common] > @trustification-ui/[email protected] start:dev
[common] > NODE_ENV=development rollup -c --watch
[common]
[port-forward] [hub]
[port-forward] [hub] > @trustification-ui/[email protected] port-forward:hub
[port-forward] [hub] > kubectl port-forward svc/trustification-hub -n trustification 9002:8080
[port-forward] [hub]
[port-forward] [hub] sh: kubectl: command not found
[port-forward] [hub] npm run port-forward:hub exited with code 127
[port-forward] npm run port-forward exited with code 1
[common] Using branding assets from: /Users/bob/repos/trustification/trustify/frontend/branding
[common] rollup v4.12.0
[common] bundles src/index.ts → dist/index.mjs, dist/index.cjs...
[client] [webpack-cli] Failed to load '/Users/bob/repos/trustification/trustify/frontend/client/config/webpack.dev.ts' config
[client] [webpack-cli] config/webpack.dev.ts(18,8): error TS2307: Cannot find module '@trustification-ui/common' or its corresponding type declarations.
[client]
[client] npm ERR! Lifecycle script `start:dev` failed with error:
[client] npm ERR! Error: command failed
npm ERR!   in workspace: @trustification-ui/[email protected]
[client] npm ERR!   at location: /Users/bob/repos/trustification/trustify/frontend/client
[client] npm run start:dev:client exited with code 1
[common] created dist/index.mjs, dist/index.cjs in 787ms

          Because otherwise tests will fail. Having `init` will run that methods twice (for two different tests) and the second one will fail. That's why the CI just failed, because the logger was already initialized by the first test.

Originally posted by @ctron in #81 (comment)

Let's use the test-log crate to obviate this, similar to what we've done in other workspaces.

          Why do we need to reimplement [upstream](https://docs.rs/sea-orm/latest/sea_orm/trait.ConnectionTrait.html)?

Originally posted by @jcrossley3 in #77 (comment)

Importing a single SBOM took:

[2024-04-16T11:07:11.134Z INFO  trustify_module_ingestor::service::sbom] Ingested - took 57m 28s 566ms 326us 659ns

I don't have any detailed information which one this was as the log was full of warnings. However, I do think that we need gather some data and start working on it.

Tracking:

• #177

Some day SHA256 will be broken maybe, so let's go ahead and hash SHA384 and whatever else seems sensible.

We have other .gitignore here:
https://github.com/trustification/trustify/blob/main/frontend/.gitignore

Maybe just add the actual necessary changes to the root/.gitignore ?

For PM mode, we're enjoying the embedded postgres.

I'd like us to be able to use the Trusted Lang Extension and Apache AGE perhaps.

Can we somehow do the embedded route, but with baked-in extensions? Dunno!

Some stuff is under /api/v1

Some stuff is right at the root.

We should be more consistent.

In light of #47

Having a Cargo.toml at the root simplifies the dev cycle.

Further, let's put everything that's not a cargo workspace beneath a single directory. Do we like etc/, var/ or something else?

At the moment we are using a Path Parameter to define the name of an importer. Although it works, we should follow common practices when we define our endpoints, here is a basic example:

• GET /personas: List of personas
• POST /personas: Create a single persona. Any info required goes in the body of the POST request
• GET /personas/{id}: get a single persona
• PUT /personas/{id}: update a persona
• DELETE /personas/{id}: delete a persona

So /api/v1/importer/{name} should be changed by /api/v1/importer/ and the name should be defined in the body of the request.

cargo test

error: failed to run custom build command for `sequoia-openpgp v1.19.0`
Caused by:
  process didn't exit successfully: `Desktop/tc/trustify/target/debug/build/sequoia-openpgp-49fc058589fa4216/build-script-build` (exit status: 1)
  --- stderr
  No cryptographic backend selected.

  Sequoia requires a cryptographic backend.  This backend is selected at compile
  time using feature flags.

  See https://crates.io/crates/sequoia-openpgp#crypto-backends

nettle related ? #61

In theory, $PWD/.trustify/... should be retained across runs and not blow away the DB each time by default.

In reality, with the exception of for_test(...) we don't use db::DbStrategy::Managed(Arc<...>), while we do use config::DbStrategy::Managed but ultimately manage it from the trustd wrapper around the DB. The Db we construct and give to Graph is External from its POV, which makes complete sense.

I suspect we should rework the for_test(...) logic to be more test-fixure-y, and less ingrained in Db/Graph logic.

I think this should be enough...

cargo run --bin trustd -- --auth-disabled

...to make this url functional in a browser:

http://localhost:8080/swagger-ui/openapi.json

It doesn't currently work. Let's fix either it or my expectations, thanks!

Just SHAs.

Some SBOMs provide a long list of messages like:

[2024-04-16T11:18:45.031Z WARN  trustify_module_graph::graph::sbom] unable to ingest relationships involving a non-fully-qualified package pkg://generic/libnestegg
[2024-04-16T11:18:45.054Z WARN  trustify_module_graph::graph::sbom] unable to ingest relationships involving a non-fully-qualified package pkg://generic/libnestegg
[2024-04-16T11:18:46.405Z WARN  trustify_module_graph::graph::sbom] unable to ingest relationships involving a non-fully-qualified package pkg://golang/sync/atomic
[2024-04-16T11:18:46.428Z WARN  trustify_module_graph::graph::sbom] unable to ingest relationships involving a non-fully-qualified package pkg://golang/sync/atomic
[2024-04-16T11:18:46.452Z WARN  trustify_module_graph::graph::sbom] unable to ingest relationships involving a non-fully-qualified package pkg://golang/sync/atomic
[2024-04-16T11:18:46.475Z WARN  trustify_module_graph::graph::sbom] unable to ingest relationships involving a non-fully-qualified package pkg://golang/sync/atomic

Each entry takes seconds to process. I am wondering if it is necessary to process those entries multiple times? And of course the question is: why is that so? what can a user do? how can we properly report this? (ok, more than one question).

Should we define DTOs or are we happy just using the DB Models for transferring data between UI and REST APIs?
I think we should not use the DB models directly but DTOs. What do you think?

Right now ingesting SBOMs slow (see #165). A first investigation shows that a lot of traffic happens on the tables around packages (package versions, qualified packages, …). I think it makes sense exploring alternative ways of storing this information.

Ideas:

• flat table: package url, split up, qualifiers as JSON field (?)
• ltree: might not work as we don't have actual "labels"
• using values as primary key (vs auto-increments), not sure this works

One aspect of this is also that question if a package "belongs" to an SBOM. Or is just referenced by an SBOM. The question is, what will happen when an SBOM gets deleted. IMHO this should remove all packages stored in the database as well.

We need an endpoint e.g: GET /advisories That returns a paginated list of advisories. The endpoint should get as an input:

• offset, limit (for pagination)
• sorting
• Filtering (let's also discuss which are the filters we are planning to support. So far I think severity is a good one)

The endpoint should return as an output:

• List of items that match the input criteria
• Number of total items that match the input criteria

The data used will serve to populate the following table

          Not for this PR, but when we add a 3rd format, probably smart to shuffle into a `fn` on `Format` itself to handle the delegation to the appropriate loader?

Originally posted by @bobmcwhirter in #114 (comment)

Just as @helio-frota initially did the workspace=true changes to our Cargo.toml's, I believe we can go a step further.

https://github.com/trustification/trustify/blob/main/modules/importer/Cargo.toml#L7-L10

Where sub-crates use a path = '...' we can also specify that at the workspace root (I think), and move every one to workspace=true instead of having to possibly adjust lots of paths if we shuffle things around.

Instead of having a database ID field and a digest field, there should be a single digest field. The digest should be the same as used for identifying the document in the storage/filesystem.

Can make filtering results a bit more concise.

          I've also used this pattern, and while not on your shoulders, I feel like we are repeating ourselves a lot, and perhaps @jcrossley3 could get us some more ergonomics around using search/sort.

Originally posted by @bobmcwhirter in #167 (comment)

It is not possible to know which is the Body required to be send on the importers endpoints:

This affects multiple endpoints related to the importers

Yeah?

What's needed for authentication?

Do we need a KeyCloak? Can we defer (for dev at least?) to GH OAuth or similar?

Required REST endpoints for trustify-ui

• SBOMs
• CVEs
• Advisories
• Packages

          Maybe `.ok_or(Error::InvalidFormat)?` which will turn a None into a throw error, or a Some into the underlying value without unwrap.  Sorta the err-variant of `unwrap_or(...)`

Originally posted by @bobmcwhirter in #114 (comment)

➜  trustify git:(main) rg "Database\:\:for_test"
modules/ingestor/src/service/cve/loader.rs
80:        let db = Database::for_test("ingestors_cve_loader").await?;

modules/importer/src/test.rs
30:    let db = Database::for_test("test_default").await.unwrap();
125:    let db = Database::for_test("test_oplock").await.unwrap();

modules/ingestor/src/service/advisory/osv/loader.rs
154:        let db = Database::for_test("ingestors_osv_loader").await?;

modules/ingestor/src/endpoints/advisory.rs
74:        let db = Database::for_test("upload_advisory_csaf").await?;
101:        let db = Database::for_test("upload_advisory_osv").await?;
128:        let db = Database::for_test("upload_unknown_format").await?;

modules/graph/src/graph/vulnerability/mod.rs
146:        let db = Database::for_test("ingest_cve").await?;
173:        let db = Database::for_test("get_advisories_from_cve").await?;

modules/graph/src/graph/advisory/mod.rs
469:        let db = Database::for_test("ingest_advisories").await?;
507:        let db = Database::for_test("ingest_affected_package_version_range").await?;
564:        let db = Database::for_test("ingest_fixed_package_version").await?;
624:        let db = Database::for_test("ingest_advisory_cve").await?;
651:        let db = Database::for_test("advisory_affected_vulnerability_assertions").await?;
692:        let db = Database::for_test("advisory_not_affected_vulnerability_assertions").await?;

modules/graph/src/graph/sbom/mod.rs
685:        let db = Database::for_test("ingest_sboms").await?;
710:        let db = Database::for_test("ingest_and_fetch_sboms_describing_purls").await?;
760:        let db = Database::for_test("ingest_and_locate_sboms_describing_cpes").await?;
810:        let db = Database::for_test("transitive_dependency_of").await?;
879:        let db = Database::for_test("ingest_contains_packages").await?;
951:        let db = Database::for_test("sbom_vulnerabilities").await?;

modules/graph/src/graph/package/package_version.rs
218:        let db = Database::for_test("package_version_not_affected_assertions").await?;

modules/graph/src/graph/sbom/spdx.rs
174:        let db = Database::for_test("parse_spdx_quarkus").await?;
228:        let db = Database::for_test("parse_spdx_openshift").await?;
280:        let db = Database::for_test("parse_spdx").await?;

modules/graph/src/graph/package/mod.rs
618:        let db = Database::for_test("ingest_packages").await?;
651:        let db = Database::for_test("ingest_package_versions_missing_version").await?;
676:        let db = Database::for_test("ingest_package_versions").await?;
711:        let db = Database::for_test("get_versions_paginated").await?;
768:        let db = Database::for_test("ingest_qualified_packages_transactionally").await?;
808:        let db = Database::for_test("ingest_qualified_packages").await?;
866:        let db = Database::for_test("ingest_package_version_ranges").await?;
918:        let db = Database::for_test("package_affected_assertions").await?;
1011:        let db = Database::for_test("package_not_affected_assertions").await?;
1066:        let db = Database::for_test("package_vulnerability_assertions").await?;
1130:        let db = Database::for_test("advisories_mentioning_package").await?;

modules/graph/src/graph/package/qualified_package.rs
142:        let db = Database::for_test("vulnerability_assertions").await?;

It works with the same database name

let db = Database::for_test("a").await?;

#136 (comment)

But I don't know the reason so I'm asking, thanks 👍

For documents which have their own document ID and which can change over time (like CSAF documents), there must be a way to get that latest document by its document ID. When searching, only the latest document must be searched and returned (unless we want an additional API to search through the history).

Can't consider every value a String, obviously. Probably need to use sea_query::Value instead...

[jim@fedora trustify]$ http localhost:8080/api/v1/search/advisory 'q==denial&published>2023-11-03'
HTTP/1.1 500 Internal Server Error
access-control-allow-credentials: true
access-control-expose-headers: content-type
content-encoding: gzip
content-type: application/json
date: Mon, 15 Apr 2024 15:51:47 GMT
transfer-encoding: chunked
vary: Origin, Access-Control-Request-Method, Access-Control-Request-Headers
vary: accept-encoding

{
    "error": "Internal",
    "message": "database error: Query Error: error returned from database: operator does not exist: timestamp with time zone > text"
}

Current status:

https://github.com/trustification/trustify/blob/52b2cc9dfaa325ffcfdaee30e38371e444443b1f/modules/ingestor/src/endpoints/advisory.rs#L14C1-L26C23

#[utoipa::path(
    tag = "ingestor",
    request_body = Vec <u8>,
    params(
        ("format" = String, Query, description = "Format of the submitted advisory document (`csaf`, `osv`, ...)"),
        ("location" = String, Query, description = "Source the document came from"),
    ),
    responses(
        (status = 201, description = "Upload a file"),
        (status = 400, description = "The file could not be parsed as an advisory"),
    )
)]
#[post("/advisories")]

Problem

• Format & location query parameters are required to upload a file

It means that the UI can not just take a file and upload it. It requires the user fill a form to specify the format of the file he is uploading, which is in my opinion prone to error and not user friendly. Imagine yourself uploading a file using CURL but you can not just take a file and upload it, but you have to determine which is the format of it. The problem is for humans, probably not for other apps

Proposed solution

• The FILE should be the only required thing, everything else should be determined by the server

  • For the location query parameter: Technically the UI can be deployed in a different domain than the Backend, There can be multiple UI domains pointing to the same backend. location should be determined in the backend side as the relative path of the rest endpoint used e.g. /advisories
  • for the format query parameter. There are JSON Schemas for each of the formats supported. Why we can just use the schemas to validate to which one the file belongs to? e.g. User uploads a file => the server compares the file to each of the supported json schemas => the format is determined by the server as the schema to which the file matches. That should be it. The JSON Schemas are

• CSAF Schema: https://github.com/oasis-tcs/csaf/blob/master/csaf_2.0/json_schema/csaf_json_schema.json

• OSV Schema https://github.com/ossf/osv-schema/blob/main/validation/schema.json

• CVE
  Schema: https://github.com/CVEProject/cve-schema/blob/master/schema/v5.0/CVE_JSON_5.0_schema.json

I find the warnings spamming the CI reports massively annoying and distracting.

My expectation would be that if there warnings like that, a PR won't get merged before they are resolved. Or, we disable them as we don't seem to care.

Right now, reviewing a PR and getting dozens of unrelated warnings is not helping.

Since a recent change the infrastructure part is no longer wrapping the lifecycle of the application, but only of the http part. That should be changed.

Whatever trustd run should be wrapped with the infrastructure stuff. That would also eliminate the need to use println.