Git Product home page Git Product logo

opossum-tool / opossumui Goto Github PK

View Code? Open in Web Editor NEW
54.0 2.0 25.0 36.74 MB

A light-weight app to audit and inventory large codebases for open source license compliance.

License: Apache License 2.0

Shell 0.18% JavaScript 0.47% HTML 0.08% TypeScript 99.19% Batchfile 0.08%
spdx package-dependency remediation license-scan copyright-scan software-composition-analysis codescan oss-compliance software-bill-of-materials

opossumui's Introduction

logo

OpossumUI: A compliance audit/review frontend.

OpossumUI is a tool to

  • discover open source software used in applications.
  • review licenses.
  • generate reports from an open source code scan.

Features

  • use of multiple scanners (open source and/or proprietary).
  • currently integrates with OSS Review Toolkit, FOSSLight and ScanCode.
  • unified interface for browsing scanner evidence.
  • simple navigation through the codebase’s file tree.
  • create attributions for individual files or groups.

License REUSE status GitHub release (latest by date) build workflow build workflow

screenshots_of_the_ui

Use Cases

  • A team performing audits for open source license compliance.
  • Producing legal docs such as inventory (BOMs) and license conflicts that need to be remediated.
  • OpossumUI can be used to visually inspect (QA) existing attributions, identify any false positives or incorrect attributions by manual selection and improve existing data.
  • To aid M&A due diligence activities. OpossumUI can be used by acquirers for blind audits of intellectual property, since only compliance-relevant metadata is exposed in the app without the need to ever share the source code.

Motivation

OpossumUI was developed with the goal to build a tool for managing and combining open source compliance data from different sources. While existing analysis tools for software compliance can provide good information, using multiple of such tools often leads to huge amounts of data due to an increased detection rate. Even though the results can be merged and noise can be filtered through automatic tools, final manual revisions are often necessary. So, OpossumUI was born: A light-weight app for review of compliance information for large codebases.

Integration in the ecosystem

To integrate well with different analysis tools, different converters are provided to generate input files in json format that can be opened with OpossumUI. After completing the review process with OpossumUI, the information can be outputted in different formats.

integration

Getting Started

Input files

To work with OpossumUI, an input file can be opened to visualize and edit license compliance data of a project. Input files typically have the file ending .opossum, while opening the deprecated file format that has the file ending .json is still supported. In this case the app will inform the user that the deprecated file format is automatically converted to the new file format.

Generating input files

Result files from ScanCode and SCANOSS can be converted into OpossumUI input files using opossum-tool/opossum.lib.hs. This tool can also generate OpossumUI input files from spdx and merge several OpossumUI input files.

Result files (yaml / json) from the OSS Review Toolkit can be converted into OpossumUI input files via a reporter. It uses metadata from the analyzer as well as scan results from the scanner. The implementation is in oss-review-toolkit/ort and the new reporter output is called Opossum.

For details of the file format, see file formats.

How to get and run OpossumUI

Check out our short getting started video:

click to play video

Get the latest release

Download the latest release for your OS from GitHub.

Running the app

Linux

Run the executable OpossumUI-for-linux.AppImage

macOS

Run OpossumUI in OpossumUI-for-mac.zip.

Windows

Run OpossumUI-for-win.exe to install the OpossumUI. Then open OpossumUI from the start menu.

Working with OpossumUI

Check out our short video, which presents a basic workflow.

For an in-depth explanation, please read the Users's Guide.

File formats

Files with a .opossum extension are zip-archives which contain an input.json (must be provided) together with an output.json (optional). JSON schemas for both the input and output files are available. Example files can be found under example files.

Input file

It has to be generated through external tools and provided to the app. Contains 5 main fields:

  • metadata: contains some project-level information,
  • resources: defines the file tree,
  • externalAttributions: contains all attributions which are provided as signals (preselected signals will be automatically used by the app to create attributions in the output file),
  • resourcesToAttributions: links attributions to file paths,
  • frequentlicenses: A list of licenses that can be selected in a dropdown when the user enters a license name.

There are additional fields which are optional:

  • attributionBreakpoints: a list of folder paths where attribution inference stops, e.g. node_modules."

  • filesWithChildren: a list of folders that are treated as files. This can be used to attach another file tree to files like package.json, usually also setting an attribution breakpoint.

  • baseUrlsForSources: a map from paths to the respective base url. The base url should contain a {path} placeholder. E.g.

    "baseUrlsForSources": {
      "/": "https://github.com/opossum-tool/opossumUI/blob/main/{path}"
    }
  • externalAttributionSources: used to store a mapping of short names for attribution sources to full names and priorities used for sorting in the PackagePanel. Entries with higher numbers have a higher priority. E.g.:

    "externalAttributionSources": {
      "SC": {
        "name": "ScanCode",
        "priority": 1
      }
    }

Output file

Contains four main fields:

  • metadata: contains some project-level information,
  • manualAttributions: contains all attributions created by the user or preselected,
  • resourcesToAttributions: links attributions to file paths,
  • resolvedExternalAttributions: used to store which signal attributions have been resolved, as they are hidden in the UI.

Exporting data

In addition to the default output file, OpossumUI provides the following export options.

Exporting SPDX documents

An SPDX document can be exported in the json and the yaml format through the ExportSPDX (yaml) and SPDX (json) option in the File menu.

Exporting BOM-like CSV files

These can be exported through the ExportCompact / Detailed component list option in the File menu. Both component list files contain a list of all attributions that are present in the project, including package name, version, copyright, license name and URL. In addition, the detailed component list is more comprehensive and includes the PURL and its subcomponents, as well as the license texts.

Exporting follow-up document

This can be exported through the ExportFollow-Up option in the File menu. Similar to the component list, it contains attributions with licenses flagged for legal review through the Follow-Up checkbox in the UI.

Limitations

SPDX License Expressions are only partially supported at the moment. Currently, a license expression can only be entered as license name of a package. The full license text of the different licenses (e.g. GPL-2.0-only OR BSD-2-Clause) that apply should also be entered in the license text field.

Developer's guide

Contributions to the project are welcome. See Contributing.

Licensing

OpossumUI is licensed under Apache-2.0, documentation is licensed under CC0-1.0. For contributions, we use the Developer Certificate of Origin (DCO) process via sign-offs in every commit, to help ensure licensing criteria are met.

opossumui's People

Contributors

alexzurbonsen avatar antonbauhofer avatar benedikt-richter avatar billwuzl avatar davidmundelius avatar dependabot[bot] avatar florianschepers avatar florianscheperstng avatar jakobschubert avatar jonas-tai avatar lennartclaas avatar leslielazzarino avatar markusobendrauf avatar maxhbr avatar mayayunx avatar meretp avatar mstykow avatar nicarl avatar renovate[bot] avatar samantatarun avatar sebathomas avatar someshkhandelia avatar syotfs avatar vasily-pozdnyakov avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar

opossumui's Issues

Enhancement: Pull license information from external sources

In case valid PURL or URL is specified I would like to be able to pull license and copyright information from external sources such as pypi, clearly defined or github.
In the best case this could be somehow configurable, in a first POC one could limit the supported sources.

Helpful links/resources:
https://api.clearlydefined.io/api-docs/
https://docs.github.com/en/rest/reference/licenses
curl -sG -H 'Host: pypi.org' -H 'Accept: application/json' https://pypi.org/pypi/numpy/json

Get rid of double scrollbars in Signals column

As a user I do not want to have two scroll bars in the signals column. The result should satisfy the following goals:

  • it should be easy to skip to another section of signals, e.g. jump to ScanCode
  • if there are just a few signals, e.g. 4 from different sources for a file, they should all be visible at first glance
  • If there are many signals, it should be easy to get an understanding of the "most common signals" over all sources

Ideas:

  • A possible solution is to have - besides the accordions for signals, signals in folder content, and attributions in folder contents - accordions that contain the respective signals from different sources. These would initially be collapsed. If a user wants to look into one they have to expand the respective source. This could remove the double scrollbar.
  • Another idea would be to remove the accordions completely and put everything into a single virtual list (maybe with some way to quickly go to a certain section).
  • A scroll trough accordeon would be the premium solution.

Now:
image

Terminal is flooded with warnings when running tests

When running tests, e.g. using yarn test:unit the terminal is flooded with warnings like:

  Warning: An update to $Component inside a test was not wrapped in act(...).
      
      When testing, code that causes React state updates should be wrapped into act(...):
      
      act(() => {
        /* fire events that update state */
      });
      /* assert on the output */

Some of these warnings can be fixed by wrapping the testStore.dispatch(...) into an act(...) in the tests. However, not all of these warnings are fixable by this approach.

The goal of the current ticket is to get rid of the warnings.

Note: this might be related to our upgrade to React 18.

Support license expressions explicitly

Currently, there is a free text field that allows adding licenses (or license expressions):

Screenshot 2022-09-12 at 14 13 55

The scope of this issue is to support license expressions explicitly in the UI. I.e. there are several fields for licenses that can be concatenated using AND and OR. Furthermore, exceptions like GPL with a classpath exception have to be handled.

Multiple popup windows

User can open multiple pop up windows (e.g. "Show Project Metadata") at the same time one above the others and have to close them one by one later.

Enable keyboard navigation in package lists

Is your feature request related to a problem? Please describe.
Currently, navigating the package lists, a user has to click each package individually in order to traverse the list.
For the user that implies the need to click a lot.

Navigation:

  • simple up-down navigation with arrows
  • experiment if it's better to select the item with space or if an item should be immediately selected after the navigation event

Describe the solution you'd like
As a user navigating a package list, I want to be able to use keyboard navigation.

E2E tests for accordion workers

Two e2e test are required:

  • one with a small file, that checks that contained signals and contained attributions are correctly displayed in the accordion
  • one with a file big enough to get web workers out of memory
    • about 100MB each for signals and attributions should be enough, check
    • keep the input parsing time to a minimum
      • uncompressed
      • maybe just one attribution each with 100MB text?

Show percentage for signals in folder content

As a user I want to get a quick overview of the percentage of files covered by a signal in the folder content.
Alternatively the total number might be sufficient.

Note: So far we do not use inference for signals (unlike we do for attributions). This seems to be necessary for this feature to make sense.

Add table showing critical licenses

Add a table showing critical licenses (clustered by criticality high or medium). The table should have two columns: One should contain the license name and the other one total number.
Make sure to use deduplicated data as described in #943.

The chosen attribution is not always highlighted

In the attribution view the chosen attribution is not highlighted (background color is not dark) in the list of all attributions if any of the required fields within the description is empty (e.g. license name, version...) and vice versa when hover on item.

  • This is how it works for a complete atrribution.

Capture

  • With empty fields.

Capture1

Signals in Folder content not shown with ORT test file

What is wrong

When opening a large file the Signals in Folder Contents tab does not show any content even though there are signals further down the tree.
Moreover, an error message is logged:

Web worker error in workers context provider:  DOMException: Failed to execute 'postMessage' on 'Worker': Data cannot be cloned, out of memory.

What is expected

The Signals in Folder Contents tab should show some content.
How it looks:
bug_opossum
How it should look like:
opossum_without_bug

Steps to reproduce

Open https://github.com/opossum-tool/OpossumUI/blob/main/example-files/input_from_ort.json.gz and navigate to the root folder.

Enhancement: Allow SPDX JSON documents as input

Many analysis and inventory tools already create SPDX documents. Instead of requiring special OpossumUI JSON schema as input, why not allow using an SPDX JSON document as input? This makes sense especially if OpossumUI can already export SPDX JSON/YAML documents. It would also make OpossumUI more interoperable with existing SBOM tooling.

Improvements for the Signals per Sources table of statistics popup

The following Improvements for the Signals per Sources table should be implemented:

  • use some deduplication for licenses
    • deduplicate by stripping emtpy spaces, dashes, making everything lowercase
    • For display use the license name occuring most often
  • move Signals per Sources table to the bottom of the popup
  • highlight criticality using color (the same we use at other places). Be creative and check what looks good.

Require namespace for certain PURLs

As some purls are highly ambiguous without namespace it should be required to be present for a purl to be valid in the following cases (hardcoded as config):

  • github
  • maven

(chore) Replace fetch with axios

Currently we have some parts where we fetch data from external APIs with fetch. In #472 we started using axios (mostly because it handles error codes for you).

Known usages of fetch:

  • useFetchPackageInfo

Do a search to find any other occurrences. Also look at #472 to see how requests can be mocked.

Improve security of "open URL" feature in Opossum

The first time the user opens a URL, we want to show a warning message with just the domain name and three options:

  • Accept for now and open the link.
  • Accept forever and open the link (store domain in config).
  • Cancel

Hide signals of attributed resources in "signals in folder content" panel

Is your feature request related to a problem? Please describe.
Currently, as a user, I see all the signals attached to resources contained in a folder in the "signals in folder content" panel. However, I,m only interested in signals, that are attached to resources which do not have attribution attached (since these are "unresolved").
Such signals are not of interest, as an attribution that is attached to a resource propagates down and also applies to all its children (until there is another attribution attached to a child). In consequence, the signals are no longer relevant for all resources higher in the resource tree, as there is already an attribution.

Describe the solution you'd like
Therefore, these signals should be hidden in "signals in folder content" panel.

Improve loading spinner

The current implementation of a loading spinner is not the best. As users we want the following:

  • center window on macOs
  • center spinner in Window
  • make it smaller
    Now:
    image

Package card for first party attributions without content is empty

When opening an input file that has signals that consist of only a first party flag, the corresponding package card displays the "first party icon" but no text (see screenshot). Instead it should display some standard placeholder text like e.g. "First Party" or "is first party".

Screenshot 2022-06-13 at 11 28 28

Show small resource tree instead of plain list of resources

Currently, we show a plain list of resources in the "Resources for selected attribution" popup:

Screenshot 2022-09-19 at 15 11 14

Instead a small resource tree containing just these resources should be displayed (basically a filtered tree of all resources). This is similar to the case for the attribution view: #962.

Add huge OpossumUI input file

For testing purposes it would be good to have a large OpossumUI input file that contains a lot of signals from (many) different sources where the field criticality is non-empty.
This would be, in particular, helpful for testing the project statistics popup and performance in general.

[Story] Enable overview of critical licenses in resource tree

Currently the criticality of signals is displayed in the package cards. This enables the user to quickly identify signals that are critical.
However, there is no way to easily identify the resources that have critical signals attached.

It would be valuable to have the following:

  • Resources with critical licenses are highlighted (done, see #1028)
    • The icons indicating presence of signals shall be colored based on highest criticality for resources with critical signals and also their tooltip texts should be modified:
      Screenshot 2022-09-28 at 11 34 38
  • There is a way to easily navigate through these resources
    • Introduce a second clickable progress bar for critical licenses (#1029)
      • there are three colors for no, high, and low criticality (only resources that have a signal but no attribution should be included)
      • there should be a nice on hover message incl the number of resources with critical (high and low separately) signal
      • there is a toggle (right next to existing bar) to switch between the new progress bar and the current one
    • clicking on the new bar brings the user to resources with critical signals and no attributions (high criticality first), see #1030

Enhancement: Faciliate finding a PURL for a signal

Current situation

Package URLs are notoriously difficult to fill in for the user. They are complicated and often there are several possibilities for a given package (e.g. pkg:mavencentral/junit/junit/4.6 and pkg:github/junit/junit4/4.6). #216 partly reduces this pain point as by fetching information from package managers one can often create a valid and meaningful PURL.
However, without a URL and incomplete/ambiguous license information (4.6 vs 4.6.0) it is still very hard to derive a PURL.

Suggestion

ClearlyDefined (CD) offers an endpoint to query their database with a search string.
Example:

curl -X GET https://api.clearlydefined.io/definitions?pattern=junit%404.6
[
  "sourcearchive/mavencentral/junit/junit/4.6",
  "sourcearchive/mavencentral/org.robolectric/junit/4.6",
  "maven/mavencentral/junit/junit/4.6",
  "maven/mavencentral/org.robolectric/junit/4.6",
  "maven/mavencentral/org.robolectric/junit/4.6-alpha-1",
  "sourcearchive/mavencentral/org.robolectric/junit/4.6-alpha-1",
  "maven/mavencentral/org.robolectric/junit/4.6.1",
  "sourcearchive/mavencentral/org.robolectric/junit/4.6.1",
  "maven/mavencentral/org.fluentlenium/fluentlenium-junit/4.6.1",
  "maven/mavencentral/io.cucumber/cucumber-junit/4.6.0",
  "sourcearchive/mavencentral/org.apache.openejb/openejb-junit/4.6.0",
  "sourcearchive/mavencentral/io.cucumber/cucumber-junit/4.6.0",
  "maven/mavencentral/org.apache.openejb/openejb-junit/4.6.0.2",
  "maven/mavencentral/org.apache.openejb/openejb-junit/4.6.0.1",
  "maven/mavencentral/org.fluentlenium/fluentlenium-junit/4.6.2",
  "maven/mavencentral/org.fluentlenium/fluentlenium-junit/4.6.0",
  "maven/mavencentral/org.apache.openejb/openejb-junit/4.6.0",
  "sourcearchive/mavencentral/org.ops4j.pax.exam/pax-exam-invoker-junit/4.6.0",
  "maven/mavencentral/org.ops4j.pax.exam/pax-exam-invoker-junit/4.6.0"
]

I suggest to offer a popup where the user can query this endpoint and sees a list of valid coordinates from (CD).
The user can then select an appropriate coordinate and we convert it to a valid PURL.
Another interesting endpoint is:

curl -X GET https://api.clearlydefined.io/definitions/maven/mavencentral/junit/junit/4.6
{
...
 "licensed": {
    "declared": "BSD-3-Clause",
    "toolScore": {
      "total": 75,
      "declared": 30,
      "discovered": 0,
      "consistency": 15,
      "spdx": 15,
      "texts": 15
    },
    "facets": {
      "core": {
        "attribution": {
          "unknown": 207,
          "parties": [
            "Copyright (c) 2000-2006, www.hamcrest.org"
          ]
        },
        "discovered": {
          "unknown": 207,
          "expressions": [
            "BSD-3-Clause"
          ]
        },
        "files": 208
      }
    },
    "score": {
      "total": 75,
      "declared": 30,
      "discovered": 0,
      "consistency": 15,
      "spdx": 15,
      "texts": 15
    }
  },
  "coordinates": {
    "type": "maven",
    "provider": "mavencentral",
    "namespace": "junit",
    "name": "junit",
    "revision": "4.6"
  },
...
}

One could enrich the list shown to the user with information from this endpoint to help choosing the good coordinate and at the same time get valuable copyright and license information.

Relevant links

https://api.clearlydefined.io/api-docs/
https://github.com/package-url/purl-spec/blob/master/PURL-TYPES.rst

Correct ORT repository reference in README

There are multiple references to the ORT fork opossum-tool/oss-review-toolkit in the README.
As this fork is not accessible, and as ORT itself notes support for OpossumUI, I assume this is out of date information and the reference should instead be to oss-review-toolkit/ort?

Locations where this reference occurs:

Apologies if I'm misreading (or if the issue is considered noise, I could not find an issue template).

Green color should propagate upwards in tree

If all resources in a folder have an attribution the user shall get visible feedback about that.

Currently, there is no visual feedback for the user (ElectronBackend should be green):
Screenshot 2022-02-07 at 11 34 35

Important: check performance afterwards.

Show file tree in Attribution view

As a user I want to see the resource tree in the attribution view. This is to see where linked resources are located. Probably it should be "read only" and look similar to the resource tree in the Audit view.
Currently:
image

Replace pre-calculation of resourcesWithAttributedChildren

resourcesWithAttributedChildren have been introduced in AttributionData back at the time when we were showing down-arrows for signals and attributions (and were not virtualizing the tree). This data it is not optimized for the folder colors, and expensive to compute. Therefore:

  1. remove it
  2. if tree navigation becomes slower add new pre-computed data, that is:
    2.1 optimized to get the folder color,
    2.2 can be updated with cheap (aka fast) calculation if an attribution is created/deleted or a signal is hidden.

Log errors in frontend in backend log file

There is a log file for the electron backend. Include there error messages from the frontend.

Also see whether logging in backend in general could be improved. E.g. log file exports

Follow-up export triggered multiple times

Opening the example file opossum_input.json in OpossumUI and triggering the Follow-up export, the respective export is triggered multiple times
Screenshot 2022-07-18 at 16 35 19

Sometimes this causes the UI to be stuck in the following state:
Screenshot 2022-07-18 at 16 35 40

This behavior seems to have been introduced in #815

Change highlighting at breakpoints

If a user clicks on a directory in the Audit view, he expects the highlighting to "change"/stop at breakpoints or explicit attributions.
This should help to highlight what is / would be affected by an explicit attribution to the currently focused directory.
Now:
image
Should be:
image

Clean up project statistics popup

After we introduce new graphs, tables and styling on project statistic popup it would be nice to finalize and clean up:

  • check that styling is consistent
  • check if there are any bugs or any things to improve
  • refactor and clean up the code e.g. remove (or merge) duplicating styles...

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.