oss-review-toolkit / ort Goto Github PK

A suite of tools to automate software compliance checks.

Home Page: https://oss-review-toolkit.org

License: Apache License 2.0

Shell 2.22% Kotlin 87.09% Go 0.03% Python 0.34% Ruby 0.17% JavaScript 5.93% CSS 0.47% HTML 1.72% Dockerfile 0.35% Rust 0.01% FreeMarker 0.49% Scala 0.12% Emacs Lisp 0.01% Haskell 0.54% Roff 0.14% Java 0.01% Swift 0.22% Vim Snippet 0.13% Starlark 0.02% C++ 0.01%

license-scan package-scan package-manager dependencies dependency-graph license copyright spdx copyright-scan compliance

ort's Introduction

Introduction

The OSS Review Toolkit (ORT) is a FOSS policy automation and orchestration toolkit that you can use to manage your (open source) software dependencies in a strategic, safe and efficient manner.

You can use it to:

Generate CycloneDX, SPDX SBOMs, or custom FOSS attribution documentation for your software project
Automate your FOSS policy using risk-based Policy as Code to do licensing, security vulnerability, InnerSource and engineering standards checks for your software project and its dependencies
Create a source code archive for your software project and its dependencies to comply with certain licenses or have your own copy as nothing on the internet is forever
Correct package metadata or licensing findings yourself, using InnerSource or with the help of the FOSS community

ORT can be used as a library (for programmatic use), via a command line interface (for scripted use), or via its CI integrations. It consists of the following tools which can be combined into a highly customizable pipeline:

Analyzer: Determines the dependencies of projects and their metadata, abstracting which package managers or build systems are actually being used.
Downloader: Fetches all source code of the projects and their dependencies, abstracting which Version Control System (VCS) or other means are used to retrieve the source code.
Scanner: Uses configured source code scanners to detect license / copyright findings, abstracting the type of scanner.
Advisor: Retrieves security advisories for used dependencies from configured vulnerability data services.
Evaluator: Evaluates custom policy rules along with custom license classifications against the data gathered in preceding stages and returns a list of policy violations, e.g. to flag license findings.
Reporter: Presents results in various formats such as visual reports, Open Source notices or Bill-Of-Materials (BOMs) to easily identify dependencies, licenses, copyrights or policy rule violations.
Notifier: Sends result notifications via different channels (like emails and / or JIRA tickets).

Also see the list of related tools that help with running ORT.

Documentation

For detailed information, see the documentation on the ORT Website.

Installation

System requirements

ORT is being continuously used on Linux, Windows and macOS by the core development team, so these operating systems are considered to be well-supported.

To run the ORT binaries (also see Installation from binaries) at least Java 11 is required. Memory and CPU requirements vary depending on the size and type of project(s) to analyze / scan, but the general recommendation is to configure Java with 8 GiB of memory and to use a CPU with at least 4 cores.

# This will give the Java Virtual Machine 8GB Memory.
export JAVA_OPTS="$JAVA_OPTS -Xmx8g"

If ORT requires external tools to analyze a project, these tools are listed by the ort requirements command. If a package manager is not list listed there, support for it is integrated directly into ORT and does not require any external tools to be installed.

From binaries

Preliminary binary artifacts for ORT are currently available via JitPack. Please note that due to limitations with the JitPack build environment, the reporter is not able to create the Web App report.

From sources

Install the following basic prerequisites:

Git (any recent version will do).

Then clone this repository.

git clone https://github.com/oss-review-toolkit/ort
# If you intend to run tests, you have to clone the submodules too.
cd ort
git submodule update --init --recursive

Build using Docker

Install the following basic prerequisites:

Docker 18.09 or later (and ensure its daemon is running).
Enable BuildKit for Docker.

Change into the directory with ORT's source code and run docker build -t ort .. Alternatively, use the script at scripts/docker_build.sh which also sets the ORT version from the Git revision.

Build natively

Install these additional prerequisites:

Java Development Kit (JDK) version 11 or later; also remember to set the JAVA_HOME environment variable accordingly.

Change into the directory with ORT's source code and run ./gradlew installDist (on the first run this will bootstrap Gradle and download all required dependencies).

Basic usage

Depending on how ORT was installed, it can be run in the following ways:

If the Docker image was built, use
```
docker run ort --help
```
You can find further hints for using ORT with Docker in the documentation.
If the ORT distribution was built from sources, use
```
./cli/build/install/ort/bin/ort --help
```
If running directly from sources via Gradle, use
```
./gradlew cli:run --args="--help"
```
Note that in this case the working directory used by ORT is that of the cli project, not the directory gradlew is located in (see gradle/gradle#6074).

Want to Help or have Questions?

All contributions are welcome. If you are interested in contributing, please read our contributing guide. To get quick answers to any of your questions, we recommend you join our Slack community.

License

See the LICENSE file in the root of this project for license details.

OSS Review Toolkit (ORT) is a Linux Foundation project and part of ACT.

ort's People

Contributors

Stargazers

Watchers

Forkers

pombredanne jeffmcaffer jefshi88 haikoschol daniellandau adambhere enterstudio misha-codescoop romaniakovlev ddelbondio lenucksi architectureofthings dmitry-solomadin maxhbr tiandiyixian dl9pf annania andreas-bauer bosch-io tanzhongyi003 moo24 weshare-umigmbh pasacl97 flixx tuanpham-bosch olrudenk appurandheer heliocastro verifa cpandit201 snowladygirl2013 ddudnik unique-identifier craigrbarnes boschcrank roshats th3w4y benhe119 kchang264 oliverfendt uszeiss magneticz fb33 mercedes-benz abhaykatheria canvasslabs c123r modulexcite emanuelstanciu pietbrauer ajittk tohch4 yangchenghuang sthagen woznik vaeterchenfrost jjhhong zhernovs knsuzuki daniel-eder scanoss wdahman joshovi dlminvestments jagusztinl pctf bs-ondem yongicobay aalmann darthhater fossabot samuel-kerrien dejan2609 jeroenknoops goyalshivam7 rohan27s doubleopen-project codeakki weiling103 kukawski lcastillo-here korsin peterhsshin karanmarjara inno3 vbasem orquestradigital-actions nancheal hade telekom mrutkows sachinpatkar cybernetics adam-pateltech edulix jimjag bbroo ekmixon mbenziane huiwenhan

ort's Issues

analyzer: Add Bundler support

Add support for the Bundler dependency manager to the analyzer module.

http://bundler.io/

YARN 0.27 causes command version check in analyzer to fail

Here is the output from analyzer:

The following package managers are activated:
        Gradle, Maven, SBT, NPM, PIP
Scanning project path:
        /app
NPM projects found in:
        oss-attribution-builder/package.json
17:46:30.994 [main] INFO  com.here.ort.util.UtilsKt$log$1 - Running 'npm --version'...
17:46:30.999 [main] DEBUG com.here.ort.util.UtilsKt$log$1 - Keeping temporary files:
17:46:31.000 [main] DEBUG com.here.ort.util.UtilsKt$log$1 - /tmp/npm6011353000025587699.stdout
17:46:31.000 [main] DEBUG com.here.ort.util.UtilsKt$log$1 - /tmp/npm6832287346807849534.stderr
17:46:31.242 [main] INFO  com.here.ort.util.UtilsKt$log$1 - Running 'yarn --version'...
17:46:31.242 [main] DEBUG com.here.ort.util.UtilsKt$log$1 - Keeping temporary files:
17:46:31.243 [main] DEBUG com.here.ort.util.UtilsKt$log$1 - /tmp/yarn2289262920421819612.stdout
17:46:31.243 [main] DEBUG com.here.ort.util.UtilsKt$log$1 - /tmp/yarn792803821624333627.stderr
Exception in thread "main" com.vdurmont.semver4j.SemverException: Invalid version (no patch version): 0.27
        at com.vdurmont.semver4j.Semver.<init>(Semver.java:81)
        at com.vdurmont.semver4j.Semver.<init>(Semver.java:20)
        at com.vdurmont.semver4j.Requirement.isSatisfiedBy(Requirement.java:515)
        at com.here.ort.util.ProcessCaptureKt.checkCommandVersion(ProcessCapture.kt:117)
        at com.here.ort.util.ProcessCaptureKt.checkCommandVersion$default(ProcessCapture.kt:114)
        at com.here.ort.analyzer.managers.NPM.prepareResolution(NPM.kt:116)
        at com.here.ort.analyzer.PackageManager.resolveDependencies(PackageManager.kt:73)
        at com.here.ort.analyzer.Main$main$5.accept(Main.kt:247)
        at com.here.ort.analyzer.Main$main$5.accept(Main.kt:52)
        at java.util.LinkedHashMap.forEach(LinkedHashMap.java:684)
        at com.here.ort.analyzer.Main.main(Main.kt:245)

If I run yarn --version here is the output:

0.27

Gradle: Add warnings to the Tooling API model

Currently the Tooling API model classes only have fields for errors. Also add fields for warnings, because not everything reported there should be an error, for example the usage of flat dir repositories (see #251).

Add CONTRIBUTING.md

Lack of file makes it difficult to determine how ORT maintainers like to see contributions.

PIP: Process setup.py even when requirements.txt is present

Currently the pip package manager will be called with the first file it recognizes, and requirements.txt takes precedence over setup.py. That means the analyzer might not catch any of the metadata contained in setup.py. The implementation should be changed to always take setup.py into account.

Support the BndTools build system

In order to build OSGi bundles
http://bndtools.org/
can be used.

Not sure if you could scan bnd.bnd files to at least list possible dependencies as I assume analyzing its dependencies is not that easy as one can import raw JARs and bundle them in your target JAR.

Add CocoaPods (iOS) as supported package manager

Add support for the CocoaPods dependency manager to the analyzer module.

https://cocoapods.org/

Add Conan (C/C++) as supported package manager

Add support for the Conan dependency manager to the analyzer module.

https://conan.io/

Add LiD as supported license scanner

Add support for the LiD license scanner to the scanner module.

https://github.com/codeauroraforum/lid
(previously) https://source.codeaurora.org/external/qostg/lid/about/

PIP: Support "sanitized" project names

The Python Packaging Guide specifies what characters can be used in the name field in setup.py. For some illegal characters (e.g. spaces), setuptools does not throw an error, but replaces them with dashes during installation.

The analyzer uses pydep, which hooks setup() and captures it's keyword arguments verbatim. Therefore it fails when searching for the original name in the output of pipdeptree.

Integrate with Sonatype OSS Index for vulnerability reporting

See https://ossindex.sonatype.org/ and related resources:

yarn version is checked in the wrong way

Analyzer checks yarn's version using yarn --version while yarn expects yarn version.

Add Carthage (iOS/Cocoa) as supported package manager

Add support for the Carthage dependency manager to the analyzer module.

https://github.com/Carthage/Carthage

Gradle checks still fail after following installation instructions

I'll just dump the log here.

Tests should skip if they rely on external tools. Installation of external tools is not documented.
Internal tools should be built. It's unclear whether they are, or are external tools. licensee in particular appears to come from a submodule, but is not built.
See log for more

$ ./gradlew check
Starting a Gradle Daemon (subsequent builds will be faster)
Parallel execution with configuration on demand is an incubating feature.

> Task :utils-test:detektCheck 
. :utils:detektCheck
> Task :graph:detektCheck 
. :analyzer:detektCheck
> Task :utils-test:detektCheck 


1 kotlin files were analyzed.

Ruleset: comments
Ruleset: complexity
Ruleset: empty-blocks
Ruleset: exceptions
Ruleset: performance
Ruleset: potential-bugs
Ruleset: style

Complexity Report:
        - 27 lines of code (loc)
        - 4 source lines of code (sloc)
        - 2 logical lines of code (lloc)
        - 18 comment lines of code (cloc)
        - 0 McCabe complexity (mcc)
        - 0 number of total code smells
        - 450 % comment source ratio
        - 0 mcc per 1000 lloc
        - 0 code smells per 1000 lloc

Project Statistics:
        - number of properties: 0
        - number of functions: 0
        - number of classes: 0
        - number of packages: 1
        - number of kt files: 1


detekt finished in 1958 ms.

> Task :graph:detektCheck 


1 kotlin files were analyzed.

Ruleset: comments
Ruleset: complexity
Ruleset: empty-blocks
Ruleset: exceptions
Ruleset: performance
Ruleset: potential-bugs
Ruleset: style

Complexity Report:
        - 160 lines of code (loc)
        - 104 source lines of code (sloc)
        - 74 logical lines of code (lloc)
        - 27 comment lines of code (cloc)
        - 13 McCabe complexity (mcc)
        - 0 number of total code smells
        - 25 % comment source ratio
        - 175 mcc per 1000 lloc
        - 0 code smells per 1000 lloc

Project Statistics:
        - number of properties: 16
        - number of functions: 3
        - number of classes: 0
        - number of packages: 1
        - number of kt files: 1


detekt finished in 2393 ms.

> Task :downloader:detektCheck 
.......
> Task :model:detektCheck 
.................

17 kotlin files were analyzed.

Ruleset: comments
Ruleset: complexity
Ruleset: empty-blocks
Ruleset: exceptions
Ruleset: performance
Ruleset: potential-bugs
Ruleset: style

Complexity Report:
        - 1705 lines of code (loc)
        - 848 source lines of code (sloc)
        - 650 logical lines of code (lloc)
        - 648 comment lines of code (cloc)
        - 63 McCabe complexity (mcc)
        - 0 number of total code smells
        - 76 % comment source ratio
        - 96 mcc per 1000 lloc
        - 0 code smells per 1000 lloc

Project Statistics:
        - number of properties: 64
        - number of functions: 28
        - number of classes: 31
        - number of packages: 1
        - number of kt files: 17


detekt finished in 2868 ms.

> Task :analyzer:detektCheck 
........
> Task :downloader:detektCheck 
..............

21 kotlin files were analyzed.

Ruleset: comments
Ruleset: complexity
Ruleset: empty-blocks
Ruleset: exceptions
Ruleset: performance
Ruleset: potential-bugs
Ruleset: style

Complexity Report:
        - 3313 lines of code (loc)
        - 2189 source lines of code (sloc)
        - 1553 logical lines of code (lloc)
        - 588 comment lines of code (cloc)
        - 277 McCabe complexity (mcc)
        - 0 number of total code smells
        - 26 % comment source ratio
        - 178 mcc per 1000 lloc
        - 0 code smells per 1000 lloc

Project Statistics:
        - number of properties: 295
        - number of functions: 93
        - number of classes: 29
        - number of packages: 2
        - number of kt files: 21


detekt finished in 3201 ms.

> Task :utils:detektCheck 
........

8 kotlin files were analyzed.

> Task :scanner:detektCheck 
........oader:compileKotlin
> Task :utils:detektCheck 

Ruleset: comments
Ruleset: complexity
Ruleset: empty-blocks
Ruleset: exceptions
Ruleset: performance
Ruleset: potential-bugs
Ruleset: style

Complexity Report:
        - 1414 lines of code (loc)
        - 851 source lines of code (sloc)
        - 594 logical lines of code (lloc)
        - 349 comment lines of code (cloc)
        - 128 McCabe complexity (mcc)
        - 0 number of total code smells
        - 41 % comment source ratio
        - 215 mcc per 1000 lloc
        - 0 code smells per 1000 lloc

Project Statistics:
        - number of properties: 124
        - number of functions: 36
        - number of classes: 4
        - number of packages: 2
        - number of kt files: 8


detekt finished in 2778 ms.

> Task :analyzer:detektCheck 
........................

32 kotlin files were analyzed.

Ruleset: comments
Ruleset: complexity
Ruleset: empty-blocks
Ruleset: exceptions
Ruleset: performance
Ruleset: potential-bugs
Ruleset: style

Complexity Report:
        - 4263 lines of code (loc)
        - 2651 source lines of code (sloc)
        - 1585 logical lines of code (lloc)
        - 902 comment lines of code (cloc)
        - 358 McCabe complexity (mcc)
        - 0 number of total code smells
        - 34 % comment source ratio
        - 225 mcc per 1000 lloc
        - 0 code smells per 1000 lloc

Project Statistics:
        - number of properties: 460
        - number of functions: 116
        - number of classes: 36
        - number of packages: 6
        - number of kt files: 32


detekt finished in 3429 ms.

> Task :scanner:detektCheck 
.....

13 kotlin files were analyzed.

Ruleset: comments
Ruleset: complexity
Ruleset: empty-blocks
Ruleset: exceptions
Ruleset: performance
Ruleset: potential-bugs
Ruleset: style

Complexity Report:
        - 1657 lines of code (loc)
        - 1022 source lines of code (sloc)
        - 730 logical lines of code (lloc)
        - 356 comment lines of code (cloc)
        - 169 McCabe complexity (mcc)
        - 0 number of total code smells
        - 34 % comment source ratio
        - 231 mcc per 1000 lloc
        - 0 code smells per 1000 lloc

Project Statistics:
        - number of properties: 145
        - number of functions: 45
        - number of classes: 16
        - number of packages: 2
        - number of kt files: 13


detekt finished in 3997 ms.

> Task :downloader:test 

com.here.ort.downloader.vcs.GitTest > GitTest.Detected Git version is not empty STARTED

com.here.ort.downloader.vcs.GitTest > GitTest.Detected Git version is not empty PASSED

com.here.ort.downloader.vcs.GitTest > GitTest.Git detects non-working-trees STARTED

com.here.ort.downloader.vcs.GitTest > GitTest.Git detects non-working-trees PASSED

com.here.ort.downloader.vcs.GitTest > GitTest.Git correctly detects URLs to remote repositories STARTED

> Task :analyzer:test 

com.here.ort.analyzer.MergedResultsBuilderTest > MergedResultsBuilderTest.MergedResultsBuilder merges results from all files STARTED

com.here.ort.analyzer.MergedResultsBuilderTest > MergedResultsBuilderTest.MergedResultsBuilder merges results from all files PASSED

com.here.ort.analyzer.YamlFilePackageCurationProviderTest > YamlFilePackageCurationProviderTest.Provider can read YAML file STARTED

com.here.ort.analyzer.YamlFilePackageCurationProviderTest > YamlFilePackageCurationProviderTest.Provider can read YAML file PASSED

com.here.ort.analyzer.YamlFilePackageCurationProviderTest > YamlFilePackageCurationProviderTest.Provider returns only matching curations STARTED

com.here.ort.analyzer.YamlFilePackageCurationProviderTest > YamlFilePackageCurationProviderTest.Provider returns only matching curations PASSED

com.here.ort.utils.NpmTest > expandShortcutURL.should do nothing for empty URLs STARTED

com.here.ort.utils.NpmTest > expandShortcutURL.should do nothing for empty URLs PASSED

com.here.ort.utils.NpmTest > expandShortcutURL.should properly handle NPM shortcut URLs STARTED

com.here.ort.utils.NpmTest > expandShortcutURL.should properly handle NPM shortcut URLs PASSED

com.here.ort.util.PackageManagerTest > findManagedFiles.should find all managed files STARTED

com.here.ort.util.PackageManagerTest > findManagedFiles.should find all managed files PASSED

com.here.ort.util.PackageManagerTest > findManagedFiles.should find only files for active package managers STARTED

com.here.ort.util.PackageManagerTest > findManagedFiles.should find only files for active package managers PASSED

com.here.ort.util.PackageManagerTest > findManagedFiles.should find no files if no package managers are active STARTED

com.here.ort.util.PackageManagerTest > findManagedFiles.should find no files if no package managers are active PASSED

com.here.ort.util.PackageManagerTest > findManagedFiles.should fail if the provided file is not a directory STARTED

com.here.ort.util.PackageManagerTest > findManagedFiles.should fail if the provided file is not a directory PASSED

> Task :downloader:test 

com.here.ort.downloader.vcs.GitTest > GitTest.Git correctly detects URLs to remote repositories PASSED

com.here.ort.downloader.vcs.GitTest > GitTest.Detected Git working tree information is correct STARTED

com.here.ort.downloader.vcs.GitTest > GitTest.Detected Git working tree information is correct PASSED

com.here.ort.downloader.vcs.GitTest > GitTest.Git correctly lists remote tags STARTED

com.here.ort.downloader.vcs.GitTest > GitTest.Git correctly lists remote tags PASSED

> Task :scanner:funTest 

com.here.ort.scanner.HttpCacheTest > HttpCacheTest.HTTP GET returns what was PUT STARTED

com.here.ort.scanner.HttpCacheTest > HttpCacheTest.HTTP GET returns what was PUT PASSED

com.here.ort.scanner.ScanPathTest > ScanPathTest.BoyterLc recognizes our own LICENSE STARTED

> Task :downloader:test 

com.here.ort.downloader.vcs.CvsTest > CvsTest.Detected CVS version is not empty STARTED

com.here.ort.downloader.vcs.CvsTest > CvsTest.Detected CVS version is not empty SKIPPED

com.here.ort.downloader.vcs.CvsTest > CvsTest.CVS detects non-working-trees STARTED

com.here.ort.downloader.vcs.CvsTest > CvsTest.CVS detects non-working-trees SKIPPED

com.here.ort.downloader.vcs.CvsTest > CvsTest.CVS correctly detects URLs to remote repositories STARTED

com.here.ort.downloader.vcs.CvsTest > CvsTest.CVS correctly detects URLs to remote repositories SKIPPED

com.here.ort.downloader.vcs.CvsTest > CvsTest.Detected CVS working tree information is correct STARTED

com.here.ort.downloader.vcs.CvsTest > CvsTest.Detected CVS working tree information is correct SKIPPED

com.here.ort.downloader.vcs.CvsTest > CvsTest.CVS correctly lists remote tags STARTED

com.here.ort.downloader.vcs.CvsTest > CvsTest.CVS correctly lists remote tags SKIPPED

com.here.ort.downloader.MercurialTest > MercurialTest.Detected Mercurial version is not empty STARTED

com.here.ort.downloader.MercurialTest > MercurialTest.Detected Mercurial version is not empty SKIPPED

com.here.ort.downloader.MercurialTest > MercurialTest.Mercurial detects non-working-trees STARTED

com.here.ort.downloader.MercurialTest > MercurialTest.Mercurial detects non-working-trees SKIPPED

com.here.ort.downloader.MercurialTest > MercurialTest.Mercurial correctly detects URLs to remote repositories STARTED

com.here.ort.downloader.MercurialTest > MercurialTest.Mercurial correctly detects URLs to remote repositories SKIPPED

com.here.ort.downloader.MercurialTest > MercurialTest.Detected Mercurial working tree information is correct STARTED

com.here.ort.downloader.MercurialTest > MercurialTest.Detected Mercurial working tree information is correct SKIPPED

com.here.ort.downloader.MercurialTest > MercurialTest.Mercurial correctly lists remote tags STARTED

com.here.ort.downloader.MercurialTest > MercurialTest.Mercurial correctly lists remote tags SKIPPED

com.here.ort.downloader.SubversionTest > SubversionTest.Detected Subversion version is not empty STARTED

com.here.ort.downloader.SubversionTest > SubversionTest.Detected Subversion version is not empty PASSED

com.here.ort.downloader.SubversionTest > SubversionTest.Subversion detects non-working-trees STARTED

com.here.ort.downloader.SubversionTest > SubversionTest.Subversion detects non-working-trees PASSED

com.here.ort.downloader.SubversionTest > SubversionTest.Subversion correctly detects URLs to remote repositories STARTED

> Task :analyzer:funTest 

com.here.ort.analyzer.BabelTest > Babel dependencies.should be correctly analyzed STARTED

com.here.ort.analyzer.BabelTest > Babel dependencies.should be correctly analyzed FAILED
    java.io.IOException: Cannot run program "npm": error=2, No such file or directory

        Caused by:
        java.io.IOException: error=2, No such file or directory

com.here.ort.analyzer.MainTest > MainTest.Activating only Gradle works STARTED

> Task :scanner:funTest 

com.here.ort.scanner.ScanPathTest > ScanPathTest.BoyterLc recognizes our own LICENSE PASSED

com.here.ort.scanner.ScanPathTest > ScanPathTest.Licensee recognizes our own LICENSE STARTED

com.here.ort.scanner.ScanPathTest > ScanPathTest.Licensee recognizes our own LICENSE FAILED
    java.io.IOException: Cannot run program "licensee" (in directory ".."): error=2, No such file or directory

        Caused by:
        java.io.IOException: error=2, No such file or directory

com.here.ort.scanner.ScanPathTest > ScanPathTest.ScanCode recognizes our own LICENSE STARTED

> Task :downloader:test 

com.here.ort.downloader.SubversionTest > SubversionTest.Subversion correctly detects URLs to remote repositories PASSED

com.here.ort.downloader.SubversionTest > SubversionTest.Detected Subversion working tree information is correct STARTED

com.here.ort.downloader.SubversionTest > SubversionTest.Detected Subversion working tree information is correct PASSED

com.here.ort.downloader.SubversionTest > SubversionTest.Subversion correctly lists remote tags STARTED

> Task :analyzer:funTest 

com.here.ort.analyzer.MainTest > MainTest.Activating only Gradle works PASSED

com.here.ort.analyzer.MainTest > MainTest.Activating only NPM works STARTED

com.here.ort.analyzer.MainTest > MainTest.Activating only NPM works FAILED
    java.io.IOException: Cannot run program "npm": error=2, No such file or directory

        Caused by:
        java.io.IOException: error=2, No such file or directory

com.here.ort.analyzer.MainTest > MainTest.Merging into single results file creates correct output STARTED

com.here.ort.analyzer.MainTest > MainTest.Merging into single results file creates correct output PASSED

com.here.ort.analyzer.MainTest > MainTest.Package curation data file is applied correctly STARTED

com.here.ort.analyzer.MainTest > MainTest.Package curation data file is applied correctly PASSED

com.here.ort.analyzer.NpmTest > NPM should.resolve shrinkwrap dependencies correctly STARTED

com.here.ort.analyzer.NpmTest > NPM should.resolve shrinkwrap dependencies correctly FAILED
    java.io.IOException: Cannot run program "npm": error=2, No such file or directory

        Caused by:
        java.io.IOException: error=2, No such file or directory

com.here.ort.analyzer.NpmTest > NPM should.resolve package-lock dependencies correctly STARTED

com.here.ort.analyzer.NpmTest > NPM should.resolve package-lock dependencies correctly FAILED
    java.io.IOException: Cannot run program "npm": error=2, No such file or directory

        Caused by:
        java.io.IOException: error=2, No such file or directory

com.here.ort.analyzer.NpmTest > NPM should.show error if no lockfile is present STARTED

com.here.ort.analyzer.NpmTest > NPM should.show error if no lockfile is present FAILED
    java.io.IOException: Cannot run program "npm": error=2, No such file or directory

        Caused by:
        java.io.IOException: error=2, No such file or directory

com.here.ort.analyzer.NpmTest > NPM should.show error if multiple lockfiles are present STARTED

com.here.ort.analyzer.NpmTest > NPM should.show error if multiple lockfiles are present FAILED
    java.io.IOException: Cannot run program "npm": error=2, No such file or directory

        Caused by:
        java.io.IOException: error=2, No such file or directory

com.here.ort.analyzer.NpmTest > NPM should.resolve dependencies even if the node_modules directory already exists STARTED

com.here.ort.analyzer.NpmTest > NPM should.resolve dependencies even if the node_modules directory already exists FAILED
    java.io.IOException: Cannot run program "npm": error=2, No such file or directory

        Caused by:
        java.io.IOException: error=2, No such file or directory

com.here.ort.analyzer.NpmTest > yarn should.resolve dependencies correctly STARTED

com.here.ort.analyzer.NpmTest > yarn should.resolve dependencies correctly FAILED
    java.io.IOException: Cannot run program "npm": error=2, No such file or directory

        Caused by:
        java.io.IOException: error=2, No such file or directory

com.here.ort.analyzer.PipTest > setup py dependencies.should be resolved correctly for spdx-tools-python STARTED

> Task :downloader:test 

com.here.ort.downloader.SubversionTest > SubversionTest.Subversion correctly lists remote tags PASSED

com.here.ort.downloader.VersionControlSystemTest > For an absolute working directory, getPathToRoot  .should work if given absolute paths STARTED

com.here.ort.downloader.VersionControlSystemTest > For an absolute working directory, getPathToRoot  .should work if given absolute paths PASSED

com.here.ort.downloader.VersionControlSystemTest > For an absolute working directory, getPathToRoot  .should work if given relative paths STARTED

com.here.ort.downloader.VersionControlSystemTest > For an absolute working directory, getPathToRoot  .should work if given relative paths PASSED

com.here.ort.downloader.VersionControlSystemTest > For a relative working directory, getPathToRoot  .should work if given absolute paths STARTED

com.here.ort.downloader.VersionControlSystemTest > For a relative working directory, getPathToRoot  .should work if given absolute paths PASSED

com.here.ort.downloader.VersionControlSystemTest > For a relative working directory, getPathToRoot  .should work if given relative paths STARTED

com.here.ort.downloader.VersionControlSystemTest > For a relative working directory, getPathToRoot  .should work if given relative paths PASSED

com.here.ort.downloader.VersionControlSystemTest > splitUrl for Bitbucket.should not modify URLs without a path STARTED

com.here.ort.downloader.VersionControlSystemTest > splitUrl for Bitbucket.should not modify URLs without a path SKIPPED

com.here.ort.downloader.VersionControlSystemTest > splitUrl for Bitbucket.should split tree URLs STARTED

com.here.ort.downloader.VersionControlSystemTest > splitUrl for Bitbucket.should split tree URLs PASSED

com.here.ort.downloader.VersionControlSystemTest > splitUrl for Bitbucket.should split blob URLs STARTED

com.here.ort.downloader.VersionControlSystemTest > splitUrl for Bitbucket.should split blob URLs PASSED

com.here.ort.downloader.VersionControlSystemTest > splitUrl for GitHub.should not modify URLs without a path STARTED

com.here.ort.downloader.VersionControlSystemTest > splitUrl for GitHub.should not modify URLs without a path PASSED

com.here.ort.downloader.VersionControlSystemTest > splitUrl for GitHub.should not fail for a user called blob or a project called tree STARTED

com.here.ort.downloader.VersionControlSystemTest > splitUrl for GitHub.should not fail for a user called blob or a project called tree PASSED

com.here.ort.downloader.VersionControlSystemTest > splitUrl for GitHub.should split tree URLs STARTED

com.here.ort.downloader.VersionControlSystemTest > splitUrl for GitHub.should split tree URLs PASSED

com.here.ort.downloader.VersionControlSystemTest > splitUrl for GitHub.should split blob URLs STARTED

com.here.ort.downloader.VersionControlSystemTest > splitUrl for GitHub.should split blob URLs PASSED

com.here.ort.downloader.VersionControlSystemTest > splitUrl for GitHub.should split extra path components STARTED

com.here.ort.downloader.VersionControlSystemTest > splitUrl for GitHub.should split extra path components PASSED

com.here.ort.downloader.VersionControlSystemTest > splitUrl for GitLab.should not modify URLs without a path STARTED

com.here.ort.downloader.VersionControlSystemTest > splitUrl for GitLab.should not modify URLs without a path PASSED

com.here.ort.downloader.VersionControlSystemTest > splitUrl for GitLab.should split tree URLs STARTED

com.here.ort.downloader.VersionControlSystemTest > splitUrl for GitLab.should split tree URLs PASSED

com.here.ort.downloader.VersionControlSystemTest > splitUrl for GitLab.should split blob URLs STARTED

com.here.ort.downloader.VersionControlSystemTest > splitUrl for GitLab.should split blob URLs PASSED

> Task :downloader:funTest 

com.here.ort.downloader.BabelTest > BabelTest.Babel packages should be correctly downloaded STARTED

> Task :analyzer:funTest 

com.here.ort.analyzer.PipTest > setup py dependencies.should be resolved correctly for spdx-tools-python PASSED

com.here.ort.analyzer.PipTest > requirements txt dependencies.should be resolved correctly for example-python-flask STARTED

> Task :downloader:funTest 

com.here.ort.downloader.BabelTest > BabelTest.Babel packages should be correctly downloaded PASSED

com.here.ort.downloader.DirectoryTest > DirectoryTest.Creates directories for Gradle submodules STARTED

com.here.ort.downloader.DirectoryTest > DirectoryTest.Creates directories for Gradle submodules PASSED

com.here.ort.downloader.DownloaderTest > DownloaderTest.Downloads and unpacks JAR source package STARTED

com.here.ort.downloader.DownloaderTest > DownloaderTest.Downloads and unpacks JAR source package PASSED

com.here.ort.downloader.DownloaderTest > DownloaderTest.Download of JAR source package fails when hash is incorrect STARTED

com.here.ort.downloader.DownloaderTest > DownloaderTest.Download of JAR source package fails when hash is incorrect PASSED

com.here.ort.downloader.DownloaderTest > DownloaderTest.Falls back to downloading source package when download from VCS fails STARTED

com.here.ort.downloader.DownloaderTest > DownloaderTest.Falls back to downloading source package when download from VCS fails PASSED

com.here.ort.downloader.DownloaderTest > DownloaderTest.Can download source artifact from SourceForce STARTED

com.here.ort.downloader.DownloaderTest > DownloaderTest.Can download source artifact from SourceForce PASSED

com.here.ort.downloader.BeanUtilsTest > BeanUtilsTest.BeanUtils SVN tag should be correctly downloaded STARTED

> Task :analyzer:funTest 

com.here.ort.analyzer.PipTest > requirements txt dependencies.should be resolved correctly for example-python-flask PASSED

com.here.ort.analyzer.SbtTest > Dependencies of the.external 'directories' project should be detected correctly STARTED

com.here.ort.analyzer.SbtTest > Dependencies of the.external 'directories' project should be detected correctly FAILED
    java.io.IOException: Cannot run program "sbt" (in directory "src/funTest/assets/projects/external/directories"): error=2, No such file or directory

        Caused by:
        java.io.IOException: error=2, No such file or directory

com.here.ort.analyzer.GradleTest > GradleTest.Root project dependencies are detected correctly STARTED

com.here.ort.analyzer.GradleTest > GradleTest.Root project dependencies are detected correctly PASSED

com.here.ort.analyzer.GradleTest > GradleTest.Project dependencies are detected correctly STARTED

com.here.ort.analyzer.GradleTest > GradleTest.Project dependencies are detected correctly PASSED

com.here.ort.analyzer.GradleTest > GradleTest.External dependencies are detected correctly STARTED

com.here.ort.analyzer.GradleTest > GradleTest.External dependencies are detected correctly PASSED

com.here.ort.analyzer.GradleTest > GradleTest.Unresolved dependencies are detected correctly STARTED

com.here.ort.analyzer.GradleTest > GradleTest.Unresolved dependencies are detected correctly PASSED

com.here.ort.analyzer.GradleTest > GradleTest.Fails nicely for Gradle version < 3.3 STARTED

> Task :downloader:funTest 

com.here.ort.downloader.BeanUtilsTest > BeanUtilsTest.BeanUtils SVN tag should be correctly downloaded PASSED

com.here.ort.downloader.vcs.GitDownloadTest > GitDownloadTest.Git can download a given revision STARTED

com.here.ort.downloader.vcs.GitDownloadTest > GitDownloadTest.Git can download a given revision PASSED

com.here.ort.downloader.vcs.GitDownloadTest > GitDownloadTest.Git can download only a single path STARTED

com.here.ort.downloader.vcs.GitDownloadTest > GitDownloadTest.Git can download only a single path PASSED

com.here.ort.downloader.vcs.GitDownloadTest > GitDownloadTest.Git can download based on a version STARTED

com.here.ort.downloader.vcs.GitDownloadTest > GitDownloadTest.Git can download based on a version PASSED

com.here.ort.downloader.vcs.GitDownloadTest > GitDownloadTest.Git can download only a single path based on a version STARTED

com.here.ort.downloader.vcs.GitDownloadTest > GitDownloadTest.Git can download only a single path based on a version PASSED

com.here.ort.downloader.vcs.CvsDownloadTest > CvsDownloadTest.CVS can download a given revision STARTED

com.here.ort.downloader.vcs.CvsDownloadTest > CvsDownloadTest.CVS can download a given revision SKIPPED

com.here.ort.downloader.vcs.CvsDownloadTest > CvsDownloadTest.CVS can download only a single path STARTED

com.here.ort.downloader.vcs.CvsDownloadTest > CvsDownloadTest.CVS can download only a single path SKIPPED

com.here.ort.downloader.vcs.CvsDownloadTest > CvsDownloadTest.CVS can download based on a version STARTED

com.here.ort.downloader.vcs.CvsDownloadTest > CvsDownloadTest.CVS can download based on a version SKIPPED

com.here.ort.downloader.vcs.CvsDownloadTest > CvsDownloadTest.CVS can download only a single path based on a version STARTED

com.here.ort.downloader.vcs.CvsDownloadTest > CvsDownloadTest.CVS can download only a single path based on a version SKIPPED

com.here.ort.downloader.vcs.MercurialDownloadTest > MercurialDownloadTest.Mercurial can download a given revision STARTED

com.here.ort.downloader.vcs.MercurialDownloadTest > MercurialDownloadTest.Mercurial can download a given revision SKIPPED

com.here.ort.downloader.vcs.MercurialDownloadTest > MercurialDownloadTest.Mercurial can download only a single path STARTED

com.here.ort.downloader.vcs.MercurialDownloadTest > MercurialDownloadTest.Mercurial can download only a single path SKIPPED

com.here.ort.downloader.vcs.MercurialDownloadTest > MercurialDownloadTest.Mercurial can download based on a version STARTED

com.here.ort.downloader.vcs.MercurialDownloadTest > MercurialDownloadTest.Mercurial can download based on a version SKIPPED

com.here.ort.downloader.vcs.MercurialDownloadTest > MercurialDownloadTest.Mercurial can download only a single path based on a version STARTED

com.here.ort.downloader.vcs.MercurialDownloadTest > MercurialDownloadTest.Mercurial can download only a single path based on a version SKIPPED

com.here.ort.downloader.vcs.SubversionDownloadTest > SubversionDownloadTest.Subversion can download a given revision STARTED

> Task :analyzer:funTest 

com.here.ort.analyzer.GradleTest > GradleTest.Fails nicely for Gradle version < 3.3 PASSED

com.here.ort.analyzer.GradleTest > GradleTest.Is compatible with Gradle >= 3.3 STARTED

com.here.ort.analyzer.GradleTest > GradleTest.Is compatible with Gradle >= 3.3 SKIPPED

com.here.ort.analyzer.MavenTest > MavenTest.jgnash parent dependencies are detected correctly STARTED

com.here.ort.analyzer.MavenTest > MavenTest.jgnash parent dependencies are detected correctly PASSED

com.here.ort.analyzer.MavenTest > MavenTest.jgnash-core dependencies are detected correctly STARTED

> Task :scanner:funTest 

com.here.ort.scanner.ScanPathTest > ScanPathTest.ScanCode recognizes our own LICENSE PASSED

> Task :downloader:funTest 

com.here.ort.downloader.vcs.SubversionDownloadTest > SubversionDownloadTest.Subversion can download a given revision FAILED
    com.here.ort.downloader.DownloadException: Subversion failed to download from URL 'https://svn.code.sf.net/p/sendmessage/code'.

        Caused by:
        java.io.IOException: Running 'svn checkout https://svn.code.sf.net/p/sendmessage/code --depth empty .' in directory '/tmp/tmp2273889995595760709.tmp' failed with exit code 1:
        svn: E170013: Unable to connect to a repository at URL 'https://svn.code.sf.net/p/sendmessage/code'
        svn: E175002: Unexpected HTTP status 504 'Gateway Time-out' on '/p/sendmessage/code'

com.here.ort.downloader.vcs.SubversionDownloadTest > SubversionDownloadTest.Subversion can download only a single path STARTED
^C

scanner: Limit scans to specific dependency scopes

Add a parameter to the scanner module to select which scopes will be scanned.

No license and copyright information in the files

Since this is an open Source compliance toolkit, I would love to see the best practises for a good OSS project realized in this project.
Some of the best practises are:

Put a copyrihgt notice in each file
Put the relevant Strandard License Header in file (or a statement like This file is licensed under --SPDX Short Identifier --

The FSFE hast set up a Project with best practizes relevant for smart license compliance support, the proejct is called reuse. Please see https://reuse.software/
No file of the current release carries a standarde license haeder, there is no license information in the files at all. The same applies to the copyright information, except the README.md file.

Regarding the README.md file, please remove the statement "See the LICENSE file in the root of this project for license details." because no license scanner can "auto conclude" such a statement.

The file oss-review-toolkit-preview/analyzer/src/funTest/assets/projects/synthetic/mixed/NPMNestedMaven/package.json
{
"name": "npmnestedmaven",
"version": "1.0.0",
"description": "A dummy NPM project with a nested Maven project",
"scripts": {
"test": "echo "Error: no test specified" && exit 1"
},
"author": "Sebastian Schuberth",
"license": "ISC",
"repository": {
"type": "git",
"url": "https://github.com/heremaps/oss-review-toolkit.git"
}
}
Carries a license information which doea not fit. Even in case of test files you should care for license hygiene

Add support for darcs VCS

See http://darcs.net/, which seems to be popular in the Haskell community.

downloader: Add Mercurial support

Add support for Mercurial repositories to the downloader module.

Add NuGet (.Net) as supported package manager

Add support for the NuGet dependency manager to the analyzer module.

https://www.nuget.org/

Consider using a "purl" as the package identifier

Consider redesigning the package identifier as a "puurl" as described at nexB/scancode-toolkit#805 (comment).

analyzer: Add PHP Composer support

Add support for the PHP Composer package manger to the analyer module.

https://getcomposer.org/

PIP: Python 3 Support

The current Python/pip implementation does not try to determine which Python version to use for the project being analyzed and instead always uses Python 2.7.

There are ways to specify supported Python versions in both setup.py and requirements.txt files (as well as tox.ini and other files commonly found in Python projects). The analyzer could use that information to decide which version of Python/pip to invoke for gathering information about dependencies.

Broken links in README.md

There are broken links to the subprojects analyzer, graph, downloader, and scanner in the README file. Also, the link to the license file isn't correct.

PIP: Update PyPI API endpoint

PyPI recently moved from pypi.python.org to pypi.org. When querying the API for package metadata, we should use this domain instead of relying on working HTTP redirects.

Add AWS S3 as supported remote cache

For re-using already known scan results, ORT can use a remote cache. Currently only Artifactory is supported but we would like to add AWS S3 support as it more commonly used and cheaper to set up than Artifactory instance.

NPM: Add support for optional dependencies

Add support for the optionalDependencies scope to NPM. Make sure that the report contains a warning in case an optional dependency could not be installed.

Add Cargo (Rust) as supported package manager

Add support for the Cargo dependency manager to the analyzer module.

https://github.com/rust-lang/cargo
https://doc.rust-lang.org/cargo/

Merge output of analyzer in single YAML file

Currently the analyzer creates one YAML file per defintion file. These files different files can be hard to handle, and also can contain a lot of duplication in the package list.

Add an option to the analyzer to create a merged YAML file that contains the results for all definition files. The structure should be:

Repository: Description of the scanned repository, with VCS information if available.
Projects: List of the projects, as they appear in the individual analyzer results. Needs to reference the path of the definition file it describes.
Packages: A list of all packages, merged from the package lists of the individual analyzer results.

Add BitBake (Yocto) as supported package manager

Add support for the BitBake (Yocto) dependency manager to the analyzer module.

Links:

BitBake manual: https://www.yoctoproject.org/docs/1.6/bitbake-user-manual/bitbake-user-manual.html
Useful Bitbake commands: https://community.nxp.com/docs/DOC-94953

bitbake-layers show-layers
bitbake-layers show-recipes

Bitbake repos: https://git.yoctoproject.org/
Projects starting with poky-* and meta-* likely contain bitbake files

downloader: Add CVS support

Add support for CVS repositories to the downloader module.

Am I right in thinking that analyser is trusting package-manager metadata?

In general I've been advocating building everything from source, so that we can be sure that we are working with known components (and extra points for verifying bit-for-bit reproducibility).

If we trust the package managers there are at least a couple of risks:

the metadata is wrong (someone forgot to bump version information, or similar)
the content is wrong (maybe it's been compromised, or we've been pointed at a fake url)

Add support for Hex packages

Add support for Hex packages (projects written in Elixir and Erlang) to the ORT analyzer module.

About Hex:

Hex is a package manager for the BEAM ecosystem, any language that compiles to run on the BEAM VM, such as Elixir and Erlang, can be used to build Hex packages.... Many build tools support Hex packages including Mix for Elixir projects, rebar and erlang.mk for Erlang projects.

However, the use of the build tools is not as clear-cut as this, see also Requirements 1 below. In general, a Hex package can use the following build tools:

Mix
rebar3
rebar3 with erlang.mk
both Mix and rebar3

Requirements

Detect whether a project is a Hex project and whether it uses Mix or rebar

Mix:

The presence of mix.exs and/or mix.lock may indicate that the projext build tool is Mix

rebar:

The presence of rebar.config, rebar.lock, and/or src/.app.src file makes it likely the project build tool is rebar3.

We could use the above files for the globsForDefinitionFiles

However, the use of the build tools is not as clean-cut as this. For example, the project ssl_verify_fun makes use of both Mix and rebar3.

Detect declared license for a package

Mix:

Licenses can be declared in the mix.exs see https://github.com/semlabs/kudos/blob/c36f23760992793962fd90aabf0630ccbd6e7530/mix.exs#L41

Is this a standard? Mix docs do not provide an answer.

rebar3:

Licenses can be declared in [src/.app.src file](https://hex.pm/docs/rebar3_publish#adding-metadata-to-code-clas (and Mix/rebar3) sinlinesrcltmyappgtappsrccode)

Get dependency tree including package names, versions

Options:

rebar3:

use rebar3 deps or rebar3 pkgs, but note there is no machine-readable output:

→ rebar3 deps
...
cowboy* (package)
recon* (git source)
erlware_commons (locked git source)
getopt* (locked git source)
providers (locked hg source)
relx (locked git source)Erlang have scopes

→ rebar3 tree
...
|- bootstrap-0.0.2 (git repo)
|- dirmon-0.1.0 (project app)any 
|- file_monitor-0.1 (git repo)
|- peeranha-0.1.0 (git repo)
|  |- gproc-git (git repo)
|  |- interclock-0.1.2 (git repo)
|  |  |- bitcask-1.7.0 (git repo)
|  |  |  |- lager-2.1.1 (hex package)
|  |  |  |  |- goldrush-0.1.6 (hex package)
|  |  |- itc-1.0.0 (git repo)
|  |- merklet-1.0.0 (git repo)
|- recon-2.2.2 (git repo) on which to test Mix/rebar3 support
|- uuid-1.5.0 (git repo)
|  |- quickrand-1.5.0 (git repo)

Documentation:

Mix:

install and use licensir -- lists all licenses (Mix projects only)

(or) write a script (in Erlang?) to loop over definition files and use Mix.Dep API (see https://github.com/semlabs/kudos/blob/master/lib/kudos.ex#L83-L100), but generate JSON

https://repo.hex.pm???

Obtain the source for a dependency

How can ORT's downloader download sources for an Erlang package?

Further investigation for Mix required: https://hexdocs.pm/mix/Mix.SCM.html.

Determine how to separate code dependencies from build/test ones

Are scopes supported/recognised in Mix/rebar3?

rebar3 has the concept of profiles (default, native, prod, test) which we possibly could use to do this.

Implementation is not to use exeternal plugins to analyze Hex (Mix/rebar3) projects

Example Hex projects:

Mix:

https://github.com/jshmrtn/iona -- mix.exs names dependencies and package license

https://github.com/unnawut/licensir -- mix.exs names dependencies

rebar3:

https://github.com/certifi/erlang-certifi -- no dependencies in rebar.config, licenses in src/certifi.app.src

https://github.com/benoitc/erlang-idna

rebar3 and erlang.mk:

https://hex.pm/packages/mimerl -- rebar.config does not list dependencies, they are fetched by erlang.mk

Both Mix and rebar3:

https://hex.pm/packages/ssl_verify_fun -- neither mix.exs nor rebar.config list any dependencies

Add code coverage badge to README.md

Add Bower (JavaScript) as supported package manager

Add support for the Bower package manager to the analyzer module.

https://bower.io/

Support the Swift Package Manager (iOS)

Add support for the Swift dependency manager to the analyzer module.

https://github.com/apple/swift-package-manager

swift package show-dependencies --format json is the command to fetch dependencies according to https://www.cocoawithlove.com/blog/package-manager-fetch.html

Related materials:

analyzer: Add Godep support

Add support for Godep to the analyzer module.

https://github.com/tools/godep

downloader: Implement downloading of source artifacts

If the VCS of a package can not be used, for example because it is undefined, invalid, or requires authorization, fall back to using the source artifact to get the source code.

Support Licensee as a scanner

As another simple example on how to integrate a scanner, adding support for Licensee would be nice.

Support LicenseFinder as a scanner

As another simple example on how to integrate a scanner, adding support for LicenseFinder would be nice.

Support the BUCK build system

See ~~https://buckbuild.com/~~ https://buck.build/ as used e.g. by https://github.com/SeleniumHQ/selenium/tree/9a65d4cb3ab44d5d1c77532aece8517d7678388d/java and older versions of Gerrit (newer versions have moved to Bazel).

downloader: Add SVN support

Add support for Subversion repositories to the downloader module.

downloader: Add functional tests

Add functional test to the downloader module to verify its functionality.

NPM/PNPM: Add support for peer dependencies

Add support for the peerDependencies scope to NPM. Be aware that the specific version of the peer dependency will only be resolved in the context of a project using the library.

npm-install-peers can be used to install peer dependencies, see this relevant ticket:
spatie/npm-install-peers#13

./gradlew installDist fails on Task :analyzer:compileKotlin

Hi,

I can't compile the current master branch and get an error on Task :analyzer:compileKotlin

Steps to reproduce the problem

I have :

installed OpenJDK 8

$ dpkg -l
ii openjdk-8-jdk:amd64 8u162-b12-1 amd64 OpenJDK Development Kit (JDK)

Ensured the JAVA_HOME environment variable is set.

$ echo $JAVA_HOME
/usr/lib/jvm/java-8-openjdk-amd64

cloned the repo and launched the build:

git clone --recurse-submodules https://github.com/heremaps/oss-review-toolkit.git
cd oss-review-toolkit
./gradlew installDist

Expected result

Successfull build

Actual result

FAILURE: Build failed with an exception.
What went wrong:
Execution failed for task ':analyzer:compileKotlin'.

(More details below)

Did I miss somethig ?
Thanks,
Camille

Task :analyzer:compileKotlin
file or directory '/home/camille/Devel/oss-review-toolkit/analyzer/src/main/java', not found
Build cache key for task ':analyzer:compileKotlin' is 4b3ff08342112e8deffd6c1f2d4a5f2b
Task ':analyzer:compileKotlin' is not up-to-date because:
Task has failed previously.
All input files are considered out-of-date for incremental task ':analyzer:compileKotlin'.
file or directory '/home/camille/Devel/oss-review-toolkit/analyzer/src/main/java', not found
file or directory '/home/camille/Devel/oss-review-toolkit/analyzer/src/main/java', not found
Using Kotlin incremental compilation
Options for KOTLIN DAEMON: IncrementalCompilationOptions(super=CompilationOptions(compilerMode=INCREMENTAL_COMPILER, targetPlatform=JVM, reportCategories=[0], reportSeverity=2, requestedCompilationResults=[0]), areFileChangesKnown=false, modifiedFiles=null, deletedFiles=null, workingDir=/home/camille/Devel/oss-review-toolkit/analyzer/build/kotlin/compileKotlin, customCacheVersionFileName='gradle-format-version.txt', customCacheVersion=4, resultDifferenceFile=/home/camille/Devel/oss-review-toolkit/analyzer/build/kotlin/compileKotlin/build-history.bin, friendDifferenceFile=null, usePreciseJavaTracking=truelocalStateDirs=[/home/camille/Devel/oss-review-toolkit/analyzer/build/classes/kotlin/main])

e: /home/camille/Devel/oss-review-toolkit/analyzer/src/main/kotlin/managers/Bundler.kt: (144, 13): No value passed for parameter 'definitionFilePath'

Task :analyzer:compileKotlin FAILED
[KOTLIN] deleting /home/camille/Devel/oss-review-toolkit/analyzer/build/classes/kotlin/main on error
[KOTLIN] deleting /home/camille/Devel/oss-review-toolkit/analyzer/build/classes/kotlin/main on error
:analyzer:compileKotlin (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 5.363 secs.

FAILURE: Build failed with an exception.

What went wrong:
Execution failed for task ':analyzer:compileKotlin'.

Compilation error. See log for more details

BUILD FAILED in 7s
17 actionable tasks: 1 executed, 16 up-to-date

PIP: Read `project_urls` attribute from `setup.py`

A VCS repository from which releases are created is preferrable to the location of a source artifact ("sdist" in Python lingo). There is a rarely used attribute in setup.py where this information can be recorded. Make sure this information is captured, if it is present.

Installation instructions and tests should be improved

Here's what I did.

Checked out the project: worked (duh)
Ran ./gradlew sync: worked
Ran ./gradlew check: failed
- Some tests failed because external executables such as hg could not be found; should skip these tests.
- Some tests failed because internal dependencies such as /scanner/src/funTest/assets/scanners/scancode-toolkit/ could not be found; should just work.

Turns out that for that last issue, pulling git submodules gets one step further, so that should be added to the instructions.

But the test case still fails; the subsequent step that looks for configure in the same directory now works. I'm wondering why you're looking for configure if not to build binaries in that directory, and if so, why they're not built.

Lastly, gradlew check at some point just hangs, or seems to hang, with no indication what's going on. I killed it after about 15 minutes run-time.

At this point I'd abandon the project unless I had immediate need for it. I'm not sure how non-technical folk just interested in a compliance tool would react :)

Support the Bazel build system

See https://bazel.build/ as used by newer versions of Gerrit .

Tasks

Beta Give feedback

Add support for Bazel workspaces #8449

analyzer enhancement
Enhance parsing of MODULE.bazel files #8450

analyzer enhancement
Support external package managers for Bazel (like Maven or Pip) #8451

analyzer enhancement
Take Bazel's patches to source artifacts into account #8452

analyzer enhancement
Support dynamic linking for Bazel #8453

analyzer enhancement
Options

Improve SBT support

Currently, we support SBT by running sbt makePom and then using our Maven-backend on the generated POMs. While that works fairly well, there are edge-cases where the results are incomplete e.g. for some inter-module dependencies or dynamically generated projects. Ideas to improve this include using Coursier directly, or maybe yet better the Build Server Protocol for Scala.

Add `provider` to Package

Packages have a packageManager (e.g, npm, maven, ...). Since a given type of package could come from many different places, the Package should also talk about a provider. The provider should not be the URL of the repository, rather the notional name of the repository (e.g., npmjs.org, github.com). This allows the repos to move and change their URL structure without affecting the identity of the data stored in ORT.

Capture the VCS revision actually fetched

In the Git downloader there is some code to refine the vcs revision that is fetched (e.g., looking up the package version as a tag). That gets the right source but the revision fetched is never recorded in the package data as far as I can tell. This info is key to get out and recorded somewhere.