eclipse / dash-licenses Goto Github PK

View Code? Open in Web Editor NEW

46.0 11.0 31.0 1.32 MB

Extract license information from content.

Home Page: http://projects.eclipse.org/projects/technology.dash

License: Eclipse Public License 2.0

Java 100.00%

dash-licenses's People

Stargazers

Watchers

dash-licenses's Issues

Make the implementation more extensible by adding dependency injection

Passing values (e.g. the settings, and logging) around via constructors is cumbersome. Further, things like the ContentIdParsers should be configurable and extendable (e.g., the FlatFileReader constructor hardcodes the parsers that it uses, this should be extendable). Dependency injection will make this sort of configuration easier.

Publish maven plugin

In order to be able to use the new maven-plugin it has to be published to maven repository. It's too early to think of maven central so probably https://repo.eclipse.org/ is best.

Consider switching to JCommander

Based on a cursory look, JCommander seems much simpler and provides validation. Consider switching.

Not all problems are visible with a multi-module POM

In https://dev.eclipse.org/ipzilla/show_bug.cgi?id=22896#c5, @waynebeaton mentioned that some Kotlin dependencies were identified as problematic but I am unable to see them in my tool output. Could you please help me run it with the right arguments? Using aba7d20 and ggrep (GNU grep) 3.6 packaged by Homebrew.

Dec 08, 2020 4:58:27 PM org.eclipse.dash.licenses.foundation.EclipseFoundationSupport queryLicenseData
INFO: Querying Eclipse Foundation for license data for 203 items.
Dec 08, 2020 4:58:30 PM org.eclipse.dash.licenses.foundation.EclipseFoundationSupport queryLicenseData
INFO: Found 148 items.
Dec 08, 2020 4:58:30 PM org.eclipse.dash.licenses.clearlydefined.ClearlyDefinedSupport queryLicenseData
INFO: Querying ClearlyDefined for license data for 55 items.
Dec 08, 2020 4:58:33 PM org.eclipse.dash.licenses.clearlydefined.ClearlyDefinedSupport queryLicenseData
INFO: Found 55 items.
License information could not be automatically verified for the following content:

maven/mavencentral/com.google.code.findbugs/jsr305/3.0.2
maven/mavencentral/javax.servlet.jsp.jstl/javax.servlet.jsp.jstl-api/1.2.2
maven/mavencentral/org.apache.jena/jena-querybuilder/3.14.0
maven/mavencentral/org.glassfish.jersey.bundles.repackaged/jersey-guava/2.25.1

This content is either not correctly mapped by the system, or requires review.

Repo is https://github.com/eclipse/lyo @v4.0.0.RC (HEAD)

P.S. Thank you for finally buiding this tool 😍

Missing D?

The exmple

$ mvn org.eclipse.dash:license-tool-plugin:license-check -dash.iplab.token=<token> -dash.projectId=technology.dash

seems to miss Ds, should be:

$ mvn org.eclipse.dash:license-tool-plugin:license-check -Ddash.iplab.token=<token> -Ddash.projectId=technology.dash

Batik 1.13 from Orbit not approved for some reason

While testing the Maven plugin for Eclipse ESCET, I get the following as output of the license check:

[INFO] --- license-tool-plugin:0.0.1-SNAPSHOT:license-check (license-check) @ org.eclipse.escet.root ---
Jun 06, 2021 3:04:45 PM org.eclipse.dash.licenses.foundation.EclipseFoundationSupport queryLicenseData
INFO: Querying Eclipse Foundation for license data for 242 items.
Jun 06, 2021 3:04:46 PM org.eclipse.dash.licenses.foundation.EclipseFoundationSupport lambda$queryLicenseData$3
INFO: Found 227 items.
Jun 06, 2021 3:04:46 PM org.eclipse.dash.licenses.clearlydefined.ClearlyDefinedSupport queryLicenseData
INFO: Querying ClearlyDefined for license data for 4 items.
Jun 06, 2021 3:04:47 PM org.eclipse.dash.licenses.clearlydefined.ClearlyDefinedSupport lambda$queryLicenseData$2
INFO: Found 4 items.
[INFO] License information could not be automatically verified for the following content:
[INFO] 
[INFO] p2/orbit/p2.eclipse-plugin/org.apache.batik.anim/1.13.0.v20201130-1756
[INFO] p2/orbit/p2.eclipse-plugin/org.apache.batik.awt.util/1.13.0.v20201124-1832
[INFO] p2/orbit/p2.eclipse-plugin/org.apache.batik.bridge/1.13.0.v20201124-1840
[INFO] p2/orbit/p2.eclipse-plugin/org.apache.batik.dom.svg/1.13.0.v20201124-1839
[INFO] p2/orbit/p2.eclipse-plugin/org.apache.batik.dom/1.13.0.v20201124-1839
[INFO] p2/orbit/p2.eclipse-plugin/org.apache.batik.ext/1.13.0.v20201124-1832
[INFO] p2/orbit/p2.eclipse-plugin/org.apache.batik.gvt/1.13.0.v20201124-1835
[INFO] p2/orbit/p2.eclipse-plugin/org.apache.batik.parser/1.13.0.v20201124-1833
[INFO] p2/orbit/p2.eclipse-plugin/org.apache.batik.script/1.13.0.v20201124-1840
[INFO] p2/orbit/p2.eclipse-plugin/org.apache.batik.xml/1.13.0.v20201124-1831
[INFO] 
[INFO] This content is either not correctly mapped by the system, or requires review.

There are 10 problematic dependencies reported, for Batik 1.13. However, we obtain these from Eclipse Orbit 2021-03, at https://download.eclipse.org/tools/orbit/downloads/drops/R20210223232630/, so these should be approved. For instance, the first one is obtained from https://download.eclipse.org/tools/orbit/downloads/drops2/R20210223232630/repository/plugins/org.apache.batik.anim_1.13.0.v20201130-1756.jar.

It seems the license check tool wrongly reports these as not allowed.

Run the tool on the tool during Maven build

Run the tool on the tool during Maven build.

Agree on a common data format

To work on different ways to collect the list of dependencies, there is a need to agree on a data format.

Right now a dependency is identified by its
ClearlyDefined coordinate system, stored in the ContentId java class:

dash-licenses/src/main/java/org/eclipse/dash/licenses/ContentId.java

Lines 12 to 18 in 13f6bcd

 public class ContentId implements IContentId { 

 private String type; 

 private String source; 

 private String namespace; 

 private String name; 

 private String version;

The idea of this coordinate system is to map values of technology-specific repositories (Maven Repository, NPM, …) to something that is technology-independent (see software dependencies coordinates for mapping examples).

I see at least two other attributes that needs to be captured:

1) Source repository (where does the dependency comes from)

The provider value used by ClearlyDefined is not really clear. I have raised clearlydefined/clearlydefined#125 to see if the ClearlyDefined project has plans in this area.

In all cases, knowing the repository where the dependency was downloaded from is interesting from an audit point of view and should be captured if available.

For example the Eclipse Foundation host several maven repositories under https://repo.eclipse.org/ and having a dependency coming from an other Eclipse project induces less checks from the IP point of view.

2) Usage

In the context of a given build, a dependency (and its transitive closure) has a specific usage:

Yarn defines:

normal dependencies
development dependencies
peer dependencies
optional dependencies
bundled dependencies

Maven scopes:

compile
provided
runtime
test
system
import

The Yarn Dependency type and the maven scope not exactly the same.
Probably like the ClearlyDefined coordinate system, a common vocabulary with its mapping from the different technologies could also be defined.

This information is needed in order to evaluate the "works with" dependencies (see issue #13)

I think that JSON could be used:

{
    "coordinates": {
        "type": "maven",
        "provider": "mavencentral",
        "namespace": "com.fasterxml.jackson.core",
        "name": "jackson-annotations",
        "revision": "2.9.3"
    },
    "repositoryLocation": "https://jcenter.bintray.com/",
    "usage": "normal"
}

Augment IP Logs with output from the Dash License Tool

IPZilla had originally been used to both track the process of reviewing and approving content, and tracking its use. Piggyback CQs, for example, exist only for tracking use. We actually eliminated the need for piggybacks some time ago.

The obvious solution is to leverage the output of this tool. The "-summary" option can be used to generate a CSV summary of the dependencies, their license, and approval state.

I'm thinking that it could be a handy addition to the repository. That is, use the tool to generate a file named DEPENDENCIES and just commit it in the repository as the record of third party dependencies. Note that this would make the NOTICES file redundant (which is almost certainly out-of-date most of the time anyway), so we can eliminate that file.

Note that the tool doesn't currently provide license information for content that it identifies as Eclipse project content. I'm hopeful that I'll be able to extend the backend to support that. There's no requirement to include Eclipse project content in the log, so one option is to just eliminate it before feeding it to the tool (e.g., pipe through "grep -v eclipse.org").

For the time being, I have been (though not 100% consistently) running builds, generating the DEPENDENCIES file, and uploading it as an additional attachment on the IP Log review CQs.

Update "IPLab" URL

We had to move the "iplab" repository (used for requesting IP Team review), rendering the default URL incorrect. Apparently redirects don't work through the API, so we need to update the default setting.

I've already made this change (f012912); I've created this issue to track the change.

Maven regex pattern removes version qualifier

The regex pattern used by the MavenIdParser strips any qualifier from the version string. This might be appropriate for handling p2 bundles (which I have no clue about) but it is certainly not correct for plain Maven dependencies.

For example the dependency com.google.guava:guava:jar:28.0-jre:compile results in a ID with version 28.0 when it should be 28.0-jre. The same problem would arise for versions containing a qualifier as often used in OSGi bundles like 1.3.0.Final or 1.2.1.RELEASE. This behavior results in failure to determine license information for such dependencies.

FMPOV the resulting ContentId instance's version should be the verbatim copy of the version string from the maven coordinates.
If we need extra handling for p2 bundles then we probably should add a P2IdParser which detects a p2 bundle by means of the group ID starting with p2.eclipse-.

Artifacts with Licensed score < 75 are accepted if overall is >= 75

It's my understanding, according to the Eclipse Handbook, that the Overall score should not play any role when deciding if an artifact has an approved license, but only the Licensed score should be considered. However, according to

dash-licenses/core/src/main/java/org/eclipse/dash/licenses/clearlydefined/ClearlyDefinedSupport.java

Lines 136 to 137 in d6eae6c

 if (data.getEffectiveScore() >= context.getSettings().getConfidenceThreshold() 

 || data.getLicenseScore() >= context.getSettings().getConfidenceThreshold()) {

, it seems that if an artifact has a Licensed score < 75 but an overall >= 75, it's indeed accepted from the tool. As an example, https://clearlydefined.io/definitions/maven/mavencentral/org.apache.camel/camel-core/3.10.0 is reported as Accpeted:

...
maven/mavencentral/org.apache.camel/camel-core/3.10.0, Apache-2.0, approved, clearlydefined
...

Is this correct, or we should just consider the Licensed value?

org.eclipse.rcp_root being rejected by dash

Hello EMO,

The Dash license check on LSP4E is rejecting some content from Eclipse itself:

License information could not be automatically verified for the following content:

maven/mavencentral/p2.p2-installable-unit/org.eclipse.rcp_root/4.18.0.v20201202-1800

Is this something that can be tuned on the dash licenses side? Please let me know what additional info you need.

Thanks
Jonah

xmlgraphics 2.3.0 with valid CQ is shown as missing

Running license tool on tycho codebase shows p2/orbit/p2.eclipse-plugin/org.apache.xmlgraphics/2.3.0.v20190515-0436 ()
as not auto verified but there is https://dev.eclipse.org/ipzilla/show_bug.cgi?id=19789 for it.

Support for other technologies via CLI

We currently know how to generate dependency lists from Maven, Gradle, NPM, Yarn... there are other technologies that we need to support, like golang, C/C++, ... This may not require a modification to the tool, but rather an understanding of how to capture dependency lists from the various technologies and deliver them to the tool.

Of course, there may be an opportunity to extend the format (Maven, pURL, ClearlyDefined) supported by the tool in the form of additional ContentIdParser implementations.

This issue is intended as an umbrella for other issues that address specific technology. That is, if you're able to provide support for a currently unsupported technology, open a separate issue for it.

gradle - {strict} marker

Using

./gradlew app:dependencies | grep -Poh "[^:\s]+:[^:]+:[^:\s]+" | sort | uniq

on a project I get:

com.android.support.constraint:constraint-layout:1.1.3
com.android.support.constraint:constraint-layout-solver:1.1.3
com.android.support.constraint:constraint-layout-solver:{strictly
com.android.support.constraint:constraint-layout:{strictly

The original gradle output looks like

com.android.support.constraint:constraint-layout-solver:{strictly 1.1.3} -> 1.1.3

Adapting the regex to

./gradlew app:dependencies | grep -Poh "[^:\s]+:[^:]+:[^:\s\{]+" | sort | uniq

suppress these extra lines. I'm not sure, if the suppress the {strictly 1.1.3} -> 1.1.3 has some downsides.

Validate the project id

I anticipate a couple of different uses for the project id.

e.g.,

There are a handful of cases where content is approved for use by individual but not all projects (e.g., board exceptions for content under licenses that are not on our approved list).
Automatically generating IP Team review requests (#51)

The id is only valuable if it's valid. We should check to ensure that the project id matches one that is on our list.

Ignore "NONE" as a discovered license

"NONE" in the list of discovered licenses indicates that somebody has curated the data and has overridden the harvester's discovery of a bogus license on a file when no license is actually specified.

When we're reviewing the discovered licenses, just skip "NONE" (i.e., treat it in the same manner that we treat cases where no license is discovered).

Automate filing third-party content review requests

I happily run dash-license tool and get a clear and useful output

$ java -jar ~/git/dash-licenses/core/target/org.eclipse.dash.licenses-0.0.1-SNAPSHOT.jar package-lock.json 
Feb 02, 2021 10:29:29 AM org.eclipse.dash.licenses.foundation.EclipseFoundationSupport queryLicenseData
INFO: Querying Eclipse Foundation for license data for 145 items.
Feb 02, 2021 10:29:30 AM org.eclipse.dash.licenses.foundation.EclipseFoundationSupport queryLicenseData
INFO: Found 71 items.
Feb 02, 2021 10:29:30 AM org.eclipse.dash.licenses.clearlydefined.ClearlyDefinedSupport queryLicenseData
INFO: Querying ClearlyDefined for license data for 74 items.
Feb 02, 2021 10:29:32 AM org.eclipse.dash.licenses.clearlydefined.ClearlyDefinedSupport queryLicenseData
INFO: Found 74 items.
License information could not be automatically verified for the following content:

npm/npmjs/-/copy-anything/2.0.1
npm/npmjs/-/less/3.13.1
npm/npmjs/-/sass/1.32.6
npm/npmjs/-/typescript/4.1.3
npm/npmjs/@angular/language-service/11.1.1

This content is either not correctly mapped by the system, or requires review.

Now, I'd like the extra step forward, just like pushing on GitHub suggests a link to directly create a pull request: a prepopulated link to open a CQ for those

How to consolidate maven coordiantes for past CQs?

I really like this tool!

using it for Eclipse/Californium, I got aware, that some CQs are not found:

maven/mavencentral/com.google.code.gson/gson/2.8.2, Apache-2.0, approved, CQ14005
maven/mavencentral/com.upokecenter/cbor/4.0.0, unknown, restricted, none

I already created a CQ for that a year ago, see CQ21057. I used the summary "com.upokecenter.cbor-4.0.0".
I don't know, how this tool locates CQs from the IContentId, but if there is a possibility to consolidate that mapping information, let me know (e.g. a APi to add the maven-coordinates to past CQs).

Create a Maven plug-in

I assume that the Maven dependency model must be accessible by plugins directly. That is, we should be able to create a Maven plugin that takes input from the Maven dependency model and avoid parsing text (parsing text is imperfect, time consuming, and potentially error-prone).

I expect that we can take inspiration from the Maven "dependency" plugin.

Replace apache httpclient with Java 11 one

Java ships HttpClient itself. Not as advanced as the Apache one but for the simple cases here should be enough. I can provide a patch for that if it's wanted change. Less dependencies is goood.

Return non-0 in case a dependency is invalid

Looking at the code, it seems like the application will always return success when listing the dependencies+IP status. In order to more easily and reliably integrate with other tools and workflows if the application would return an error/non-0 code in case a dependency isn't conform to Eclipse IP Policy. This tool could then be used to fail a build and automate a -1 vote on code-review.

Accept package-lock.json files in v2 format

npm since v7 uses a new lockfile format for storing dependencies in package-lock.json.

Current behavior when the CLI is used with the new format: No dependencies are found and thus no licenses are checked.

Expected behavior: CLI can also parse the new format and check the licenses.

Main changes in the lockfile format as far as I can overlook it:

dependencies has been renamed to packages
There is a dependency describing the dependencies of the library itself (dependency with key "")
All other dependency names start with prefix "node_modules/"

As an example an old format lockfile:

{
	"name": "bumlux",
	"version": "1.0.0",
	"lockfileVersion": 1,
	"requires": true,
	"dependencies": {
		"@babel/code-frame": {
			"version": "7.5.5",
			"resolved": "https://registry.npmjs.org/@babel/code-frame/-/code-frame-7.5.5.tgz",
			"integrity": "sha512-27d4lZoomVyo51VegxI20xZPuSHusqbQag/ztrBC7wegWoQ1nLREPVSKSW8byhTlzTKyNE4ifaTA6lCp7JjpFw==",
			"dev": true,
			"requires": {
				"@babel/highlight": "^7.0.0"
			}
		}
}

and the same one in v2:

{
  "name": "bumlux",
  "version": "1.0.0",
  "lockfileVersion": 2,
  "requires": true,
  "packages": {
    "": {
      "name": "bumlux",
      "version": "2.0.0",
      "license": "SEE LICENSE IN LICENSE",
      "dependencies": {},
      "devDependencies": {
        "@babel/code-frame": "7.5.5"
      }
    },
    "node_modules/@babel/code-frame": {
      "version": "7.12.13",
      "resolved": "https://registry.npmjs.org/@babel/code-frame/-/code-frame-7.12.13.tgz",
      "integrity": "sha512-HV1Cm0Q3ZrpCR93tkWOYiuYIgLxZXZFVG2VgK+MBWjUqZTundupbfx2aXarXuw5Ko5aMcjtJgbSs4vUGBS5v6g==",
      "dev": true,
      "dependencies": {
        "@babel/highlight": "^7.12.13"
      }
    }
}

Skip blank and header lines in Maven-generated dependencies list

When one uses the dependencies:list feature, the output includes some blank and header lines. For example:

The following files have been resolved:
   none


The following files have been resolved:
   p2.eclipse-plugin:com.google.gson:jar:2.8.6.v20201231-1626:system
...

These are identified by the tool as invalid content, but are still included in the output (and in the generated summary file):

License information could not be automatically verified for the following content:

Invalid:    none
Invalid: The following files have been resolved:
...

At very least these lines should just be ignored. Perhaps consider adding an option to ignore all invalid content.

Accept a content.xml as input

We should be able to map content in a p2 repository to ClearlyDefined identifiers.

For example, a content.xml file has entries like this:

...
<unit id='org.hamcrest.core' version='1.3.0.v201303031735' singleton='false'>
...

Which we can map to p2/orbit/p2.eclipse-plugin/org.hamcrest.core/1.3.0.v201303031735.

Note that the Eclipse Foundation's backend can handle these and has compatible mappings for many (perhaps most) Java libraries that we've reviewed. ClearlyDefined knows nothing of any of this.

The p2/orbit part is the convention that I've used for all content served via p2 repositories. Clearly, the orbit part isn't generally correct; so this may evolve as we learn more.

Provide option to fail Maven build on non-approved dependencies

The license check Maven plugin is nice. It would be even better if it also had an option to fail the build if any dependencies are non-approved. Then we could enable that for releases, to ensure we don't release with non-approved dependencies.

yarn list vs yarn.lock

The current documentation suggests using yarn list, processing that to a yarn.deps and then running the tool.

Has using yarn.lock been considered as an option? The yarn.lock is machine readable, and lists the versions actually used by the project.

(I was going to write some justification, but instead I wrote a tool, see PR in a moment)

Report more information in the output

I'm envisioning an update to the process for handling third party libraries where the committer need only create a single CQ that lists all of the third party content that needs review.

Currently the tool outputs a list of "problematic" third party content as ClearlyDefined URLs.

In the process of identifying those URLs, we actually discard a lot of information. I'd like to turn the implementation around a bit and actually gather the information that we find together and provide it in some form in the output. Right now the "support" classes make decisions on what to do. I'm thinking that we have the "support" classes just gather the information and then make decisions based on the totality of what is gathered. Then, again, the output would summarize the totality of what we discover (e.g., "such-and-such was rejected in CQ 1234, but ClearlyDefined gives it a passing grade" along with URLs, etc.)

I'm thinking that we could then send that output to the IP Team and have them sort it out. The simplest form would be to just copy and paste the output into a CQ. After we have that, we can think about more extensive changes, potentially involving IPZilla itself.

Sort out how to deal with "works with" dependencies

"Works with" dependencies are a type of dependency that does not require full scrutiny of the IP Team. Often-times, these dependencies are not under open source licenses or are under incompatible licenses. Works with dependencies should not be directly distributed. They are, by definition, either optional or can be replaced with alternatives.

It is common, for example, to see them in a Maven dependency list.

The transitive closure of the dependencies of a works with dependency are themselves under the umbrella of the works with and so require no further scrutiny.

"License information could not automatically verified" for Jakarta content

I have run the license tool on CDT (full maven.deps). I had a few false negatives (I believe).

This issue is for Jakarta entries that I believe are fine as they are from Eclipse:

p2/orbit/p2.eclipse-plugin/org.glassfish.hk2.api/2.6.1.v20200513-1859 ()
p2/orbit/p2.eclipse-plugin/org.glassfish.jersey.inject.jersey-hk2/2.30.1.v20200512-1802 ()
p2/orbit/p2.eclipse-plugin/org.glassfish.hk2.osgi-resource-locator/1.0.3.v20200509-0149 ()
p2/orbit/p2.eclipse-plugin/org.glassfish.jersey.apache.connector/2.30.1.v20200513-1859 ()
p2/orbit/p2.eclipse-plugin/jakarta.xml.bind/2.3.3.v20200525-2159 ()
p2/orbit/p2.eclipse-plugin/org.glassfish.hk2.locator/2.6.1.v20200513-1859 ()
p2/orbit/p2.eclipse-plugin/org.glassfish.hk2.utils/2.6.1.v20200513-1859 ()

Running on Mac

Hi Wayne,

Knowing some of us are using a MacOS machines, will it make sense to mention in the readme file for Mac users must install the GNU grep on their machine first

brew install grep
and then replace each instance of grep in the doc by ggrep?

Actually, the current grep on Mac doesn't support the -P option.

Kind regards

Provide more simple examples

All of the examples provided in the documentation are in the form magic-dependency-command | java -jar org.eclipse.dash.licenses-<version>.jar -. Could there please be some simple examples of what needs to be sent to the jar, for projects that don't have a magic dependency command? It took me a while to figure out the syntax for checking something on github is git/github/eclipse/mosquitto/2.0.9, for example.

An NPE happens when using a package-lock.json as input

An NPE happens when using a package-lock.json as input:

$ java -jar  org.eclipse.dash.licenses.jar package-lock.json
Nov 12, 2020 3:26:17 PM org.eclipse.dash.licenses.foundation.EclipseFoundationSupport queryLicenseData
INFO: Querying Eclipse Foundation for license data for 253 items.
Nov 12, 2020 3:26:19 PM org.eclipse.dash.licenses.foundation.EclipseFoundationSupport queryLicenseData
INFO: Found 89 items.
Nov 12, 2020 3:26:19 PM org.eclipse.dash.licenses.clearlydefined.ClearlyDefinedSupport queryLicenseData
INFO: Querying ClearlyDefined for license data for 164 items.
Exception in thread "main" java.lang.NullPointerException
	at org.eclipse.dash.licenses.LicenseChecker.lambda$getLicenseData$2(LicenseChecker.java:55)
	at org.eclipse.dash.licenses.clearlydefined.ClearlyDefinedSupport.lambda$queryLicenseData$1(ClearlyDefinedSupport.java:114)
	at java.base/java.util.Map.forEach(Map.java:661)
	at org.eclipse.dash.licenses.clearlydefined.ClearlyDefinedSupport.queryLicenseData(ClearlyDefinedSupport.java:111)
	at org.eclipse.dash.licenses.LicenseChecker.lambda$getLicenseData$3(LicenseChecker.java:54)
	at org.eclipse.dash.licenses.util.Batchifier.batchify(Batchifier.java:77)
	at org.eclipse.dash.licenses.LicenseChecker.getLicenseData(LicenseChecker.java:58)
	at org.eclipse.dash.licenses.cli.Main.lambda$main$2(Main.java:100)
	at java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
	at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:658)
	at org.eclipse.dash.licenses.cli.Main.main(Main.java:87)

Running tests failed on a `mvn clean package`

I'm using the README so cloned this repository and ran mvn clean package and facing problems in tests. I can switch off the tests but can you add probably more information to README? Probably I don't have enough dependencies or so.

Apache Maven 3.6.3 
Java version: 14.0.1

------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running org.eclipse.dash.licenses.tests.TestSuite
Nov 09, 2020 8:03:27 AM org.eclipse.dash.licenses.clearlydefined.ClearlyDefinedSupport queryLicenseData
INFO: Querying ClearlyDefined for license data for 1 items.
Tests run: 91, Failures: 0, Errors: 1, Skipped: 3, Time elapsed: 8.327 sec <<< FAILURE!
testMatchAgainstClearlyDefined()(ClearlyDefinedSupportTests)  Time elapsed: 6.752 sec  <<< ERROR!
java.lang.RuntimeException: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at org.eclipse.dash.licenses.clearlydefined.ClearlyDefinedSupport.queryLicenseData(ClearlyDefinedSupport.java:123)
	at org.eclipse.dash.licenses.tests.ClearlyDefinedSupportTests.testMatchAgainstClearlyDefined(ClearlyDefinedSupportTests.java:45)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:564)
	at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:688)
	at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
	at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:149)
	at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:140)
	at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:84)
	at org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
	at org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
	at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
	at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$6(TestMethodTestDescriptor.java:210)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:206)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:131)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:65)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:139)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:129)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:127)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:126)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:84)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1510)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:38)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:143)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:129)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:127)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:126)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:84)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1510)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:38)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:143)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:129)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:127)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:126)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:84)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:32)
	at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)
	at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:51)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:108)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:88)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:54)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:67)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:52)
	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:96)
	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:84)
	at org.junit.platform.runner.JUnitPlatform.run(JUnitPlatform.java:139)
	at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:564)
	at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
	at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
	at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at java.net.http/jdk.internal.net.http.HttpClientImpl.send(HttpClientImpl.java:568)
	at java.net.http/jdk.internal.net.http.HttpClientFacade.send(HttpClientFacade.java:119)
	at org.eclipse.dash.licenses.clearlydefined.ClearlyDefinedSupport.queryLicenseData(ClearlyDefinedSupport.java:106)
	... 75 more
Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:325)
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:268)
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:263)
	at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:645)
	at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:464)
	at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:360)
	at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396)
	at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:445)
	at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1260)
	at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1247)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:691)
	at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1192)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1510)
	at java.net.http/jdk.internal.net.http.common.SSLFlowDelegate.lambda$executeTasks$3(SSLFlowDelegate.java:1109)
	at java.net.http/jdk.internal.net.http.HttpClientImpl$DelegatingExecutor.execute(HttpClientImpl.java:155)
	at java.net.http/jdk.internal.net.http.common.SSLFlowDelegate.executeTasks(SSLFlowDelegate.java:1104)
	at java.net.http/jdk.internal.net.http.common.SSLFlowDelegate.doHandshake(SSLFlowDelegate.java:1070)
	at java.net.http/jdk.internal.net.http.common.SSLFlowDelegate$Reader.processData(SSLFlowDelegate.java:474)
	at java.net.http/jdk.internal.net.http.common.SSLFlowDelegate$Reader$ReaderDownstreamPusher.run(SSLFlowDelegate.java:264)
	at java.net.http/jdk.internal.net.http.common.SequentialScheduler$SynchronizedRestartableTask.run(SequentialScheduler.java:175)
	at java.net.http/jdk.internal.net.http.common.SequentialScheduler$CompleteRestartableTask.run(SequentialScheduler.java:147)
	at java.net.http/jdk.internal.net.http.common.SequentialScheduler$SchedulableTask.run(SequentialScheduler.java:198)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
	at java.base/java.lang.Thread.run(Thread.java:832)
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:439)
	at java.base/sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:306)
	at java.base/sun.security.validator.Validator.validate(Validator.java:264)
	at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:285)
	at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144)
	at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:623)
	... 21 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at java.base/sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
	at java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
	at java.base/java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297)
	at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:434)
	... 26 more


Results :

Tests in error: 
  testMatchAgainstClearlyDefined()(ClearlyDefinedSupportTests): javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

Tests run: 91, Failures: 0, Errors: 1, Skipped: 3

Sort summary file generated by Maven plugin

It would be great if the summary file generated by the Maven plugin would be sorted. That would be much more readable.

Quickly extract dependency list from yarn.lock file

I've been tinkering with some options to quickly parse a dependency list out of a yarn.lock file using CLI.

This is probably not as efficient as it can be, but it seems to work:

cat yarn.lock | grep -Poh '(?<=resolved\s").*(?=")' | grep -Poh '([\w-]+)\/([\w-]+)-(\d+(?:\.\d+){2})' | sed 's/\(.*\)-/\1@/'

The gist is that we can get the exact match from the resolved field in each entry (the interesting part is marked by brackets):

y18n@^4.0.0:
  version "4.0.0"
  resolved "https://registry.yarnpkg.com/y18n/[-/y18n-4.0.0].tgz#..."
  integrity sha512-...

In the process, I noticed that NpmJsIdParser doesn't correctly handle "-" as a namespace. In fact, the regular expression that we're currently using is problematic in other ways (e.g., it doesn't correctly handle extra slashes).

Add a P2IdParser

If we need extra handling for p2 bundles then we probably should add a P2IdParser which detects a p2 bundle by means of the group ID starting with p2.eclipse-.

Add logging

From an end user's perspective, the license tool is effectively a black box. We need some logging.

Minimally, we should log every interaction with a service. Configurable, finer grained logging should permit a more detailed record of the decisions made by the tool.

The implementation should take into consideration that the CLI is just one example of how the tool will be used and that the core functionality may be leveraged in other ways; e.g. a Maven Plugin (#7). Consider leveraging dependency injection (#8) in this implementation.

I believe that SLF4J is the current gold standard for logging. Convince me I'm wrong.

Dependencies with resolved CQs are still shown as restricted

Dependencies with resolved CQs are still shown as restricted.

For example, we already have CQs for the following runtime dependencies:
[email protected](CQ22348)
[email protected](CQ22349)
[email protected](CQ22350)
[email protected](CQ22351)
[email protected](CQ22352)
[email protected](CQ22353)
[email protected](CQ22354)
[email protected](CQ22355)
[email protected](CQ22358)
[email protected](CQ22359)

They were resolved a week ago.

The next run of dash-license shows that dependencies are still rejected:

…
npm/npmjs/-/attr-accept/1.1.3, unknown, restricted, none
…
npm/npmjs/-/dom-helpers/3.4.0, unknown, restricted, none
…
npm/npmjs/-/file-selector/0.1.12, unknown, restricted, none
…
npm/npmjs/-/json5/1.0.1, unknown, restricted, none
…
npm/npmjs/-/nan/2.14.1, unknown, restricted, none
…
npm/npmjs/-/popper.js/1.16.1, unknown, restricted, none
…
npm/npmjs/-/prop-types-extra/1.1.1, unknown, restricted, none
…
npm/npmjs/-/react-dropzone/9.0.0, unknown, restricted, none
…
npm/npmjs/-/vscode-languageclient/6.1.3, unknown, restricted, none
…
npm/npmjs/-/warning/4.0.3, unknown, restricted, none
…

dash-licenses commit

b907560

Steps to reproduce:

Clone and build dash-licenses.

$ git clone https://github.com/eclipse/dash-licenses.git
$ cd dash-licenses && mvn clean install
$ cd yarn && yarn install && cd -

Download the target yarn.lock file.

$ wget https://raw.githubusercontent.com/che-incubator/che-dashboard-next/master/yarn.lock

Run dash-licenses

$ node yarn/index.js | java -jar target/org.eclipse.dash.licenses-0.0.1-SNAPSHOT.jar -

The new file DEPENDENCIES will be created

Add timeout CLI param

Currently the request timeout for HTTP commands is hard coded to 60 seconds in ISettings.java:78.

This isn't enough for medium sized package-lock.json files.

As a user I would like to have another CLI-param (e.g. -request-timeout <timeout>) to set the timeout in seconds without having to change the code.

Some feedback, running dash-license on Eclipse Theia main repo

Hi!

The Eclipse Theia project has been using an experimental Intellectual Property Clearance Approach, that was proposed by the Foundation to help deal with our numerous and often updated production dependencies. I think this was the precursor to this tool here.

I've tried dash-license for the first time on Theia's main repo. The results are interesting (in a positive way) and not that far from what we were obtaining manually.

dash-licenses catches some dependencies that pass the manual process. I think for the most part this is due to the fact that this tool is more strict: it looks at the discovered license(s) while we were visually inspecting, on the clearlydefined desktop, the declared license(s) field that's visible without expanding each dependency. Also we were not considering the "confidence" levels.

In total, we have ~35 production dependencies, that previously passed our manual check, but do not pass dash-licenses.

Here's the DEPENDENCIES file I obtained (renamed so GitHub lets me upload it here)
DEPENDENCIES.TXT

Next step: Should we open one CQ per "suspicious" dependency?

yarn runtime vs development dependencies

This may not be an issue for the tool, but rather the CQ process?

At the moment to develop in yarn it takes hundreds of development time packages that are not part of the runtime distribution. For example, the cdt-gdb-adapter has ~500 items listed in yarn.deps when run on cdt-gdb-adapter code base. However, if you only consider the runtime dependencies the list is just 5 items.

One idea of how to automate is this:

yarn init -y
yarn add cdt-gdb-adapter
node <path to>licesnes/yarn/index.js  | grep -v 'cdt-gdb-adapter' > yarn.deps

More sophisticated parsing/matching of SPDX expressions

The is a quick and dirty "for now" solution. The reasoning is that a conjunctive expression is weird and needs attention. For expressions that include disjunction, we just split on the disjunction and see if the required license is in the list. For single license expressions, we just string match the text. This is good enough for many (perhaps most or almost all) cases.

The LicenseSupport#getStatus method is used to determine whether or not we can accept content distributed under terms described by a particular SPDX expression.

e.g.

licenseSupport.getStatus("EPL-2.0") => Status.Approved
licenseSupport.getStatus("GPL-2.0") => Status.Restricted
licenseSupport.getStatus("EPL-2.0 OR GPL-2.0") => Status.Approved
licenseSupport.getStatus("EPL-2.0 AND GPL-2.0") => Status.Restricted

Where it gets tricky (and this is likely a corner case) is when conjunction and disjunction are both present. The following expressions should be approved because EPL-2.0 is fine. But the current implementation will reject them because of the "AND" (again, the current implementation is "quick and dirty").

licenseSupport.getStatus("EPL-2.0 OR (GPL-2.0 AND Apache-2.0)") => Status.Rejected
licenseSupport.getStatus("EPL-2.0 OR (MIT AND Apache-2.0)") => Status.Rejected

I don't think that this actually impacts us, but the SPDX representation of licenses with exceptions has changed. I don't think that this impacts any licenses that are currently on our white list. e.g., GPL-2.0 WITH Classpath-exception-2.0 (deprecated) is equivalent to GPL-2.0-with-classpath-exception. Again, since any use of GPL-2.0 requires investigation by the IP Team, this particular case probably isn't an issue for us.

In any case (and as much as I love to build parsers), I have to believe that there is an existing library that can help us with this.

Maven plugin should not output to stderr

I was experimenting with using the license check tool as part of the Eclipse ESCET build. I get the following in the output:

[INFO] --- license-tool-plugin:0.0.1-SNAPSHOT:license-check (license-check) @ org.eclipse.escet.root ---
Jun 06, 2021 3:04:45 PM org.eclipse.dash.licenses.foundation.EclipseFoundationSupport queryLicenseData
INFO: Querying Eclipse Foundation for license data for 242 items.
Jun 06, 2021 3:04:46 PM org.eclipse.dash.licenses.foundation.EclipseFoundationSupport lambda$queryLicenseData$3
INFO: Found 227 items.
Jun 06, 2021 3:04:46 PM org.eclipse.dash.licenses.clearlydefined.ClearlyDefinedSupport queryLicenseData
INFO: Querying ClearlyDefined for license data for 4 items.
Jun 06, 2021 3:04:47 PM org.eclipse.dash.licenses.clearlydefined.ClearlyDefinedSupport lambda$queryLicenseData$2
INFO: Found 4 items.
[INFO] License information could not be automatically verified for the following content:
[INFO] 
[INFO] p2/orbit/p2.eclipse-plugin/org.apache.batik.anim/1.13.0.v20201130-1756
[INFO] p2/orbit/p2.eclipse-plugin/org.apache.batik.awt.util/1.13.0.v20201124-1832
[INFO] p2/orbit/p2.eclipse-plugin/org.apache.batik.bridge/1.13.0.v20201124-1840
[INFO] p2/orbit/p2.eclipse-plugin/org.apache.batik.dom.svg/1.13.0.v20201124-1839
[INFO] p2/orbit/p2.eclipse-plugin/org.apache.batik.dom/1.13.0.v20201124-1839
[INFO] p2/orbit/p2.eclipse-plugin/org.apache.batik.ext/1.13.0.v20201124-1832
[INFO] p2/orbit/p2.eclipse-plugin/org.apache.batik.gvt/1.13.0.v20201124-1835
[INFO] p2/orbit/p2.eclipse-plugin/org.apache.batik.parser/1.13.0.v20201124-1833
[INFO] p2/orbit/p2.eclipse-plugin/org.apache.batik.script/1.13.0.v20201124-1840
[INFO] p2/orbit/p2.eclipse-plugin/org.apache.batik.xml/1.13.0.v20201124-1831
[INFO] 
[INFO] This content is either not correctly mapped by the system, or requires review.

The following lines from this output, are on stderr, rather than logged via Maven:

Jun 06, 2021 3:04:45 PM org.eclipse.dash.licenses.foundation.EclipseFoundationSupport queryLicenseData
INFO: Querying Eclipse Foundation for license data for 242 items.
Jun 06, 2021 3:04:46 PM org.eclipse.dash.licenses.foundation.EclipseFoundationSupport lambda$queryLicenseData$3
INFO: Found 227 items.
Jun 06, 2021 3:04:46 PM org.eclipse.dash.licenses.clearlydefined.ClearlyDefinedSupport queryLicenseData
INFO: Querying ClearlyDefined for license data for 4 items.
Jun 06, 2021 3:04:47 PM org.eclipse.dash.licenses.clearlydefined.ClearlyDefinedSupport lambda$queryLicenseData$2

I think it should:

Be logged via Maven.
Not be as logged as errors, as this appears to be information on what the tool is doing.
Not log class/function names (odd lines), but only the actual information (even lines)

Regressions June 10 to June 11 in license check tool

For Eclipse ESCET I've integrated the license check into a daily Jenkins job. It succeeded the last few days. It failed today. The following is a the automated check for differences in dependencies and their license check result, by comparing DEPENDENCIES.txt which is in our Git repo and DEPENDENCIES.generated.processed.txt which is generated during the Jenkins job. Yesterday this did not lead to a diff, to day it does:

13:32:33  Checking for differences between generated and stored dependency lists...
13:32:33  --- DEPENDENCIES.txt	2021-06-11 11:29:43.722015370 +0000
13:32:33  +++ DEPENDENCIES.generated.processed.txt	2021-06-11 11:32:32.407978954 +0000
13:32:33  @@ -6,36 +6,36 @@
13:32:33   maven/mavencentral/org.codehaus.groovy/groovy-groovydoc/3.0.8, Apache-2.0, approved, clearlydefined
13:32:33   maven/mavencentral/org.codehaus.groovy/groovy-json/3.0.8, Apache-2.0, approved, clearlydefined
13:32:33   maven/mavencentral/org.codehaus.groovy/groovy/3.0.8, Apache-2.0, approved, clearlydefined
13:32:33  -p2/orbit/p2.eclipse-plugin/com.ibm.icu/67.1.0.v20200706-1749, ICU AND Unicode-TOU AND BSD-3-Clause AND BSD-2-Clause AND LicenseRef-ipadic-license AND LicenseRef-Public-Domain, approved, CQ22320
13:32:33  -p2/orbit/p2.eclipse-plugin/com.sun.jna.platform/4.5.1.v20190425-1842, Apache-2.0 OR LGPL-2.1-or-later, approved, CQ15796
13:32:33  -p2/orbit/p2.eclipse-plugin/com.sun.jna/4.5.1.v20190425-1842, Apache-2.0 OR LGPL-2.1-or-later, approved, CQ15797
13:32:33  -p2/orbit/p2.eclipse-plugin/io.github.java-diff-utils/4.8.0.v20201027-1614, Apache-2.0, approved, CQ22697
13:32:33  +p2/orbit/p2.eclipse-plugin/com.ibm.icu/67.1.0.v20200706-1749, unknown, restricted, none
13:32:33  +p2/orbit/p2.eclipse-plugin/com.sun.jna.platform/4.5.1.v20190425-1842, Apache-2.0 OR LGPL-2.1-or-later, approved, CQ15645
13:32:33  +p2/orbit/p2.eclipse-plugin/com.sun.jna/4.5.1.v20190425-1842, Apache-2.0 OR LGPL-2.1-or-later, approved, CQ15644
13:32:33  +p2/orbit/p2.eclipse-plugin/io.github.java-diff-utils/4.8.0.v20201027-1614, unknown, restricted, none
13:32:33   p2/orbit/p2.eclipse-plugin/javax.annotation/1.3.5.v20200909-1856, CDDL-1.1 OR GPL-2.0, approved, CQ16910
13:32:33   p2/orbit/p2.eclipse-plugin/javax.inject/1.0.0.v20091030, Apache-2.0, approved, CQ3555
13:32:33  -p2/orbit/p2.eclipse-plugin/org.apache.batik.anim/1.13.0.v20201130-1756, Apache-2.0, approved, CQ22608
13:32:33  -p2/orbit/p2.eclipse-plugin/org.apache.batik.awt.util/1.13.0.v20201124-1832, Apache-2.0, approved, CQ22611
13:32:33  -p2/orbit/p2.eclipse-plugin/org.apache.batik.bridge/1.13.0.v20201124-1840, Apache-2.0, approved, CQ22621
13:32:33  -p2/orbit/p2.eclipse-plugin/org.apache.batik.constants/1.13.0.v20200622-2037, Apache-2.0, approved, CQ22304
13:32:33  -p2/orbit/p2.eclipse-plugin/org.apache.batik.css/1.13.0.v20200622-2037, Apache-2.0, approved, CQ22303
13:32:33  -p2/orbit/p2.eclipse-plugin/org.apache.batik.dom.svg/1.13.0.v20201124-1839, Apache-2.0, approved, CQ22619
13:32:33  -p2/orbit/p2.eclipse-plugin/org.apache.batik.dom/1.13.0.v20201124-1839, Apache-2.0, approved, CQ22617
13:32:33  -p2/orbit/p2.eclipse-plugin/org.apache.batik.ext/1.13.0.v20201124-1832, Apache-2.0, approved, CQ22613
13:32:33  -p2/orbit/p2.eclipse-plugin/org.apache.batik.gvt/1.13.0.v20201124-1835, Apache-2.0, approved, CQ22618
13:32:33  -p2/orbit/p2.eclipse-plugin/org.apache.batik.i18n/1.13.0.v20200622-2037, Apache-2.0, approved, CQ22301
13:32:33  -p2/orbit/p2.eclipse-plugin/org.apache.batik.parser/1.13.0.v20201124-1833, Apache-2.0, approved, CQ22616
13:32:33  -p2/orbit/p2.eclipse-plugin/org.apache.batik.script/1.13.0.v20201124-1840, Apache-2.0, approved, CQ22620
13:32:33  -p2/orbit/p2.eclipse-plugin/org.apache.batik.util/1.13.0.v20200622-2037, Apache-2.0, approved, CQ22302
13:32:33  -p2/orbit/p2.eclipse-plugin/org.apache.batik.xml/1.13.0.v20201124-1831, Apache-2.0, approved, CQ22614
13:32:33  +p2/orbit/p2.eclipse-plugin/org.apache.batik.anim/1.13.0.v20201130-1756, unknown, restricted, none
13:32:33  +p2/orbit/p2.eclipse-plugin/org.apache.batik.awt.util/1.13.0.v20201124-1832, unknown, restricted, none
13:32:33  +p2/orbit/p2.eclipse-plugin/org.apache.batik.bridge/1.13.0.v20201124-1840, unknown, restricted, none
13:32:33  +p2/orbit/p2.eclipse-plugin/org.apache.batik.constants/1.13.0.v20200622-2037, unknown, restricted, none
13:32:33  +p2/orbit/p2.eclipse-plugin/org.apache.batik.css/1.13.0.v20200622-2037, unknown, restricted, none
13:32:33  +p2/orbit/p2.eclipse-plugin/org.apache.batik.dom.svg/1.13.0.v20201124-1839, unknown, restricted, none
13:32:33  +p2/orbit/p2.eclipse-plugin/org.apache.batik.dom/1.13.0.v20201124-1839, unknown, restricted, none
13:32:33  +p2/orbit/p2.eclipse-plugin/org.apache.batik.ext/1.13.0.v20201124-1832, unknown, restricted, none
13:32:33  +p2/orbit/p2.eclipse-plugin/org.apache.batik.gvt/1.13.0.v20201124-1835, unknown, restricted, none
13:32:33  +p2/orbit/p2.eclipse-plugin/org.apache.batik.i18n/1.13.0.v20200622-2037, unknown, restricted, none
13:32:33  +p2/orbit/p2.eclipse-plugin/org.apache.batik.parser/1.13.0.v20201124-1833, unknown, restricted, none
13:32:33  +p2/orbit/p2.eclipse-plugin/org.apache.batik.script/1.13.0.v20201124-1840, unknown, restricted, none
13:32:33  +p2/orbit/p2.eclipse-plugin/org.apache.batik.util/1.13.0.v20200622-2037, unknown, restricted, none
13:32:33  +p2/orbit/p2.eclipse-plugin/org.apache.batik.xml/1.13.0.v20201124-1831, unknown, restricted, none
13:32:33   p2/orbit/p2.eclipse-plugin/org.apache.commons.exec/1.1.0.v201301240602, Apache-2.0, approved, CQ6553
13:32:33   p2/orbit/p2.eclipse-plugin/org.apache.commons.io/2.6.0.v20190123-2029, Apache-2.0, approved, CQ16712
13:32:33   p2/orbit/p2.eclipse-plugin/org.apache.commons.jxpath/1.3.0.v200911051830, Apache-2.0, approved, CQ3247
13:32:33   p2/orbit/p2.eclipse-plugin/org.apache.commons.lang/2.6.0.v201404270220, Apache-2.0, approved, CQ6183
13:32:33   p2/orbit/p2.eclipse-plugin/org.apache.commons.lang3/3.1.0.v201403281430, Apache-2.0, approved, CQ5902
13:32:33  -p2/orbit/p2.eclipse-plugin/org.apache.commons.logging/1.2.0.v20180409-1502, Apache-2.0, approved, CQ16082
13:32:33  +p2/orbit/p2.eclipse-plugin/org.apache.commons.logging/1.2.0.v20180409-1502, Apache-2.0, approved, CQ10162
13:32:33   p2/orbit/p2.eclipse-plugin/org.apache.commons.math3/3.6.1.v20200817-1830, Apache-2.0 AND BSD-3-Clause AND BSD-2-Clause, approved, CQ22025
13:32:33  -p2/orbit/p2.eclipse-plugin/org.apache.felix.scr/2.1.24.v20200924-1939, , approved, CQ22609
13:32:33  -p2/orbit/p2.eclipse-plugin/org.apache.xalan/2.7.2.v20201124-1837, Apache-2.0, approved, CQ9587
13:32:33  -p2/orbit/p2.eclipse-plugin/org.apache.xerces/2.12.1.v20210115-0812, Apache-2.0 AND W3C-19980720, approved, CQ16951
13:32:33  +p2/orbit/p2.eclipse-plugin/org.apache.felix.scr/2.1.24.v20200924-1939, Apache-2.0 AND MIT, approved, CQ21362
13:32:33  +p2/orbit/p2.eclipse-plugin/org.apache.xalan/2.7.2.v20201124-1837, Apache-2.0, approved, CQ1985
13:32:33  +p2/orbit/p2.eclipse-plugin/org.apache.xerces/2.12.1.v20210115-0812, unknown, restricted, none
13:32:33   p2/orbit/p2.eclipse-plugin/org.apache.xml.resolver/1.2.0.v201005080400, Apache-2.0, approved, CQ1441
13:32:33   p2/orbit/p2.eclipse-plugin/org.apache.xmlgraphics/2.4.0.v20200622-2037, Apache-2.0, approved, CQ22305
13:32:33   p2/orbit/p2.eclipse-plugin/org.eclipse.ant.core/3.5.800.v20200608-1251, , approved, eclipse
13:32:33  @@ -148,11 +148,11 @@
13:32:33   p2/orbit/p2.eclipse-plugin/org.eclipse.ui/3.119.0.v20210111-1350, , approved, eclipse
13:32:33   p2/orbit/p2.eclipse-plugin/org.eclipse.urischeme/1.1.300.v20210113-1544, , approved, eclipse
13:32:33   p2/orbit/p2.eclipse-plugin/org.hamcrest.core/1.3.0.v20180420-1519, BSD-2-Clause, approved, CQ7063
13:32:33  -p2/orbit/p2.eclipse-plugin/org.junit/4.13.0.v20200204-1500, EPL-1.0, approved, CQ21440
13:32:33  -p2/orbit/p2.eclipse-plugin/org.knowm.xchart/3.6.5.v20201023-2252, Apache-2.0, approved, CQ22695
13:32:33  +p2/orbit/p2.eclipse-plugin/org.junit/4.13.0.v20200204-1500, EPL-1.0, approved, CQ21439
13:32:33  +p2/orbit/p2.eclipse-plugin/org.knowm.xchart/3.6.5.v20201023-2252, unknown, restricted, none
13:32:33   p2/orbit/p2.eclipse-plugin/org.sat4j.core/2.3.5.v201308161310, , approved, CQ7311
13:32:33   p2/orbit/p2.eclipse-plugin/org.sat4j.pb/2.3.5.v201404071733, , approved, CQ7311
13:32:33  -p2/orbit/p2.eclipse-plugin/org.tukaani.xz/1.8.0.v20180207-1613, LicenseRef-Public-Domain, approved, CQ15538
13:32:33  +p2/orbit/p2.eclipse-plugin/org.tukaani.xz/1.8.0.v20180207-1613, LicenseRef-Public-Domain, approved, CQ15386
13:32:33   p2/orbit/p2.eclipse-plugin/org.w3c.css.sac/1.3.1.v200903091627, Apache-2.0 OR LicenseRef-Public-Domain OR W3C, approved, CQ1448
13:32:33   p2/orbit/p2.eclipse-plugin/org.w3c.dom.events/3.0.0.draft20060413_v201105210656, Apache-2.0, approved, CQ2492
13:32:33   p2/orbit/p2.eclipse-plugin/org.w3c.dom.smil/1.0.1.v200903091627, Apache-2.0 OR LicenseRef-Public-Domain OR W3C, approved, CQ1448

And the following is outputted by the automatic check for restricted dependencies:

13:32:33  Checking for restricted dependencies...
13:32:33  p2/orbit/p2.eclipse-plugin/com.ibm.icu/67.1.0.v20200706-1749, unknown, restricted, none
13:32:33  p2/orbit/p2.eclipse-plugin/io.github.java-diff-utils/4.8.0.v20201027-1614, unknown, restricted, none
13:32:33  p2/orbit/p2.eclipse-plugin/org.apache.batik.anim/1.13.0.v20201130-1756, unknown, restricted, none
13:32:33  p2/orbit/p2.eclipse-plugin/org.apache.batik.awt.util/1.13.0.v20201124-1832, unknown, restricted, none
13:32:33  p2/orbit/p2.eclipse-plugin/org.apache.batik.bridge/1.13.0.v20201124-1840, unknown, restricted, none
13:32:33  p2/orbit/p2.eclipse-plugin/org.apache.batik.constants/1.13.0.v20200622-2037, unknown, restricted, none
13:32:33  p2/orbit/p2.eclipse-plugin/org.apache.batik.css/1.13.0.v20200622-2037, unknown, restricted, none
13:32:33  p2/orbit/p2.eclipse-plugin/org.apache.batik.dom.svg/1.13.0.v20201124-1839, unknown, restricted, none
13:32:33  p2/orbit/p2.eclipse-plugin/org.apache.batik.dom/1.13.0.v20201124-1839, unknown, restricted, none
13:32:33  p2/orbit/p2.eclipse-plugin/org.apache.batik.ext/1.13.0.v20201124-1832, unknown, restricted, none
13:32:33  p2/orbit/p2.eclipse-plugin/org.apache.batik.gvt/1.13.0.v20201124-1835, unknown, restricted, none
13:32:33  p2/orbit/p2.eclipse-plugin/org.apache.batik.i18n/1.13.0.v20200622-2037, unknown, restricted, none
13:32:33  p2/orbit/p2.eclipse-plugin/org.apache.batik.parser/1.13.0.v20201124-1833, unknown, restricted, none
13:32:33  p2/orbit/p2.eclipse-plugin/org.apache.batik.script/1.13.0.v20201124-1840, unknown, restricted, none
13:32:33  p2/orbit/p2.eclipse-plugin/org.apache.batik.util/1.13.0.v20200622-2037, unknown, restricted, none
13:32:33  p2/orbit/p2.eclipse-plugin/org.apache.batik.xml/1.13.0.v20201124-1831, unknown, restricted, none
13:32:33  p2/orbit/p2.eclipse-plugin/org.apache.xerces/2.12.1.v20210115-0812, unknown, restricted, none
13:32:33  p2/orbit/p2.eclipse-plugin/org.knowm.xchart/3.6.5.v20201023-2252, unknown, restricted, none

I observe that:

Some CQ numbers have suddenly changed. Not sure what that is about.
More importantly, a lot of the dependencies are now suddenly restricted.

What happened here?

"License information could not automatically verified" for Tycho (compile)

I have run the license tool on CDT (full maven.deps). I had a few false negatives (I believe).

This issue is for Tycho compile time dependency:

# tycho-compiler-plugin:1.7.0:compile (null)

"License information could not automatically verified" for stuff in Orbit

I have run the license tool on CDT (full maven.deps). I had a few false negatives (I believe).

This issue is for stuff in Orbit that I believe are fine - based on them being in https://download.eclipse.org/tools/orbit/downloads/drops/S20200826182026/

p2/orbit/p2.eclipse-plugin/javax.annotation/1.3.5.v20200504-1837 ()
p2/orbit/p2.eclipse-plugin/javax.ws.rs/2.1.6.v20200505-2127 ()

Handling of special cases?

Applying that tool for Californium, I get:

maven/mavencentral/com.google.code.findbugs/jsr305/3.0.2, LGPL-2.0, restricted, emo_ip_team

That seems to be a dependency from guava, but I can't see, that this is included in the guava.jar (nor the californium proxy's jars). I assume, that's more a build/test dependency, even if the maven pom scopes it different.

Is there a intended way to handle such "special cases"?

felix.gogo.command not recognized as OK - but felix.gogo.runtime is

I guess the metadata about felix.gogo.command is not correct somehow - but not sure. Both f.g.command and f.g.runtime are in latest orbit release with CQ 22658 and CQ 22659 respectively.

If this isn't a bug in dash-licenses, I would be happy to fix it where it is - but I don't know where/how to do that.

felix.gogo.command:

 $ echo p2.eclipse-plugin:org.apache.felix.gogo.command:jar:1.1.0.v20201023-1558:system | java -jar dash-licenses/core/target/org.eclipse.dash.licenses-*-SNAPSHOT.jar -
Jan 11, 2021 8:33:45 PM org.eclipse.dash.licenses.foundation.EclipseFoundationSupport queryLicenseData
INFO: Querying Eclipse Foundation for license data for 1 items.
Jan 11, 2021 8:33:45 PM org.eclipse.dash.licenses.foundation.EclipseFoundationSupport queryLicenseData
INFO: Found 0 items.
License information could not be automatically verified for the following content:

p2/orbit/p2.eclipse-plugin/org.apache.felix.gogo.command/1.1.0.v20201023-1558

This content is either not correctly mapped by the system, or requires review.

felix.gogo.runtime

$ echo p2.eclipse-plugin:org.apache.felix.gogo.runtime:jar:1.1.2.v20201023-1558:system | java -jar dash-licenses/core/target/org.eclipse.dash.licenses-*-SNAPSHOT.jar -
Jan 11, 2021 8:33:48 PM org.eclipse.dash.licenses.foundation.EclipseFoundationSupport queryLicenseData
INFO: Querying Eclipse Foundation for license data for 1 items.
Jan 11, 2021 8:33:48 PM org.eclipse.dash.licenses.foundation.EclipseFoundationSupport queryLicenseData
INFO: Found 1 items.
Vetted license information was found for all content. No further investigation is required.

	public class ContentId implements IContentId {

	private String type;
	private String source;
	private String namespace;
	private String name;
	private String version;

	if (data.getEffectiveScore() >= context.getSettings().getConfidenceThreshold()
	\|\| data.getLicenseScore() >= context.getSettings().getConfidenceThreshold()) {

eclipse / dash-licenses Goto Github PK

dash-licenses's People

Stargazers

Watchers

Forkers

dash-licenses's Issues

dash-licenses commit

Steps to reproduce:

Recommend Projects

Recommend Topics

Recommend Org