Currently licensed includes files matching:

LEGAL_FILES = /\A(AUTHORS|COPYING|NOTICE|LEGAL)(?:\..*)?\z/i

I think COPYING is superfluous, see #84, but there are some other files (also with any extension) that often include legal notices that would be useful to automatically collect, eg:

• LICENSES, NOTICES (plural)
• THIRD-PARTY-LICENSE (variations including 3rd, no delimiter, underscore delimiter, notice, plural)

I can probably suggest a semi-sane regex and tests for this, leaving this issue until then, or if anyone else wishes to comment or implement.

In the V2 branch there's been quite a bit of cleanup, refactoring and new or changed functionality.

Changes made in v2 branch

• sources now have a base class, giving a more definitive structure for ease in developing new sources
• the inheritance between the Dependency and License classes is broken. Dependency now inherits from Licensee::Projects::FSProject and contains a license
• the License class, which represents the dependency data that gets serialized/deserialized on disk, has been renamed to DependencyRecord.
• cached records are now stored entirely in YAML as with a .dep.yml extension
• dependencies now have a number of required fields, denoted in the constructor by keyword arguments
• a new reporting infrastructure has been added to provide a way to shape licensed output differently for different scenarios. direct reporting to the UI from anywhere in the code is removed

To do

• gracefully catch and handle Licensed::Shell::Error from reporters. see if it's possible to continue execution of licensed when finding dependency data fails
• Each command follows the same pattern at this point. Investigate whether it makes sense to define a base class for the pattern which the individual sources can override.
• Create migrator for v1 -> v2 configuration and license files
• prepare for v2 launch - update version, put notification in readme

Dependency sources that use package managers are able to version the cached license data from the package manager data. This means we are storing a single piece of metadata that

• determines if a dependency has changed
• provides valuable metadata that can be used to find a specific version of the dependency at it's source

There are also sources that relate to locally stored / vendored files, such as the manifest source and go source. These sources don't use any sort of package manager, and their version information is taken from the latest git SHA of the files in the local repo. This metadata

• determines if a dependency has changed
• provides no valuable information about the dependency

It might be useful and less confusing for consumers of the cached data to explicitly separate the two pieces of cached data. e.g.

---
name: my-dependency

# optional metadata available to consumers
version: <public version (if available)>

# used by licensed to detect stale cached data
tag: <public version (if available) OR, internal git SHA>

...
---

Easy distribution != ruby. That said, there are some options available to make it easier to use licensed in non-ruby environments

docker images

It's not entirely clear what the best usage of this would be, though a base ruby image + licensed already installed might be useful?

ruby-packaged "executable"

The two most promising options I've seen in this space are

1. ruby-packer
2. traveling-ruby

They both amount to packaging up ruby + <app> into an executable. ruby-packer would be my preferred approach as it seems to have broader support for varying ruby versions and native extensions.

On the flip side, running a ruby-packer-built executable doesn't play nicely with much referencing gem information from outside the executable which is a current requirement of the rubygems source. I don't know if traveling-ruby has the same issue, but either way switching to CLI should take care of that.

For a number of grammars in Linguist we've had to manually curate these (see example).

---
type: grammar
name: fortran.tmbundle
license: permissive
curated: true

---

When running script/licensed cache it would be useful if Licensed would check to see if there is already a file with curated: true and if so, leave it in place (perhaps with a warning?).

I broke it. 😔

My recent work on Licensed::Source::NPM is causing cache entries for instances of multiple versions of the dependency. This was not the case prior to that update. Currently looking into alternatives more closely aligned to Licensed::Source::Go.

Licensed currently has a manifest source that requires users to generate or update the manifest outside of licensed. It would be useful to move that logic internal to licensed so that external scripting isn't needed to generate the manifest or keep it up to date.

This would allow licensed to check that the manifest file is up to date, which would be 👍 .

If the manifest generation is able to work with project names, that would be ✨ ✨. Without being able to tie sources to projects, generating/updating the manifest needs user interaction before licensed can run properly

1. generating/updating manifest adds new sources that are unattributed to projects
   1. user updates manifest to attribute new sources to projects
2. licensed checks manifest sources

Background

#50 talks about options for creating easily distributable ways of running licensed from non-ruby environments.

Looking specifically at the executable options, I believe ruby-packer is a better choice due to it's easier build pipeline and packaging structure; ruby-packers single executable vs traveling-rubys script + ruby + app.

The ruby-packer-built executable contains a squashed file system with ruby + app which leads to...

The problem

When licensed is run on the squashed container file system in the executable, the containers rubygems code is used. In the ruby-packer project, rubygems has local edits that explicitly disallow loading gem paths from outside the container - (1 2)

This issue presents itself when using ::Bundler::Definition#specs. The method turns un-evaluated specifications (LazySpecification) into materialized specifications (Specification). When Bundler looks through it's known specification sources, it's sources are evaluated using rubygems paths and can only find information from inside of the ruby-packer executables file system.

This doesn't work well with the bundler source, who's entire job is to load gemspecs which always exist outside of the container file system. 💣 💥

The solution

Bundler does a lot and needs to support a large number of scenarios. licensed does not - it requires that bundler has already been run and the environment is ready for sourcing dependencies. In this scenario information is available at known paths. Given that information, licensed can sidestep using ::Bundler::Definition#specs entirely.

Licensed should be able to get by with the specification information from ::Bundler::Definition#resolve combined with ::Bundler#specs_path. The latter information isn't built using path information from rubygems, which makes it 👍 to configure for use outside of the ruby-packer executable's file system.

Pipenv is a popular package and virtualenv manager for Python.

Like Ruby's Bundler which has Gemfile and Gemfile.lock, Pipenv has Pipfile and Pipfile.lock.

Note that Pipenv can produce a requirements.txt file, which I tried to use with licensed, but no luck.

$ pipenv lock --requirements > requirements.txt

$ head requirements.txt
-i https://pypi.org/simple
-e .
absl-py==0.6.1
astor==0.7.1
boto3==1.9.62
boto==2.49.0
botocore==1.12.62
bz2file==0.98
certifi==2018.11.29
chardet==3.0.4

$ licensed list
Displaying dependencies for datascience core package

$ cat .licensed.yaml
sources:
  pip: true

As part of updating the cached licenses in Linguist in github-linguist/linguist#4268, I've just updated the Licensed gem (locally) from ~> 1.1.0 to ~> 1.3.0 which has pulled in Licensed 1.3.3.

Within script/licensed we're calling Licensed::Git.version() as follows:

Dir.glob(@glob).map do |directory|
  Licensed::Dependency.new(directory, {
    "type" => Filesystem.type,
    "name" => File.basename(directory),
    "version" => Licensed::Git.version(directory)
  })
end

... where directory is a git submodule for each grammar.

With Licensed 1.1.0, this determined the correct SHA for each submodule, but on 1.3.3 it returns a completely bogus SHA as seen with the most recent update of the cached licenses I performed after updating the gem:

$ git --no-pager diff vendor/licenses/grammar/ABNF.tmbundle.txt
diff --git a/vendor/licenses/grammar/ABNF.tmbundle.txt b/vendor/licenses/grammar/ABNF.tmbundle.txt
index 9afaeecb..e14f7c48 100644
--- a/vendor/licenses/grammar/ABNF.tmbundle.txt
+++ b/vendor/licenses/grammar/ABNF.tmbundle.txt
@@ -1,10 +1,10 @@
 ---
 type: grammar
 name: ABNF.tmbundle
-version: 0042b3d7ee21b2f800a8f80d42761513f16d9a52
+version: a4d12cc8e42aae26a9469edb9fedb25a7cdbc9e9
 license: apache-2.0
 ---
-Apache License
+                                 Apache License
                            Version 2.0, January 2004
                         http://www.apache.org/licenses/
$

$ cd vendor/grammars/ABNF.tmbundle
$ git rev-parse HEAD
0042b3d7ee21b2f800a8f80d42761513f16d9a52
$
$ git show a4d12cc8e42aae26a9469edb9fedb25a7cdbc9e9
fatal: bad object a4d12cc8e42aae26a9469edb9fedb25a7cdbc9e9
$

That same SHA has been applied to multiple license files:

$ git --no-pager diff | grep -B2  a4d12cc8e42aae26a9469edb9fedb25a7cdbc9e9
 name: ABNF.tmbundle
-version: 0042b3d7ee21b2f800a8f80d42761513f16d9a52
+version: a4d12cc8e42aae26a9469edb9fedb25a7cdbc9e9
--
 name: AutoHotkey
-version: dc20ea3b615d902fc43045c1073353af5f9e0edf
+version: a4d12cc8e42aae26a9469edb9fedb25a7cdbc9e9
--
 name: Lean.tmbundle
-version: e56b352bfc2bfc6b28154a8d7d68e469a454386b
+version: a4d12cc8e42aae26a9469edb9fedb25a7cdbc9e9
--
 name: PHP-Twig.tmbundle
-version: 4fe0a237dc534a2d94a468f0ab1319c09ae1b573
+version: a4d12cc8e42aae26a9469edb9fedb25a7cdbc9e9
--
 name: atomic-dreams
-version: 234b52d4828230d7741c35af2681619b55e972eb
+version: a4d12cc8e42aae26a9469edb9fedb25a7cdbc9e9
--
 name: language-jsoniq
-version: 5e400b06f1b417746c4c6aff42a2a938b6ef2f82
+version: a4d12cc8e42aae26a9469edb9fedb25a7cdbc9e9
--
 name: llvm.tmbundle
-version: 5db1a160f09b041777d1ed8f1a98a4386a363e38
+version: a4d12cc8e42aae26a9469edb9fedb25a7cdbc9e9
--
 name: objective-c.tmbundle
-version: fb271290491fda66730f5dd90abd5c3787fb7c5e
+version: a4d12cc8e42aae26a9469edb9fedb25a7cdbc9e9
$

Further digging shows we're getting a lot of duplicate SHAs:

$ git --no-pager diff | grep "+version" | sort | uniq -c | sort -r
  39 +version: 8369d253f87d15eacb51e4c0c8ea71927358d069
  26 +version: f9c00deecfb63efad37108f7b13699fc74bccbfe
  23 +version: 2942bf9f1bf7a4871167c046c1e27d26cb198200
  23 +version: 15e2b74dec453726188e1ae3faafd2e7eb3dc191
  20 +version: e60384b01890863754a9249405f57abd9622248c
  16 +version: ed44c0b86d317b753c607742b137dce9594bed97
  14 +version: b41875df5df876ae8ac8717baff9c5f26632b886
  10 +version: ea3e79a631fb2cd1f1b3c7fa9c6f68892b478892
   8 +version: a4d12cc8e42aae26a9469edb9fedb25a7cdbc9e9
   6 +version: d39a75b68b5870f1275eb27e1c5c41383d5eff4f
   5 +version: c59c88f16e672daeae30db3ba2bc701f5657c402
   4 +version: d2c7d27d13affd542d318a1f51ae77d96cb9c652
   4 +version: 4c66582f877c1846bdab00496543c3a038e867c6
   3 +version: edabdc75a5373e84dccc059a48cfc2d7c38bbde1
   3 +version: e335d48625c33acfe1a38e8f225af31649afa093
   3 +version: cb5bc91fe3550f6332c94e19bc74b0a2489bfb27
   3 +version: a944769d61795e2d7aea17e502996dbf250f261c
   3 +version: 9ca6a5841e93ea3f8a45adb3ac79bcc878b85892
   3 +version: 99dcd501aa7fa8d4efb4459ae79841de48a562a8
   3 +version: 994bc1f13560661f94c880845c967c91116e2f87
   3 +version: 68b553ea55f042a596f83c258e6accf4ba15b004
   3 +version: 2c3069db77b1d0a89330a5fff5fd3349db1e93e6
   2 +version: b802045c5c339ffe1e5a3b36d140e1d74a002dea
   2 +version: aae48b5b5eb358ac3e09c354972de38d647b5ad0
   2 +version: a18ad1d489f740ccf52f77f528c61794818eedbb
   2 +version: 6e40de47da9e774b7de722cf2dacf2d74f40595b
   2 +version: 50c08bf29e6e4de3623d5e7177c5e05ed1200348
   2 +version: 25aa6669bed3456326c8b1a15a9de767d09ff54e
   1 +version: f0c7380132465bd4a378b4e174f6bee896a96f1e
[...]
$

Every single license now has a new version and they're all wrong from what I can see from a quick glimpse - I don't fancy checking all 309 right now 😉

I just upgraded to v2.0.0 (🎉) but running licensed cache failed:

~/my/project#master$ ./script/licensed cache
If you encounter rate-limiting errors, please set the GITHUB_TOKEN environment variable and try again.
Caching dependency records for GitHub Pages
  bundler
bundler: failed to load command: licensed (/Users/parkr/my/project/vendor/gems/ruby/2.5.0/bin/licensed)
NameError: undefined local variable or method `dir_path' for #<Licensed::Dependency:0x00007f8a56919d20>
  /Users/parkr/my/project/vendor/gems/ruby/2.5.0/gems/licensed-2.0.0/lib/licensed/dependency.rb:92:in `notice_files'
  /Users/parkr/my/project/vendor/gems/ruby/2.5.0/gems/licensed-2.0.0/lib/licensed/dependency.rb:83:in `notice_contents'
  /Users/parkr/my/project/vendor/gems/ruby/2.5.0/gems/licensed-2.0.0/lib/licensed/dependency.rb:61:in `record'
  /Users/parkr/my/project/vendor/gems/ruby/2.5.0/gems/licensed-2.0.0/lib/licensed/commands/cache.rb:39:in `evaluate_dependency'
  /Users/parkr/my/project/vendor/gems/ruby/2.5.0/gems/licensed-2.0.0/lib/licensed/commands/command.rb:91:in `block in run_dependency'
  /Users/parkr/my/project/vendor/gems/ruby/2.5.0/gems/licensed-2.0.0/lib/licensed/reporters/cache_reporter.rb:61:in `block in report_dependency'
  /Users/parkr/my/project/vendor/gems/ruby/2.5.0/gems/licensed-2.0.0/lib/licensed/reporters/reporter.rb:111:in `report_dependency'
  /Users/parkr/my/project/vendor/gems/ruby/2.5.0/gems/licensed-2.0.0/lib/licensed/reporters/cache_reporter.rb:60:in `report_dependency'
  /Users/parkr/my/project/vendor/gems/ruby/2.5.0/gems/licensed-2.0.0/lib/licensed/commands/command.rb:84:in `run_dependency'
  /Users/parkr/my/project/vendor/gems/ruby/2.5.0/gems/licensed-2.0.0/lib/licensed/commands/command.rb:63:in `block (2 levels) in run_source'
  /Users/parkr/my/project/vendor/gems/ruby/2.5.0/gems/licensed-2.0.0/lib/licensed/commands/command.rb:63:in `map'
  /Users/parkr/my/project/vendor/gems/ruby/2.5.0/gems/licensed-2.0.0/lib/licensed/commands/command.rb:63:in `block in run_source'
  /Users/parkr/my/project/vendor/gems/ruby/2.5.0/gems/licensed-2.0.0/lib/licensed/reporters/cache_reporter.rb:28:in `block in report_source'
  /Users/parkr/my/project/vendor/gems/ruby/2.5.0/gems/licensed-2.0.0/lib/licensed/reporters/reporter.rb:89:in `report_source'
  /Users/parkr/my/project/vendor/gems/ruby/2.5.0/gems/licensed-2.0.0/lib/licensed/reporters/cache_reporter.rb:26:in `report_source'
  /Users/parkr/my/project/vendor/gems/ruby/2.5.0/gems/licensed-2.0.0/lib/licensed/commands/command.rb:61:in `run_source'
  /Users/parkr/my/project/vendor/gems/ruby/2.5.0/gems/licensed-2.0.0/lib/licensed/commands/cache.rb:21:in `run_source 31 gem "jekyll-geo-pattern", "~> 0.4.1"
'
  /Users/parkr/my/project/vendor/gems/ruby/2.5.0/gems/licensed-2.0.0/lib/licensed/commands/command.rb:44:in `block (3 levels) in run_app'
  /Users/parkr/my/project/vendor/gems/ruby/2.5.0/gems/licensed-2.0.0/lib/licensed/commands/command.rb:44:in `map'
  /Users/parkr/my/project/vendor/gems/ruby/2.5.0/gems/licensed-2.0.0/lib/licensed/commands/command.rb:44:in `block (2 levels) in run_app'
  /Users/parkr/my/project/vendor/gems/ruby/2.5.0/gems/licensed-2.0.0/lib/licensed/commands/command.rb:42:in `chdir'
  /Users/parkr/my/project/vendor/gems/ruby/2.5.0/gems/licensed-2.0.0/lib/licensed/commands/command.rb:42:in `block in run_app'
  /Users/parkr/my/project/vendor/gems/ruby/2.5.0/gems/licensed-2.0.0/lib/licensed/reporters/cache_reporter.rb:14:in `block in report_app'
  /Users/parkr/my/project/vendor/gems/ruby/2.5.0/gems/licensed-2.0.0/lib/licensed/reporters/reporter.rb:66:in `report_app'
  /Users/parkr/my/project/vendor/gems/ruby/2.5.0/gems/licensed-2.0.0/lib/licensed/reporters/cache_reporter.rb:12:in `report_app'
  /Users/parkr/my/project/vendor/gems/ruby/2.5.0/gems/licensed-2.0.0/lib/licensed/commands/command.rb:41:in `run_app'
  /Users/parkr/my/project/vendor/gems/ruby/2.5.0/gems/licensed-2.0.0/lib/licensed/commands/command.rb:23:in `block (2 levels) in run'
  /Users/parkr/my/project/vendor/gems/ruby/2.5.0/gems/licensed-2.0.0/lib/licensed/commands/command.rb:23:in `map'
  /Users/parkr/my/project/vendor/gems/ruby/2.5.0/gems/licensed-2.0.0/lib/licensed/commands/command.rb:23:in `block in run'
  /Users/parkr/my/project/vendor/gems/ruby/2.5.0/gems/licensed-2.0.0/lib/licensed/reporters/reporter.rb:44:in `report_run'
  /Users/parkr/my/project/vendor/gems/ruby/2.5.0/gems/licensed-2.0.0/lib/licensed/commands/command.rb:22:in `run'
  /Users/parkr/my/project/vendor/gems/ruby/2.5.0/gems/licensed-2.0.0/lib/licensed/cli.rb:74:in `run'
  /Users/parkr/my/project/vendor/gems/ruby/2.5.0/gems/licensed-2.0.0/lib/licensed/cli.rb:14:in `cache'
  /Users/parkr/my/project/vendor/gems/ruby/2.5.0/gems/thor-0.20.3/lib/thor/command.rb:27:in `run'
  /Users/parkr/my/project/vendor/gems/ruby/2.5.0/gems/thor-0.20.3/lib/thor/invocation.rb:126:in `invoke_command'
  /Users/parkr/my/project/vendor/gems/ruby/2.5.0/gems/thor-0.20.3/lib/thor.rb:387:in `dispatch'
  /Users/parkr/my/project/vendor/gems/ruby/2.5.0/gems/thor-0.20.3/lib/thor/base.rb:466:in `start'
  /Users/parkr/my/project/vendor/gems/ruby/2.5.0/gems/licensed-2.0.0/exe/licensed:5:in `<top (required)>'
  /Users/parkr/my/project/vendor/gems/ruby/2.5.0/bin/licensed:23:in `load'
  /Users/parkr/my/project/vendor/gems/ruby/2.5.0/bin/licensed:23:in `<top (required)>'

Found when running licensed with npm v 5.5.1. It didn't appear that any of the package information out put as part of npm list --json --production ... included a realPath property. Instead I was seeing a path property.

    "array.from": {
      ...
      "path": "/Users/jonabc/.../node_modules/array.from",
      ...
    },

realPath might have existed for previous npm versions, I can't find good documentation for legacy versions. However it seems like the path property can be used to find the packages unpacked location.

Now that we have a binary build with ruby-packer it would be nice to have a

licensed --version

which would display the current version of the binary.
This could be useful for applications that want to download and run the licensed binary and automatically update to the latest version if available.
for .eg

licensed --update-licensed

would update the current copy is there's a later release version of the binary otherwise continue using the locally cached version.

Does that make sense?

Whilst implementing using the manifest method for caching the license metadata in Linguist in github-linguist/linguist#4267, I discovered that Licensed doesn't traverse submodules.

Digging into the code, it appears it is because of the call to git ls-tree --full-tree -r --name-only HEAD at

Switching this out for git ls-files --recurse-submodules --full-name is likely to get us the behaviour needed to traverse submodules.

licensed currently requires a Git repository to translate relative configuration paths into absolute paths on the local machine. This is because

1. all current usages use git repositories
2. using absolute paths in a checked in configuration file is 👎
3. licensed requires absolute paths

The combination of ☝️ led to an implementation that only included support for Git.

For broader usage it would make sense to support licesned with non-Git source control and directories not in a source control system.

The manifest dependency source is intended to be used in situations where a package manager is not available. Usages so far have been for c/c++ projects where dependencies are vendored into the project as source copied from another project.

There is significant friction in using this source when vendored source code doesn't contain an explicit LICENSE type file. This is not uncommon for c/c++ projects where license and copyright information is often copied directly into each file as a comment header. The current workflow for updating dependencies in those projects looks like

1. run licensed
   1. if licensed finds dependency updates, it updates the version and clears out the license text / metadata for each updated dependency
2. re-add license text / metadata as needed for each dependency and commit
3. re-run licensed to ensure everything is up to date 👍

Neither licensed nor licensee currently have the ability to understand license/copyright information as a comment header. licensee explicitly states that it's not a project goal to support header comments, which means that fixing this should be handled in licensed.

With npm, it is possible to have two different versions of the same dependency at the same time, with two potentially different licenses.

For example, given dependencies "a", "b", and "c", where "a" depends on "[email protected]" and "c" depends on "[email protected]", npm will essentially have a tree in node_modules that looks like this:

               ┌──────────┐               
               │          │               
               │   app    │               
               │          │               
               └──────────┘               
                     │                    
      ┌──────────────┼──────────────┐     
      ▼              ▼              ▼     
┌──────────┐   ┌──────────┐   ┌──────────┐
│          │   │          │   │          │
│    a     │   │ [email protected]  │   │    c     │
│          │   │          │   │          │
└──────────┘   └──────────┘   └──────────┘
                                    │     
                                    │     
                                    ▼     
                              ┌──────────┐
                              │          │
                              │ [email protected]  │
                              │          │
                              └──────────┘

However, when caching licenses for these dependencies, only the latest version encountered while traversing the dependency tree will be kept—in this case, "[email protected]"'s license will overwrite "[email protected]"'s.

Currently, the manifest source uses the most git commit hash of the files in each dependency to determine the version.

This mostly works well but has an annoying edge case when licensed is run on an updated file before the file is committed. When the file change is then committed, the dependency will have a new latest git commit hash. licensed status will then report an error for the cached dependency content as being out of date.

It would be great to have an alternative method for finding the dependencies "version" that is reproducible.

One suggestion was to create a hash of the file contents for each file in the dependency. This would be great for small numbers of small files but if the dependency contains a large number of files or any number of very large files, calculating the hash could become expensive. Are there cheap ways to calculate hashes for large files and/or large numbers of files?

Any other ideas?

::Bundler.default_gemfile.basename is used in the bundler source when looking for a Gemfile in the applications source path.

Using ::Bundler.default_gemfile and ::Bundler.default_lockfile in licensed will raise errors when

1. licensed is not run with an empty or nil ENV["BUNDLE_GEMFILE"]
   • e.g. not run from bundle exec

$> gem install licensed
$> licensed cache ...

2. a Gemfile is not available in the file hierarchy from root to the application source path

When both of these are true, a default Gemfile isn't found and ::Bundler.default_gemfile raises an error.

This can be worked around by explicitly disabling the Bundler source

sources:
  rubygem: false

Hi,

I continue to play with licensed under docker...

I encounter an issue when I add uuid to my package.json file. Here are the necessary files to reproduce this behaviour:

.licensed.yaml

name: 'placement-back'
cache_path: '/app/test2/.licenses'
source_path: '/app/test2'
sources:
  npm: true
allowed:
  - mit

package.json

{
  "name": "placement-back",
  "version": "0.0.0",
  "dependencies": {
    "uuid": "^3.2.1"
  }
}

And the commands/output within my docker image:

$ cd /app/test2
$ npm update -f
$ licensed cache
Caching licenses for placement-back:
  npm dependencies:
/usr/local/bundle/gems/licensed-1.1.0/lib/licensed/shell.rb:17:in `execute': command exited with status 1 (Licensed::Shell::Error)
  npm list --parseable --production --long
    npm ERR! missing: uuid@^3.2.1, required by [email protected]
	from /usr/local/bundle/gems/licensed-1.1.0/lib/licensed/source/npm.rb:66:in `npm_list_command'
	from /usr/local/bundle/gems/licensed-1.1.0/lib/licensed/source/npm.rb:55:in `package_location_command'
	from /usr/local/bundle/gems/licensed-1.1.0/lib/licensed/source/npm.rb:23:in `dependencies'
	from /usr/local/bundle/gems/licensed-1.1.0/lib/licensed/command/cache.rb:29:in `block (3 levels) in run'
	from /usr/local/bundle/gems/licensed-1.1.0/lib/licensed/command/cache.rb:20:in `map'
	from /usr/local/bundle/gems/licensed-1.1.0/lib/licensed/command/cache.rb:20:in `block (2 levels) in run'
	from /usr/local/bundle/gems/licensed-1.1.0/lib/licensed/command/cache.rb:17:in `chdir'
	from /usr/local/bundle/gems/licensed-1.1.0/lib/licensed/command/cache.rb:17:in `block in run'
	from /usr/local/bundle/gems/licensed-1.1.0/lib/licensed/command/cache.rb:12:in `each'
	from /usr/local/bundle/gems/licensed-1.1.0/lib/licensed/command/cache.rb:12:in `flat_map'
	from /usr/local/bundle/gems/licensed-1.1.0/lib/licensed/command/cache.rb:12:in `run'
	from /usr/local/bundle/gems/licensed-1.1.0/lib/licensed/cli.rb:48:in `run'
	from /usr/local/bundle/gems/licensed-1.1.0/lib/licensed/cli.rb:17:in `cache'
	from /usr/local/bundle/gems/thor-0.20.0/lib/thor/command.rb:27:in `run'
	from /usr/local/bundle/gems/thor-0.20.0/lib/thor/invocation.rb:126:in `invoke_command'
	from /usr/local/bundle/gems/thor-0.20.0/lib/thor.rb:387:in `dispatch'
	from /usr/local/bundle/gems/thor-0.20.0/lib/thor/base.rb:466:in `start'
	from /usr/local/bundle/gems/licensed-1.1.0/exe/licensed:5:in `<top (required)>'
	from /usr/local/bundle/bin/licensed:23:in `load'
	from /usr/local/bundle/bin/licensed:23:in `<main>'

I think this occurs when the README.md file is utf-8 encoded; I also see this behaviour for other packages.

I'm not sure this is a licensed bug; maybe a licensee bug as the error occurs in /usr/local/bundle/gems/licensee-9.9.1/lib/licensee/project_files/readme_file.rb:42 file?

As a workaround; I simply seach and overwrite all README/AUTHORS files with an empty content:

	find . -type f -name "README.md" -exec bash -c "echo > '{}'" \;
	find . -type f -name "Readme.md" -exec bash -c "echo > '{}'" \;
	find . -type f -name "readme.md" -exec bash -c "echo > '{}'" \;
	find . -type f -name "AUTHORS" -exec bash -c "echo > '{}'" \;

Then if I re-run the licensed command:

$ licensed cache
Caching licenses for placement-back:
  npm dependencies:
    Caching uuid (3.2.1)
License caching complete!
* placement-back npm dependencies: 1

Without (utf-8) content, it works; but I guess this is not a good workaround.

Any idea for a better process?

Regards,

Chris

As found in #30 (comment)

The errors appear to be because my package.json file is actually intended for yarn, and uses some syntax that is not npm-compatible

Yarn's package.json files attempt to be compatible with NPM package.json files but can contain differences that cause the NPM dependency source to fail.

It looks like an indicator that yarn should be used and that npm shouldn't be used is the presence of a yarn.lock file.

ref: #47 (comment)

Licensed has started using newer ruby functionality and should continue to do so. The &. operator was introduced in ruby 2.3. The required ruby version should be enforced in the gemspec to catch any issues up front with clear error messaging

/cc @mlinksva thanks for the heads up

The go source currently relies on all Golang ENV vars being set in the caller environment. This was done mostly for simplicity.

Using a settable GOPATH during dependency evaluation makes the go source more well rounded and self-contained. It also makes the go source more testable as we can set the var to fixture locations in test configurations.

Allowed path values should match existing path configuration settings values:

1. relative paths from repo root

go: 
  GOPATH: .gopath

2. expandable paths

go:
  GOPATH: ~/go

3. absolute paths

go:
  GOPATH: /path/to/gopath

Licensed needs an absolute path to use as a root for all relative path expansions from configuration files and CLI output.

Currently, licensed assumes that it's run from a git repository and uses the repository root as the absolute path to expand all relative paths. If licensed isn't run from a git repository though, everything will blow up.

In #94, a solution is proposed that allows a custom absolute path root value to be set but still doesn't handle the case when the root is unset and a git repository root is unavailable.

A nil root could be handled by

1. setting a default root value if all other options aren't available

• pwd maybe?

2. throwing an explicit error if a root value isn't available in a configuration
3. ??

Release 1.0.0 to align with making project public. This includes breaking changes from previous releases.

• Add CHANGELOG

The configuration file can specify whether sources are enabled or disabled with settings like:

sources:
  rubygem: true
  go: false
  ...

Currently, if a source is not specified it will default to enabled. A source can individually signal that it's not available via it's own #enabled? method, however a different paradigm on the configuration value may be a good idea.

Proposal:

I think the following set of rules would give users better flexibility in specifying which sources they want to run.

1. All sources are enabled via configuration when the sources configuration property is not set
2. All sources are enabled via configuration when the sources configuration property is empty
3. If any source is enabled via a true value, all sources without a true value are disabled

sources:
  rubygem: true
  npm: false
  # go: false implicitly

4. If sources only contains false property values, all other sources are enabled

sources:
  rubygem: false
  npm: false
  # go: true implicitly

The change here is in rule (3), where explicitly setting a true value implicitly disables all other sources. This gives users the option of specifying only the sources they want to run without having to keep an expansive list of all sources that should not run. It also keeps a good level of flexibility for users to not have to specify every source that should be enabled in every configuration file.

I get the following error when I try to run Licensed on my npm-based project. The error doesn't tell me what JSON file it's trying to parse, so it's pretty hard to diagnose/fix.

$ licensed list
Displaying dependencies for my-ui
  npm dependencies:
/opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/json-1.8.6/lib/json/common.rb:155:in `initialize': A JSON text must at least contain two octets! (JSON::ParserError)
	from /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/json-1.8.6/lib/json/common.rb:155:in `new'
	from /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/json-1.8.6/lib/json/common.rb:155:in `parse'
	from /Users/ips/.chefdk/gem/ruby/2.3.0/gems/licensed-1.0.1/lib/licensed/source/npm.rb:28:in `dependencies'
	from /Users/ips/.chefdk/gem/ruby/2.3.0/gems/licensed-1.0.1/lib/licensed/command/list.rb:32:in `dependencies'
	from /Users/ips/.chefdk/gem/ruby/2.3.0/gems/licensed-1.0.1/lib/licensed/command/list.rb:18:in `block (3 levels) in run'
	from /Users/ips/.chefdk/gem/ruby/2.3.0/gems/licensed-1.0.1/lib/licensed/command/list.rb:15:in `each'
	from /Users/ips/.chefdk/gem/ruby/2.3.0/gems/licensed-1.0.1/lib/licensed/command/list.rb:15:in `block (2 levels) in run'
	from /Users/ips/.chefdk/gem/ruby/2.3.0/gems/licensed-1.0.1/lib/licensed/command/list.rb:14:in `chdir'
	from /Users/ips/.chefdk/gem/ruby/2.3.0/gems/licensed-1.0.1/lib/licensed/command/list.rb:14:in `block in run'
	from /Users/ips/.chefdk/gem/ruby/2.3.0/gems/licensed-1.0.1/lib/licensed/command/list.rb:12:in `each'
	from /Users/ips/.chefdk/gem/ruby/2.3.0/gems/licensed-1.0.1/lib/licensed/command/list.rb:12:in `run'
	from /Users/ips/.chefdk/gem/ruby/2.3.0/gems/licensed-1.0.1/lib/licensed/cli.rb:48:in `run'
	from /Users/ips/.chefdk/gem/ruby/2.3.0/gems/licensed-1.0.1/lib/licensed/cli.rb:31:in `list'
	from /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/thor-0.19.1/lib/thor/command.rb:27:in `run'
	from /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/thor-0.19.1/lib/thor/invocation.rb:126:in `invoke_command'
	from /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/thor-0.19.1/lib/thor.rb:359:in `dispatch'
	from /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/thor-0.19.1/lib/thor/base.rb:440:in `start'
	from /Users/ips/.chefdk/gem/ruby/2.3.0/gems/licensed-1.0.1/exe/licensed:5:in `<top (required)>'
	from /Users/ips/.chefdk/gem/ruby/2.3.0/bin/licensed:22:in `load'
	from /Users/ips/.chefdk/gem/ruby/2.3.0/bin/licensed:22:in `<main>'

It would be great to also be able to drop licensed in a project with a gradle build and get licensed information as with the other dependency sources.

(I might have the time to investigate in starting such an implementation)

First mentioned in #21 (comment).

When licensee finds multiple license files in a project, it doesn't attempt to determine which is the "valid" license content. As a result the Project#license_file method returns nil if more than one license file is found.

This makes sense for licensee but for licensed's purposes, it may be useful to display all license file text.

Some considerations if all license files are written to the cached file

1. Are there any legal consequences to displaying multiple licenses if they are conflicting or it's unclear which one is valid?
2. There may be a mismatch between license content shown and the license metadata value
3. How to distinguish where the license content came from
   • Is this even needed? We don't label where Dependency#notices files come from in the cached file

/cc @mlinksva @benbalter

This has been a long time coming.

Using ::Bundler::* functionality directly to pull specifications from an external gemspec can be painful. Bundlers API is not stateless and can conflict with running licensed via bundle exec 🤦‍♂️. We've generally been able to work around these issues but edge cases are getting harder to track down.

Using the bundler CLI means we can execute commands in a separate shell so any existing bundler state is 🔥, and we can control the ENV to maintain current functionality.

Hi,

I've been asked to evaluate this tool in a POC for an internal application. Licensed; looks nice and interesting. I'm not a ruby developer/user, more someone from the CI/CD team, and our app is not written in ruby but uses npm as package manager. I plan to integrate a licensed check step into our Jenkins pipeline and instead of installing licensed in our CI nodes, I choose to build a docker image so I can mount our code and run licensed into a docker container.

I first start with creating a Docker image from the official ruby image (as licensed was written in ruby); here is my Dockerfile:

FROM ruby:2.3.7-jessie
RUN echo deb http://ftp.debian.org/debian jessie-backports main >> /etc/apt/sources.list
RUN apt-get update && \
    apt-get install -y cmake pkg-config npm nodejs
RUN gem install licensed
RUN useradd --no-log-init --create-home -r -u 1000 -g users test
RUN mkdir /app && \
    chown test:users /app
USER test
WORKDIR /app

Build the image:

docker build --rm -t licensed:ruby-2.3.7 .

Here is my licensed app config: .licensed.yaml

name: 'placement-back'
cache_path: '/app/test1/.licenses'
source_path: '/app/test1'
sources:
  npm: true
allowed:
  - mit

... and my package.json:

{
  "name": "placement-back",
  "version": "0.0.0",
  "dependencies": {
    "async": "^2.6.0",
    "atomic": "0.0.2"
  }
}

Then I can mount the app and run licensed; here is a simple example:

docker run -it --rm --network="host" -v /home/chris/workspace/app:/app licensed:ruby-2.3.7 bash
$ cd /app/test1
$ npm update
[email protected] node_modules/atomic
[email protected] node_modules/async
└── [email protected]
$ licensed cache
Caching licenses for placement-back:
  npm dependencies:
    Caching async (2.6.1)
    Caching lodash (4.17.10)
    Caching atomic (0.0.2)
License caching complete!
* placement-back npm dependencies: 3
$ licensed status
Checking licenses for placement-back: 3 dependencies
...
3 dependencies checked, 0 warnings found.

$ uname -a
Linux localhost.localdomain 4.15.14-1.el7.elrepo.x86_64 #1 SMP Wed Mar 28 14:16:59 EDT 2018 x86_64 GNU/Linux
$ cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 8 (jessie)"
NAME="Debian GNU/Linux"
VERSION_ID="8"
VERSION="8 (jessie)"
ID=debian

This worked fine with this simple test1 example. Great!

My problems comes when I tried to use the real package.json from my app, having something 40+ main dependencies (approx 500 dependencies in total after running npm). When I run npm update with this json file; npm returns errors. I asked the developers for help and they told me that the npm engine that I used was too old; they requires a npm 8.9.3 version. This first docker image have a npm 1.4.21 version.

Instead of creating the docker image from ruby; I thought that it should be easier to start with a nodejs image to avoid having the npm issue.

Here is my second Docker image from node:

FROM node:8.11.1
RUN echo deb http://ftp.debian.org/debian jessie-backports main >> /etc/apt/sources.list
RUN apt-get update && \
    apt-get install -y cmake pkg-config ruby ruby-dev python-pip
RUN gem install licensed
RUN mkdir /app && \
    chown node:users /app
USER node
WORKDIR /app

Build the image:

docker build -f Dockerfile.node --rm -t licensed:node-8.11 .

Then I run the same test with this new image:

docker run -it --rm --network="host" -v /home/chris/workspace/licensed:/app licensed:node-8.11 bash
$ cd /app/test1
$ npm update
npm WARN [email protected] No description
npm WARN [email protected] No repository field.
npm WARN [email protected] No license field.

+ [email protected]
+ [email protected]
added 3 packages in 4.474s
$ licensed cache
/usr/lib/ruby/2.1.0/rubygems/core_ext/kernel_require.rb:55:in `require': /var/lib/gems/2.1.0/gems/licensed-1.1.0/lib/licensed/source/pip.rb:48: syntax error, unexpected '.' (SyntaxError)
          a[k.strip] = v&.strip
                          ^
	from /usr/lib/ruby/2.1.0/rubygems/core_ext/kernel_require.rb:55:in `require'
	from /var/lib/gems/2.1.0/gems/licensed-1.1.0/lib/licensed.rb:14:in `<top (required)>'
	from /usr/lib/ruby/2.1.0/rubygems/core_ext/kernel_require.rb:55:in `require'
	from /usr/lib/ruby/2.1.0/rubygems/core_ext/kernel_require.rb:55:in `require'
	from /var/lib/gems/2.1.0/gems/licensed-1.1.0/lib/licensed/cli.rb:2:in `<top (required)>'
	from /usr/lib/ruby/2.1.0/rubygems/core_ext/kernel_require.rb:55:in `require'
	from /usr/lib/ruby/2.1.0/rubygems/core_ext/kernel_require.rb:55:in `require'
	from /var/lib/gems/2.1.0/gems/licensed-1.1.0/exe/licensed:4:in `<top (required)>'
	from /usr/local/bin/licensed:23:in `load'
	from /usr/local/bin/licensed:23:in `<main>'

Licensed failed (even a licensed help command fails).

Here are some additional information regarding config & installed gems:

$ uname -a
Linux localhost.localdomain 4.15.14-1.el7.elrepo.x86_64 #1 SMP Wed Mar 28 14:16:59 EDT 2018 x86_64 GNU/Linux
$ cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 8 (jessie)"
NAME="Debian GNU/Linux"
VERSION_ID="8"
VERSION="8 (jessie)"
ID=debian
$ gem list --local

*** LOCAL GEMS ***

addressable (2.5.2)
bigdecimal (1.2.4)
faraday (0.15.2)
io-console (0.4.2)
json (1.8.1)
licensed (1.1.0)
licensee (9.6.0)
minitest (4.7.5)
multipart-post (2.0.0)
octokit (4.9.0)
pathname-common_prefix (0.0.1)
psych (2.0.5)
public_suffix (3.0.2)
rake (10.1.0)
rdoc (4.1.0)
rugged (0.27.1)
sawyer (0.8.1)
test-unit (2.1.5.0)
thor (0.20.0)
tomlrb (1.2.7)

I don't know what exactly is missing in this docker image to run licensed. I've tried install python-pip and pip gem; and start building licensed from source but it does not fix this and it was not part of the 1st ruby image... still the same error.

Help/idea appreciated ^^

Regards,

Chris

It'll be great to add composer source (https://getcomposer.org/) to be able to enumerate php dependencies

Enable CI to run tests to validate pull requests. Travis CI and Circle CI look like the best options at the moment.

This will be require some extra thought and work due to the nature of the project. For each supported package manager, language or tool that's used we need to ensure that it is available on in the CI environment so that tests can run.

There's also the question of having one monolithic CI job that installs all dependencies runs all source tests, or breaking out each source into it's own CI job.

My 2c is to try and modularize this as much as possible and use separate CI jobs for each source.

With #65, we've removed any immediate dependency on gem octokit.

However, running bundle install still installs octokit, rugged and their dependencies even though those gems are entirely unused by licensed.

Please add support for pip

Uncertain what causes it - I migrated to licensed v2 and it was unable to find certain gems (json 2.1.0 in this case).

Example app: https://github.com/lcpriest/bug-demo-licensed
Ruby version 2.5.1

Steps to reproduce:

• 1 Run bundle install
• 2 Run licensed cache

Expected outcome

All licenses would be cached

Actual outcome

Some licenses were cached but it threw the following error

  * Errors:
    * Test-app.bundler.json
      - expected dependency path /Users/username/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/json-2.1.0 does not exist

There have been cases where a dependency is re-cached with no changes to the license text, however the license field is changed to other.

If the license text hasn't changed, we should keep the existing license to reduce noise to users.

project.license.content_normalized will return a normalized version of the license file's content. This should theoretically be the "cleanest" comparison, in that ignores non-legally significant differences like whitespace, capitalization, etc.

☝️ can be used to compare the cached license content to the detected license text and retain the existing license if a match is found

steps to reproduce

1. create all below files under dir x

Gemfile:

# frozen_string_literal:true
source "https://rubygems.org"
gem 'aws-sdk-core', '3.39.0'
gem 'licensed', :group => 'development'

Gemfile.lock:

GEM
  remote: https://rubygems.org/
  specs:
    addressable (2.5.2)
      public_suffix (>= 2.0.2, < 4.0)
    aws-eventstream (1.0.1)
    aws-partitions (1.142.0)
    aws-sdk-core (3.39.0)
      aws-eventstream (~> 1.0)
      aws-partitions (~> 1.0)
      aws-sigv4 (~> 1.0)
      jmespath (~> 1.0)
    aws-sigv4 (1.0.3)
    dotenv (2.7.1)
    faraday (0.15.4)
      multipart-post (>= 1.2, < 3)
    jmespath (1.4.0)
    licensed (2.0.1)
      bundler (>= 1.10)
      licensee (~> 9.10)
      pathname-common_prefix (~> 0.0.1)
      thor (~> 0.19)
      tomlrb (~> 1.2)
    licensee (9.11.0)
      dotenv (~> 2.0)
      octokit (~> 4.8.0)
      rugged (~> 0.24)
      thor (~> 0.19)
    multipart-post (2.0.0)
    octokit (4.8.0)
      sawyer (~> 0.8.0, >= 0.5.3)
    pathname-common_prefix (0.0.1)
    public_suffix (3.0.3)
    rugged (0.28.0)
    sawyer (0.8.1)
      addressable (>= 2.3.5, < 2.6)
      faraday (~> 0.8, < 1.0)
    thor (0.20.3)
    tomlrb (1.2.8)

PLATFORMS
  ruby

DEPENDENCIES
  aws-sdk-core (= 3.39.0)
  licensed

BUNDLED WITH
   2.0.1

licensed.yaml:

sources:
  bundler: true

2. navigate to directory x and run licensed cache -c licensed.yaml

Issue:

.licenses/bundler/aws-sdk-*.dep.yml are created with license set to none. ex:

---
name: aws-sdk-core
version: 3.46.1
type: bundler
summary: AWS SDK for Ruby - Core
homepage: http://github.com/aws/aws-sdk-ruby
license: none
licenses: []
notices: []

if one does gem specification aws-sdk-core you'd get yml containing licenses (since it is part of gemspec). i.e.

...
licenses:
- Apache-2.0
...

would it be advisable to change licensed to fallback to using gemspec to:

1. retrieve the license from the gemspec metadata
2. try to find the license text using the gem homepage . ex: http://github.com/aws/aws-sdk-ruby

or is this something that needs to be reported to licensee project to properly handle this case ?

Hi, I noticed a regression between version 1.2.0 and 1.1.0 when trying to run licensed status on a project which has a dependency on libv8:

/Users/User/.rvm/gems/ruby-2.5.1/gems/licensed-1.2.0/lib/licensed/source/bundler.rb:74:in `block in specs_for_dependencies': Unable to find a specification for libv8 (>= 6.3) in any sources (RuntimeError)

This error happens on both a mac + ubuntu with v1.2.0, but does not happen on v1.1.0

Replicating

With the following Gemfile:

# frozen_string_literal: true

source "https://rubygems.org"

git_source(:github) {|repo_name| "https://github.com/#{repo_name}" }

gem "mini_racer"

Gemfile.lock

GEM
  remote: https://rubygems.org/
  specs:
    libv8 (6.7.288.46.1)
    mini_racer (0.2.0)
      libv8 (>= 6.3)

PLATFORMS
  ruby

DEPENDENCIES
  mini_racer

BUNDLED WITH
   1.16.1

.licensed.yml

sources:
  rubygem: true

Running the following with licensed 1.2.0 currently fails:

licensed status

Version 1.2.0

/Users/User/.rvm/gems/ruby-2.5.1/gems/licensed-1.2.0/lib/licensed/source/bundler.rb:74:in `block in specs_for_dependencies': Unable to find a specification for libv8 (>= 6.3) in any sources (RuntimeError)

Version 1.1.0

licensed status
Checking licenses for example: 3 dependencies
FFF

Warnings:

/private/tmp/example/.licenses/rubygem/bundler.txt:
  - cached license data missing

/private/tmp/example/.licenses/rubygem/libv8.txt:
  - cached license data missing

/private/tmp/example/.licenses/rubygem/mini_racer.txt:
  - cached license data missing

3 dependencies checked, 3 warnings found.

Extra debugging

I can see that licensed is trying to check the following spec_path locations:

/Users/user/.rvm/gems/ruby-2.5.1/specifications/mini_racer-0.2.0.gemspec
/Users/user/.rvm/gems/ruby-2.5.1/specifications/libv8-6.7.288.46.1.gemspec

It's dying currently dying on libv8, but I can tell that the gemspec actually has an extra operating system extension that licensed wasn't expecting:

$ ls /Users/user/.rvm/gems/ruby-2.5.1/specifications/ | grep libv8
libv8-6.7.288.46.1-x86_64-darwin-16.gemspec

The file name extension is slightly different on Ubuntu too.

I assume even in development, one is still bound by the licenses included npm packages.

Licensed distributes all-in-one executables to make running licensed outside of a ruby environment possible. The executables are created using ruby-packer and builds an executable from a squashed filesystem containing ruby and licensed.

ruby-packer is currently using ruby version 2.4.X, which causes Bundler.specs_path to always evaluate to <gem path root>/ruby/2.4.0 - in my usages this was vendor/gems/ruby/2.4.0.

Licensed uses ::Bundler.specs_path to find specification files and load full information for each dependency.

When running bundle install with a ruby version that's not 2.4.x, specifications will not be under <gem path root>/ruby/2.4.0 and licensed will raise an error that specifications could not be found.

In the below example, jekyll was bundle install'd using ruby 2.5.0 and was present under vendor/gems/ruby/2.5.0.

/__enclose_io_memfs__/lib/ruby/gems/2.4.0/gems/licensed-1.3.0/lib/licensed/source/bundler.rb:74:in `block in specs_for_dependencies': Unable to find a specification for jekyll (>= 0) in any sources (RuntimeError)

Git submodules can currently be tracked with explicit listings in a manifest. The issue with using this solution is that the submodule version is tracked as the commit in the root git repository that added/changed the submodule rather than the version of the submodule itself.

Submodules are the native way to track dependencies in a git repository and they should have first class support. Submodule tracking already includes a version (commit SHA for the submodule) and a url to set as homepage. The only information that licensed tracks thats not freely available is a summary. The manifest source doesn't provide a summary either so there's no loss of information in changing to explicit submodule enumeration.

Hi,

I'm integrating licensed for a poc in our Jenkins pipeline. It could be great to get a report of all licenses checks in an XML formatted file; so we can process it using Jenkins plugins. For now, I have to analyse the XXX dependencies checked, XXX warnings found. string formatted output of licensed status command.

We can consider licensed results as tests results.

• If all licenses check passed; then the overall test is passed.
• If one license is found as unknown, or not matching any of allowed, ignored or reviewed configs; then the check/test failed.

Standards test output formats such as junit, nunit, mstest, google-test, etc. would be great cos already known by many CI tools. Junit is probably the most used one.

Organizing tests in "groups" matching the package type (npm, pip, go, etc.) and license type (mit, gpl...) would also allows to "count" the type of licenses found... providing some kind of statistics ^

Example of output in Jenkins test plugin results, processing the XML file could be:

Package                  Fail Skip Pass Total
- total                     0    0   69    69
- npm                       0    0   53    53
  - mit                     0    0   45    45
    - package1              0    0    1     1
    - package 2             ...
    - ...
  - apache-2.0                              3
    - package46
    - ...
  - bsd-3-clause                            1 
    - package49
  - isc                                     1
    - package50
  - unknown                                 3
    - package51             1    0    0     1
    - ...
- go                                       16
  - mit                                    14
    - ...
  - isc                                     1
    - ...
  - unknown                                 1
    - packagexx             1    0    0     1
    - ...

I did not fill the array; but you've got the point ;-)

Regards,

Chris

In the configuration docs, the sources config file section is shown as a list, not a dictionary. Following this pattern will give the following stack trace:

Traceback (most recent call last):
	20: from /usr/local/bundle/bin/licensed:23:in `<main>'
	19: from /usr/local/bundle/bin/licensed:23:in `load'
	18: from /usr/local/bundle/gems/licensed-1.0.0/exe/licensed:5:in `<top (required)>'
	17: from /usr/local/bundle/gems/thor-0.20.0/lib/thor/base.rb:466:in `start'
	16: from /usr/local/bundle/gems/thor-0.20.0/lib/thor.rb:387:in `dispatch'
	15: from /usr/local/bundle/gems/thor-0.20.0/lib/thor/invocation.rb:126:in `invoke_command'
	14: from /usr/local/bundle/gems/thor-0.20.0/lib/thor/command.rb:27:in `run'
	13: from /usr/local/bundle/gems/licensed-1.0.0/lib/licensed/cli.rb:24:in `status'
	12: from /usr/local/bundle/gems/licensed-1.0.0/lib/licensed/cli.rb:48:in `run'
	11: from /usr/local/bundle/gems/licensed-1.0.0/lib/licensed/command/status.rb:22:in `run'
	10: from /usr/local/bundle/gems/licensed-1.0.0/lib/licensed/command/status.rb:22:in `flat_map'
	 9: from /usr/local/bundle/gems/licensed-1.0.0/lib/licensed/command/status.rb:22:in `each'
	 8: from /usr/local/bundle/gems/licensed-1.0.0/lib/licensed/command/status.rb:23:in `block in run'
	 7: from /usr/local/bundle/gems/licensed-1.0.0/lib/licensed/command/status.rb:23:in `chdir'
	 6: from /usr/local/bundle/gems/licensed-1.0.0/lib/licensed/command/status.rb:24:in `block (2 levels) in run'
	 5: from /usr/local/bundle/gems/licensed-1.0.0/lib/licensed/command/status.rb:18:in `app_dependencies'
	 4: from /usr/local/bundle/gems/licensed-1.0.0/lib/licensed/configuration.rb:56:in `sources'
	 3: from /usr/local/bundle/gems/licensed-1.0.0/lib/licensed/configuration.rb:56:in `select'
	 2: from /usr/local/bundle/gems/licensed-1.0.0/lib/licensed/source/bundler.rb:12:in `enabled?'
	 1: from /usr/local/bundle/gems/licensed-1.0.0/lib/licensed/configuration.rb:62:in `enabled?'
/usr/local/bundle/gems/licensed-1.0.0/lib/licensed/configuration.rb:62:in `fetch': no implicit conversion of String into Integer (TypeError)

The README shows the correct type, though.

When licensed is run as bundle exec licensed ..., bundler sets BUNDLE_GEMFILE to the current Gemfile's path. This causes the bundler source to use the Gemfile that contains the licensed gem when evaluating dependencies when a different Gemfile would be found in the configuration source path.

There is already a fix for this available in test/source/bundler_test.rb that should be promoted to the source and used when finding dependencies.

I'm currently using licensed in my project. This project also has a dependency of the Webpacker gem, which includes an npm dependency of @rails/webpacker. fsevents is an optional dependency through the following dependency tree:

@rails/webpacker => webpack => watchpack => chokidar => fsevents

This means that licensed status will detect it. The problem is that fsevents is an optionalDependency in this particular NPM package. So if it fails to install, the package just falls back to IIRC, but licensed still wants it to be present. So I receive the output in this Gist.

I think, ideally, npm packages that are "optional" would be ignored by licensed. Adding it to ignored does not seem to work.

::Bundler.default_gemfile.basename is used in the bundler source when looking for a Gemfile in the applications source path.

The issue is that when licensed is run via bundle exec, Bundler sets the ENV["BUNDLE_GEMFILE"] variable which is given preference over searching for a Gemfile from the file system.

This fails when

• the application source path where dependencies are enumerated has a gems.rb file
• bundle exec runs licensed based on a Gemfile in a different location

In this scenario Bundler#gemfile_path will return $pwd/Gemfile which doesn't exist, rather than $pwd/gems.rb which does.

This seems like a low priority edge case, but might be fixable in combination with #23?

Dependency sources currently exclude non-production dependencies, e.g.:

It might be good to have an option to include all dependencies, should a user wish to use licensed to track development dependencies as well as production.

The status command checks license.text.strip.empty? when determining whether a cached license metadata file is missing license content. This can be a false-success in the following scenario

1. During licensed cache, a dependency has no license text but does have legal notices
2. During licensed cache, the dependency☝️ is saved with legal notices in license.text
3. During licensed status, the cached license has non-empty license.text (notice file contents). This results in a pass for the cached metadata even though license text is not in the cached metadata.

👋 @jonabc!

I have a project running licensed verify from ages and ages ago. This subcommand was renamed to status somewhere down the line, but we never updated. We never updated because we never noticed that verify wasn't working!

+ script/licensed verify
If you encounter rate-limiting errors, please set the GITHUB_TOKEN environment variable and try again.
Could not find command "verify".

Oops! I'm happily moving to status now that I noticed the issue, but am wondering if we could update licensed to exit 1 when we request a command that doesn't exist.

Thanks!