Git Product home page Git Product logo

lichen's Introduction

lichen 🍃

Go binary license checker. Extracts module usage information from binaries and analyses their licenses.

Features

  • Accurate module usage extraction (including transitive) from Go compiled binaries.
  • License files are resolved from local module storage.
  • Licenses are always checked against their respective versions.
  • Multi-license usage is covered out the box.
  • Local license checking using google/licenseclassifier.
  • Customisable output via text/template.
  • JSON output for further analysis and transforming into CSV, XLSX, etc.

Improvements over existing tooling

  • Some tools attempt to extract module use information from scanning code. This can be flawed, as transitive dependencies are not well represented (if at all). lichen executes go version -m [exes] to obtain accurate module usage information; only those that are required at compile time will be included. Also note that rsc/goversion has been avoided due to known issues in relation to binaries compiled with CGO enabled, and a lack of development activity.
  • Existing tools have been known to make requests against the GitHub API for license information. Unfortunately this can be flawed: the API only returns license details obtained from the HEAD of the master branch of a given repository. This also typically requires a GitHub API token to be available, as rate-limiting will kick in quite quickly. The GitHub API license detection doesn't offer any significant advantages; it itself simply uses licensee/licensee for license checking. lichen does not use the GitHub API at all.
  • In some instances, existing tools will clone the repository relating to the module. Often this is suffers from the same flaws as hitting the GitHub API, as the master branch ends up being inspected. Furthermore, some module URLs do not easily map to a git repository, resulting in the need for manual mapping in some instances. Finally, this process has a tendency to be slow. lichen takes advantage of Go tooling to retrieve the relevant file(s) in an accurate and time effective manner - go mod download is executed, and the local copy of the module is inspected for license information.

Install

go install github.com/uw-labs/lichen@latest

Note that Go must be installed wherever lichen is intended to be run, as lichen executes various Go commands (as discussed in the previous section).

Usage

By default lichen simply prints each module with its respective license. A path to at least one Go compiled binary must be supplied. Permitted licenses can be configured, along with overrides and exceptions (see Config).

lichen --config=path/to/lichen.yaml [binary ...]

Run lichen --help for further information on flags.

Note that the where lichen runs the Go executable, the process is created with the same environment as lichen itself - therefore you can set Go related environment variables (e.g. GOPRIVATE) and these will be respected.

Example

We can run lichen on itself:

$ lichen $GOPATH/bin/lichen
github.com/cpuguy83/go-md2man/[email protected]: MIT (allowed)
github.com/google/[email protected]: BSD-3-Clause (allowed)
github.com/google/[email protected]: Apache-2.0 (allowed)
github.com/hashicorp/[email protected]: MPL-2.0 (allowed)
github.com/hashicorp/[email protected]: MPL-2.0 (allowed)
github.com/lucasb-eyer/[email protected]: MIT (allowed)
github.com/mattn/[email protected]: MIT (allowed)
github.com/muesli/[email protected]: MIT (allowed)
github.com/russross/blackfriday/[email protected]: BSD-2-Clause (allowed)
github.com/sergi/[email protected]: MIT (allowed)
github.com/shurcooL/[email protected]: MIT (allowed)
github.com/urfave/cli/[email protected]: MIT (allowed)
golang.org/x/[email protected]: BSD-3-Clause (allowed)
gopkg.in/[email protected]: Apache-2.0, MIT (allowed)

...and using a custom template:

$ lichen --template="{{range .Modules}}{{range .Module.Licenses}}{{.Name | printf \"%s\n\"}}{{end}}{{end}}" $GOPATH/bin/lichen | sort | uniq -c | sort -nr
   8 MIT
   2 MPL-2.0
   2 BSD-3-Clause
   2 Apache-2.0
   1 BSD-2-Clause

Config

Configuration is entirely optional. If you wish to use lichen to ensure only permitted licenses are in use, you can use the configuration to specify these. You can also override certain defaults or force a license if lichen cannot detect one.

Example:

# minimum confidence percentage used during license classification
threshold: .80

# all permitted licenses - if no list is specified, all licenses are assumed to be allowed
allow:
  - "MIT"
  - "Apache-2.0"
  - "0BSD"
  - "BSD-3-Clause"
  - "BSD-2-Clause"
  - "BSD-2-Clause-FreeBSD"
  - "MPL-2.0"
  - "ISC"
  - "PostgreSQL"

# overrides for cases where a license cannot be detected, but the software is licensed
override:
  - path: "github.com/abc/xyz"
    version: "v0.1.0" # version is optional - if specified, the override will only apply for the configured version
    licenses: ["MIT"] # specify licenses

# exceptions for violations
exceptions:
  # exceptions for "license not permitted" type violations
  licenseNotPermitted:
    - path: "github.com/foo/bar"
      version: "v0.1.0" # version is optional - if specified, the exception will only apply to the configured version
      licenses: ["LGPL-3.0"] # licenses is optional - if specified only violations in relation to the listed licenses will be ignored
    - path: "github.com/baz/xyz"
  # exceptions for "unresolvable license" type violations
  unresolvableLicense:
    - path: "github.com/test/foo"
      version: "v1.0.1" # version is optional - if unspecified, the exception will apply to all versions

Credit

This project was very much inspired by mitchellh/golicense

Caveat emptor

Just as a linter cannot guarantee working and correct code, this tool cannot guarantee dependencies and their licenses are determined with absolute correctness. lichen is designed to help catch cases that might fall through the net, but it is by no means a replacement for manual inspection and evaluation of dependencies.

lichen's People

Contributors

antoninbas avatar bbkane avatar nick-jones avatar skitt avatar tymonx avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

lichen's Issues

Fails with Go 1.18 again

It appears to be failing on blank lines in the output of go version -m. This is the complete output:

2022/03/15 12:49:04 failed to evaluate licenses: unrecognised line: 	

New release?

Hi,

Would it be possible to tag a new release? Installing lichen on a system with Go 1.18 now results in a binary which fails in certain circumstances (#15). It’s possible to build a binary from master of course but being able to install directly would be much more convenient.

Regards,

Stephen

unrecognised version line: my-binary: devel +b7a85e0003

If a binary is built with a development build of Go, the go version -m ... output cannot be parsed. I observed this when trying go-licenses v2: https://github.com/Bobgy/go-licenses/blob/initial/v2/README.md

$ ~/git/go-licenses/v2/go-licenses csv
ListModulesInGoBinary(Path='my-binary'): unrecognised version line: my-binary: devel +b7a85e0003
$ go version -m my-binary    
my-binary: devel +b7a85e0003
        path    my-binary/cmd/my-binary
        mod     my-binary       (devel)
        dep     gopkg.in/yaml.v2        v2.4.0  h1:D8xgwECY7CYvx+Y2n4sBz93Jn9JRvxdiyyo8CTfuKaY=
$ go version
go version devel +b7a85e0003 linux/amd64

This doesn't occur when analyzing a binary built with a normal release of Go.

Vendoring support

Is there a way for this to support resolving modules from a vendor directory?

Including internal, closed source repos causes lookup failures

We have a private gitlab where we host our own go repos, and it seems, even with overrides or exceptions, I can't prevent the sum.golang.org lookup, which causes a failure -- thus preventing us from using this in CI. To be clear, our gitlab instance isn't accessible except on internal networks, so this lookup fails. See snippet for more details (I anonymized the URL, but I think you'll get the point).

$ (lichen -c lichen.yaml ./${PROJECT_NAME} | tee license_report.txt) || true
2021/07/22 15:04:44 failed to evaluate licenses: failed to fetch: exit status 1 (output: 
{
	"Path": "github.com/hashicorp/hcl",
	"Version": "v1.0.0",
	"Info": "/go/pkg/mod/cache/download/github.com/hashicorp/hcl/@v/v1.0.0.info",
	"GoMod": "/go/pkg/mod/cache/download/github.com/hashicorp/hcl/@v/v1.0.0.mod",
	"Zip": "/go/pkg/mod/cache/download/github.com/hashicorp/hcl/@v/v1.0.0.zip",
	"Dir": "/go/pkg/mod/github.com/hashicorp/[email protected]",
	"Sum": "h1:0Anlzjpi4vEasTeNFn2mLJgTSwt0+6sfsiTG8qcWGx4=",
	"GoModSum": "h1:E5yfLk+7swimpb2L/Alb/PJmXilQ/rhwaUYs4T20WEQ="
}
{
	"Path": "github.com/spf13/afero",
	"Version": "v1.3.4",
	"Info": "/go/pkg/mod/cache/download/github.com/spf13/afero/@v/v1.3.4.info",
	"GoMod": "/go/pkg/mod/cache/download/github.com/spf13/afero/@v/v1.3.4.mod",
	"Zip": "/go/pkg/mod/cache/download/github.com/spf13/afero/@v/v1.3.4.zip",
	"Dir": "/go/pkg/mod/github.com/spf13/[email protected]",
	"Sum": "h1:8q6vk3hthlpb2SouZcnBVKboxWQWMDNF38bwholZrJc=",
	"GoModSum": "h1:Ai8FlHk4v/PARR026UzYexafAt9roJ7LcLMAmO6Z93I="
}
# ...
# truncated ...
# ...
{
	"Path": "gitlab.ourcompany.com/software/whatever/thing",
	"Version": "v0.0.0-20210610205813-eb9a997ab73f",
	"Error": "gitlab.ourcompany.com/software/whatever/[email protected]: verifying go.mod: gitlab.ourcompany.com/software/whatever/[email protected]/go.mod: reading https://sum.golang.org/lookup/gitlab.ourcompany.com/software/whatever/[email protected]: 410 Gone\n\tserver response: not found: gitlab.ourcompany.com/software/whatever/[email protected]: unrecognized import path \"gitlab.ourcompany.com/software/whatever/thing\": https fetch: Get \"https://gitlab.ourcompany.com/software/whatever/thing?go-get=1\": dial tcp: lookup gitlab.everactive.com on 8.8.8.8:53: no such host",
	"Info": "/go/pkg/mod/cache/download/gitlab.ourcompany.com/software/whatever/thing/@v/v0.0.0-20210610205813-eb9a997ab73f.info"
}

Not sure if this is really a bug or a feature request, so I'll leave that up to you, but I would imagine maybe adding some sort of config option to ignore modules that start with a certain URL?

ignoreURL: gitlab.ourcompany.com

Or making it so that override directives don't query sum.golang.org at all.

Additional config for ignoring "unresolvable licence"

Hello and thank you for your work with lichen!

I’m currently in need of running lichen in environments where a substantial amount of "unresolvable licence" errors can happen (running it as part of a tool for other developers to use at my company). In my case returning an empty licence is a much better default than returning an error. And there might be others with similar needs so I suggest we add a config for this?

I have two different suggestions for additional configs:

A:

ignoreUnresolvableLicense: true
Would result in ´func ignoreUnresolvable()´ always returning true.

Or B:

unresolvableLicense:
 - regex: true 
 - path: "github.com/test/.*"

regex: true would result in unresolvableLicense's path beeing a regular expression.

Which of these suggestions would be best for lichen?

I can create a PR with this if you wan't to test it in practice, the additional code will be short.

unrecognised line: build -compiler=gc

I don't know why it's happening on fan2go

lichen /var/tmp/portage/sys-power/fan2go-0.6.2/image/usr/bin/fan2go
2022/05/07 05:08:29 failed to evaluate licenses: unrecognised line:     build   -compiler=gc

It returns an unexpected error when there are no modules used in the project

Lichen returns an unexpected error when there are no modules used in the project.

To reproduce it:

  1. Create a simple main.go file:
package main

import "fmt"

func main() {
 fmt.Println("Hi!")
}
  1. Create a simple go.mod file:
module my-module

go 1.15
  1. Build it:
go build
  1. Run lichen:
lichen ./my-module

Expected behavior:

  • success status code 0
  • no error messages

Current behavior:

  • error status code 1
  • error message: failed to evaluate licenses: failed to fetch: exit status 1 (output: go mod download: no modules specified (see 'go help mod download')

Fails with Go 1.18 dep lines instead of mod lines

At least in some circumstances, go version -m with Go 1.18 on a binary built with Go 1.18 outputs the current module as a dep line instead of the previous mod line, i.e.

        dep     github.com/submariner-io/submariner     (devel) 

instead of

        mod     github.com/submariner-io/submariner     (devel) 

lichen then fails with

{
        "Path": "github.com/submariner-io/submariner",
        "Version": "(devel)",
        "Error": "github.com/submariner-io/submariner@(devel): invalid version: reading https://proxy.golang.org/github.com/submariner-io/submariner/@v/%28devel%29.info: 410 Gone\n\ts
erver response: not found: github.com/submariner-io/submariner@(devel): invalid version: unknown revision (devel)"
}

Fails with Go 1.18

I get this error when scanning binaries built with Go 1.18 beta2.

2022/02/16 14:46:01 failed to evaluate licenses: unrecognised line: 	build	-compiler=gc

Allow overrides for unresolvable modules

Hi - I am working on replacing https://github.com/mitchellh/golicense with lichen for Project Antrea, since golicense is no longer actively maintained and lichen offers me support for Windows binaries.

I am running into one issue during module fetching:

2021/01/29 00:26:16 failed to evaluate licenses: failed to fetch: exit status 1 (output: go: cannot use relative path ../../@ to specify module

This is because Antrea uses a replace directive pointing to a local directory in one of our go.mod files: https://github.com/vmware-tanzu/antrea/blob/57dcaec0c561875dac6dee9bac72a66b9aefbab6/plugins/octant/go.mod#L14. This is a "trick" that enables us to ensure that plugins are built using the correct version of the "parent" Go module. And it works great for building.

When using golicense, I was able to provide an override ("../../": "Apache-2.0") in the config to handle that case. I'd like to be able to do something similar with lichen, but lichen applies unresolvableLicense config overrides after fetching modules and will error out early if a module cannot be fetched.

Would it be possible to provide a new exception type (e.g. moduleUnfetchable) in the config that would apply when a module cannot be fetched, instead of returning right away. Thanks!

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.