Git Product home page Git Product logo

npm-leech's Introduction

npm-leech

Small utility to leech all direct and transitive npm packages for a given package.json or package-lock.json file.

Ideal for mirroring a given set of packages to a private repository like JFrog Artifactory or NPM Registry.

    usage
      $ npm-leech [-i package.json|package-lock.json] [-o foo.tar] [-a] [-c] [-d] [-D] 

    options
      --artifactory, -a      Store scoped packages in a layout used by Artifactory
      --input, -i            source package.json or package-lock.json (default: ./package-lock.json)
      --output, -o           target tarballs tar (default: ./npm-tarballs.tar)
      --concurrency, -c      number of concurrent retrieval tasks for meta/pkg (default: 4)
      --dev, -d              leech devDependencies in source. (default: false)
      --transitive-dev, -D   CAUTION! leech all transitive devDependencies. (default: false)
      --registry, -r         NPM registry. (default: http://registry.npmjs.org/)
      --verbose, -v          Verbose output. (default: false)
      --progress, -p         Progress bar. Should not be used with -v (default: true)

    examples
      $ npm-leech -i ../../package.json -o foo.tar -c 8 -d

License

MIT - @larsw

npm-leech's People

Contributors

dependabot[bot] avatar gennon avatar larsw avatar roarbr avatar

Stargazers

avatar

Watchers

avatar avatar avatar

Forkers

mikaelbr gennon fredrick-tam pombredanne roarbr

npm-leech's Issues

npm-leech do not download dependencies with tilde (~) and caret (^) in versions

Hi,
While using npm-leech to download a package with a lot of dependencies, some dependencies failed to download with an error message like this:

pkg-err { Error: Request failed with status code 405
    at createError (/usr/lib/node_modules/npm-leech/node_modules/axios/lib/core/createError.js:16:15)
    at settle (/usr/lib/node_modules/npm-leech/node_modules/axios/lib/core/settle.js:17:12)
    at IncomingMessage.handleStreamEnd (/usr/lib/node_modules/npm-leech/node_modules/axios/lib/adapters/http.js:236:11)
    at emitNone (events.js:91:20)
    at IncomingMessage.emit (events.js:185:7)
    at endReadableNT (_stream_readable.js:978:12)
    at _combinedTickCallback (internal/process/next_tick.js:80:11)
    at process._tickCallback (internal/process/next_tick.js:104:9)
  config: 
   { url: 'http://registry.npmjs.org/accepts/~1.3.7',
...
     _header: 'GET /accepts/~1.3.7 HTTP/1.1\r\nAccept: application/json, text/plain, */*\r\nUser-Agent: axios/0.19.2\r\nHost: registry.npmjs.org\r\nConnection: close\r\n\r\n',
...
        responseUrl: 'http://registry.npmjs.org/accepts/~1.3.7',
        redirects: [],
        read: [Function] } },
  response: 
   { status: 405,
     statusText: 'Method Not Allowed',

The package.json used with npm-leech version 1.2.1:

  "name": "foo",
  "version:": "1.0.0",
  "dependencies": {
    "express": "latest"
  }
}

The package 'express' dependencies, only showing the two first:

npm show express --json dependencies
{
  "accepts": "~1.3.7",
  "array-flatten": "1.1.1",

The dependency 'array-flatten' has a "normal" semver and is downloaded without problems.
But the dependency 'accepts' has a tihlde in the semver. This causes npm-leech to download this version without resolving the tihlde to a "normal" semver first. But the registry does not understand "GET http://registry.npmjs.org/accepts/~1.3.7".

Using wireshark, this is the TCP stream when downloading the package 'accepts':

GET /accepts/~1.3.7 HTTP/1.1
Accept: application/json, text/plain, */*
User-Agent: axios/0.19.2
Host: registry.npmjs.org
Connection: close

HTTP/1.1 405 Method Not Allowed
Date: Wed, 05 May 2021 15:47:03 GMT
Content-Type: application/json
Content-Length: 63
Connection: close
Set-Cookie: __cfduid=db421dde183435235cd4cb62992f582121620229622; expires=Fri, 04-Jun-21 15:47:02 GMT; path=/; domain=.npmjs.org; HttpOnly; SameSite=Lax
CF-Ray: 64ab1ae7687ffac0-OSL
Allow: PUT
CF-Cache-Status: DYNAMIC
cf-request-id: 09decf24a30000fac0cd049000000001
Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
Server: cloudflare

{"code":"MethodNotAllowedError","message":"GET is not allowed"}

So, shouldn't npm-leech resolve tihlde and caret semvers into normal semvers before downloading packages?

CI/CD

  • GitHub actions build/test/publish to NPM

Option to store scoped packages in Artifactory layout

Hi,
In he npmjs.org registry scoped packages has a tarball path like this:

# npm info '@types/qs@latest' --json dist.tarball
"https://registry.npmjs.org/@types/qs/-/qs-6.9.6.tgz"

I.e. "@scope/packagename/-/packagename-version.tgz"
When downloaded by npm-leech, the file structure in the tar file is:

# tar tvf npm-tarballs.tar 
-rw-r--r-- 0/0            2405 2021-05-05 18:23 @types/qs-6.9.6.tgz

When this tar-file is imported to Artifactory using the web-interface, this layout from the tar file is kept in Artifactory. This is not the "correct" path inside Artifactory, it uses a path like this for scoped packages:

@scope/packagename/~/@scope/packagename-version.tgz

When importing the tar file from npm-leech into Artifactory, the incorrect layout will cause issues with scoped packages that also exist without a scope with the same name and version. Like "@types/qs" and "qs". It will cause checksum error when trying to install (npm install) a package like this using Artifactory as a registry. See this bug for more info:
https://www.jfrog.com/jira/browse/RTFACT-7668

Several issues like this is reported for Artifactory, but no response from the developer that I could see.

Pushing the packages to Artifactory using "npm publish" will go through the Artifactory API and store the file with the layout expected by Artifactory.

But I hoped this could be fixed in npm-leech by adding a option to store in Artifactory layout inside the tar file. This way the bulk import of the tar file from Artifactory web interface would work out of the box.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.