Is your feature request related to a problem? Please describe.

Follow up of
verdaccio/ui#139
verdaccio/verdaccio#1373
verdaccio/verdaccio#1408

Describe the solution you'd like

enable: feat: prevent secrets from leaking to source control

From #13, I would like to separate in different tasks the import of packages. I think the best separation is by the type of packages.

The purpose of this issue is track the import of plugin packages, composed by:

• @verdaccio/local-storage from @verdaccio/local-storage
• verdaccio-htpasswd from verdaccio-htpasswd
• verdaccio-google-cloud from verdaccio-google-cloud
• verdaccio-aws-s3-storage from verdaccio-aws-s3-storage
• verdaccio-memory from verdaccio-memory
• verdaccio-audit from verdaccio-audit
• verdaccio-auth-memory from verdaccio-auth-memory
• verdaccio-activedirectory from https://github.com/verdaccio/verdaccio-activedirectory

We should review if the current CircleCI config is adapted for the first publish of the monorepo.

Take care of next information:

1. Have a specific step for releasing with publish script.
2. When we execute the publish script, it must contains the --yes flag required by lerna publish script in CI

Create a package that contains a common Babel config for every Verdaccio project.

The syntax to include this preset in .babelrc files is:

{
  "presets": ["@verdaccio"]
}

We should allow users to customize the preset settings by enabling and disabling with some options. One example is activating React preset (for React projects) with:

{
  "presets": [
    ["@verdaccio", { "enableReact": true }]
  ]
}

One thing I want to discuss is if the package should be self-contained. This means that every project that install it will not need to install any Babel dependency (as this will include every preset but can be enabled or disabled with options object).

My reason:

I want to retrieve all packages from storage.

Refers to: verdaccio/verdaccio#706

Describe the bug
Verdaccio crashes (or rather verdaccio-memory I suppose)

To Reproduce
Launch /usr/bin/verdaccio --config /etc/verdaccio/config.yaml and create a lot of simultaneous requests (this is done by our CI pipeline launching multiple jobs simultaneously)

Expected behavior
No crash

Configuration File (cat ~/.config/verdaccio/config.yaml)

plugins: /usr/lib/node_modules

uplinks:
  npmjs:
    url: https://registry.npmjs.org/
    maxage: 24h

auth:
  htpasswd:
    file: ./htpasswd

web:
  title: Verdaccio

packages:
  '@*/*':
    access: $all
    publish: $authenticated
    unpublish: $authenticated
    proxy: npmjs

  '**':
    access: $all
    publish: $authenticated
    unpublish: $authenticated
    proxy: npmjs

logs:
  - { type: stdout, format: pretty, level: http }

store:
  memory:
    limit: 4096

listen:
  - "[::]:8080"

I tried playing around with store.memory.limit, but to no avail.

Debugging output

• $ NODE_DEBUG=request verdaccio display request calls (verdaccio <--> uplinks)
  out-request.txt

• $ DEBUG=express:* verdaccio enable extreme verdaccio debug mode (verdaccio api)
  out-extreme.txt

• $ npm -ddd prints:

npm verb cli [ '/usr/bin/node', '/usr/bin/npm', '-ddd' ]
npm info using [email protected]
npm info using [email protected]
npm timing npm:load:whichnode Completed in 2ms
npm timing config:load:defaults Completed in 2ms
npm timing config:load:file:/usr/lib/node_modules/npm/npmrc Completed in 1ms
npm timing config:load:builtin Completed in 2ms
npm timing config:load:cli Completed in 3ms
npm timing config:load:env Completed in 0ms
npm timing config:load:project Completed in 2ms
npm timing config:load:file:/root/.npmrc Completed in 0ms
npm timing config:load:user Completed in 0ms
npm timing config:load:file:/usr/etc/npmrc Completed in 1ms
npm timing config:load:global Completed in 1ms
npm timing config:load:validate Completed in 0ms
npm timing config:load:credentials Completed in 2ms
npm timing config:load:setEnvs Completed in 1ms
npm timing config:load Completed in 13ms
npm timing npm:load:configload Completed in 14ms
npm timing npm:load:setTitle Completed in 0ms
npm timing npm:load:setupLog Completed in 3ms
npm timing npm:load:cleanupLog Completed in 3ms
npm timing npm:load:configScope Completed in 0ms
npm timing npm:load:projectScope Completed in 1ms
npm timing npm:load Completed in 29ms
npm timing config:load:flatten Completed in 4ms
npm <command>

Usage:

npm install        install all the dependencies in your project
npm install <foo>  add the <foo> dependency to your project
npm test           run this project's tests
npm run <foo>      run the script named <foo>
npm <command> -h   quick help on <command>
npm -l             display usage info for all commands
npm help <term>    search for help on <term>
npm help npm       more involved overview

All commands:

    access, adduser, audit, bin, bugs, cache, ci, completion,
    config, dedupe, deprecate, diff, dist-tag, docs, doctor,
    edit, exec, explain, explore, find-dupes, fund, get, help,
    hook, init, install, install-ci-test, install-test, link,
    ll, login, logout, ls, org, outdated, owner, pack, ping,
    prefix, profile, prune, publish, rebuild, repo, restart,
    root, run-script, search, set, set-script, shrinkwrap, star,
    stars, start, stop, team, test, token, uninstall, unpublish,
    unstar, update, version, view, whoami

Specify configs in the ini-formatted file:
    /root/.npmrc
or on the command line via: npm <command> --key=value

More configuration info: npm help config
Configuration fields: npm help 7 config

[email protected] /usr/lib/node_modules/npm
npm verb exit 0
npm timing npm Completed in 217ms
npm info ok 

• $ npm config get registry prints:

https://registry.npmjs.org/

Additional context
Simply run in an LXD container running Ubuntu 20.04.

Is your feature request related to a problem?
I am using verdaccio-aws-s3-storage plugin to store my packages. So I see all my packages and verdaccio-db.json file on s3 but the htpasswd is present on my local. So when I attach my s3 bucket to other instance as my rescue plan if the instance goes down. Then user will have to create the account again. Which shouldn't be the case. So do we have any way to store my htpasswd file as well on s3.

Describe the solution you'd like
Update htpasswd file on s3 as well when new user is added.

Describe the bug
Couple of days back, I tried using verdaccio aws-s3-plugin. I attached an IAM role to the ec2 instance and added the config for s3 bucket. Unfortunately, when I tried to run the verdaccio program. I got an error saying "the specified key does not exist" and barely changed anything in my config.

To Reproduce
Steps to reproduce the behavior:

1. Go to 'verdaccio's config.yaml and add the configs to s3 bucket. Attach an instance role to ec2 instance with permission to assume role and the s3 bucket.'
2. Click on 'connect to ec2 instance, and run the verdaccio on ec2 instance'
3. Scroll down to 'the bottom'
4. See error

verdaccio
warn --- config file  - /home/ec2-user/.config/verdaccio/config.yaml
warn --- Plugin successfully loaded: verdaccio-aws-s3-storage
warn --- Plugin successfully loaded: verdaccio-memory
(node:3669) Warning: deprecate: multiple logger configuration is deprecated, please check the migration guide.
(Use `node --trace-warnings ...` to show where the warning was created)
warn --- Plugin successfully loaded: verdaccio-htpasswd
warn --- Plugin successfully loaded: verdaccio-audit
warn --- http address - http://0.0.0.0:4873/ - verdaccio/5.1.1
(node:3669) Warning: multiple addresses will be deprecated in the next major, only use one
error--- s3: [S3PackageManager _getData] aws The specified key does not exist.
error--- s3: [S3PackageManager _getData] The specified key does not exist.
error--- s3: [S3PackageManager readPackage] no such package available

Expected behavior
Should run without these errors like before

Configuration File (cat ~/.config/verdaccio/config.yaml)

plugins: ./plugins

web:
  title: Verdaccio
  darkMode: true

auth:
  htpasswd:
    file: ./htpasswd

uplinks:
  npmjs:
    url: https://registry.npmjs.org/

packages:
  '@*/*':
    # scoped packages
    access: $authenticated
    publish: name
    unpublish: name
    proxy: npmjs

  '**':
    access: $authenticated
    publish: name
    unpublish: name
    proxy: npmjs

server:
  keepAliveTimeout: 0

middlewares:
  audit:
    enabled: true

logs:
  - { type: stdout, format: pretty, level: http }
  
listen:
  - 0.0.0.0:4873

max_body_size: 1000mb

store:
  aws-s3-storage:
    bucket: bucket-name
    region: ap-southeast-2 # optional, will use aws s3's default behavior if not specified
  memory:
    limit: 1000

Debugging output

• $ NODE_DEBUG=request verdaccio display request calls (verdaccio <--> uplinks): same error
• $ DEBUG=express:* verdaccio enable extreme verdaccio debug mode (verdaccio api): same error
• $ npm -ddd prints:

npm verb cli   '-ddd' ]
npm info using [email protected]
npm info using [email protected]
npm timing npm:load:whichnode Completed in 0ms
npm timing config:load:defaults Completed in 2ms
npm timing config:load:builtin Completed in 6ms
npm timing config:load:cli Completed in 3ms
npm timing config:load:env Completed in 0ms
npm timing config:load:project Completed in 2ms
npm timing config:load:user Completed in 3ms
npm timing config:load:global Completed in 1ms
npm timing config:load:validate Completed in 2ms
npm timing config:load:credentials Completed in 2ms
npm timing config:load:setEnvs Completed in 1ms
npm timing config:load Completed in 22ms
npm timing npm:load:configload Completed in 23ms
npm timing npm:load:setTitle Completed in 0ms
npm timing npm:load:setupLog Completed in 1ms
npm timing npm:load:cleanupLog Completed in 3ms
npm timing npm:load:configScope Completed in 0ms
npm timing npm:load:projectScope Completed in 2ms
npm timing npm:load Completed in 33ms
npm timing config:load:flatten Completed in 2ms
npm <command>

Usage:

npm install        install all the dependencies in your project
npm install <foo>  add the <foo> dependency to your project
npm test           run this project's tests
npm run <foo>      run the script named <foo>
npm <command> -h   quick help on <command>
npm -l             display usage info for all commands
npm help <term>    search for help on <term>
npm help npm       more involved overview

All commands:

   access, adduser, audit, bin, bugs, cache, ci, completion,
   config, dedupe, deprecate, diff, dist-tag, docs, doctor,
   edit, exec, explain, explore, find-dupes, fund, get, help,
   hook, init, install, install-ci-test, install-test, link,
   ll, login, logout, ls, org, outdated, owner, pack, ping,
   prefix, profile, prune, publish, rebuild, repo, restart,
   root, run-script, search, set, set-script, shrinkwrap, star,
   stars, start, stop, team, test, token, uninstall, unpublish,
   unstar, update, version, view, whoami

Specify configs in the ini-formatted file:
   /Users/user/.npmrc
or on the command line via: npm <command> --key=value

More configuration info: npm help config
Configuration fields: npm help 7 config

[email protected] /Users/user/.nvm/versions/node/v10.16.0/lib/node_modules/npm
npm verb exit 0
npm timing npm Completed in 531ms
npm verb code 1

• $ npm config get registry prints:

http://ip_address_of_ec2_instance:4873/

I would like to move all plugin packages to the core repository in Verdaccio branch 5.x.

https://github.com/verdaccio/verdaccio/tree/5.x

• verdaccio-memory verdaccio/verdaccio#1974
• verdaccio-auth-memory verdaccio/verdaccio#1975
• verdaccio-audit verdaccio/verdaccio#1976
• verdaccio-aws-s3-storage verdaccio/verdaccio#1977
• verdaccio-google-cloud verdaccio/verdaccio#1978
• verdaccio-active-directory verdaccio/verdaccio#1981

The devDependency @types/activedirectory2 was updated from 1.2.0 to 1.2.1.

🚨 View failing branch.

This version is covered by your current version range and after updating it in your project the build failed.

@types/activedirectory2 is a devDependency of this project. It might not break your production code or affect downstream projects, but probably breaks your build or test tools, which may prevent deploying or publishing.

Your Greenkeeper Bot 🌴

Describe the bug
When external PRs triggers CircleCI run, the workflow fails trying to publish canary packages in our Verdaccio instance, because it tries to get the secret key for publishing. I want to migrate that part of the workflow to Github Actions and create a message in the PR with the canary version published.

Is your feature request related to a problem? Please describe.
Right now, due to some limitation if you are using aws-s3-storage plugin you can't publish a package greater than 100mb.
I have package that is about 250mb and I always end up getting socket hang up when I use the s3-plugin. When I was not using the plugin, I was able to publish this particular package by changing max-body-size in config but I guess this config does not apply when I am using the aws-s3-plugin.

Describe the solution you'd like
If an option like max-body-size could be given in aws-s3-plugin config it would be pretty good.

Describe alternatives you've considered
I tried using verdaccio-memory plugpin but I guess that does not work with aws-s3-plugin. I could not find anything else to remedy the issue.

Is your feature request related to a problem? Please describe.

optional-chaining and nullish-coalescing-operator do not come by default, we need include dependencies in babel-preset.

https://github.com/babel/babel/tree/master/packages/babel-plugin-proposal-optional-chaining
https://github.com/babel/babel/tree/master/packages/babel-plugin-proposal-nullish-coalescing-operator

https://www.npmjs.com/package/nano

Exist an old implementation here: https://github.com/verdaccio/verdaccio/tree/storage_pluggable

Example:

https://github.com/verdaccio/verdaccio-memory
https://github.com/verdaccio/local-storage

We need to explain better how to contribute to the project, a good guidelines about which commands contributors will need and the purpose, how to work with Lerna and packages (alltogether or separately), how to create and add new packages, how to flat devDependencies and how we manage dependencies.

Also, we think this project is a good point to start trying the new VSCode feature called Remote Development to give new contributors a environment with some great settings and plugins they would need to develop, but this will be done before and improved in next iterations with more features

Things to do:

• Refactor Contributor guide
• Explain Lerna + Yarn Workspaces and how we use them
• Explain useful Lerna commands
• Explain how to work with packages alltogether (example, how to run tests or lint before pushing to git)
• Explain how to work with packages in their simple unit (example, how to run compiler to see if my new feature in package x compiles)
• Explain how to create packages
• Explain how to import packages
• Explain how we manage dependencies, devDependencies and how to flatten devDeps
• (Optionally) Explain how to work with VSCode Remote Development

Describe the bug

The bug demostrates that verdaccio with s3 backend is stateful, with package-list cached in memory, and caused inconsistent and racing issues in a clustrer env. The solution is attached, and discussion are welcome.

I've setup a minimal cluster verdaccio deployment, using two verdaccio instances, s3 store and nginx as reverse proxy. The s3 plugin is slightly modified, but nothing really hit the core logic.

Before the test, I already have one package (com.littlebigfun.addressable-importer) in verdaccio.

# space is an alias for aws-cli

$ space ls s3://openupm/verdaccio/
                           PRE com.littlebigfun.addressable-importer/
2019-11-27 00:26:39        126 verdaccio-s3-db.json

$ space cp s3://openupm/verdaccio/verdaccio-s3-db.json -
{"list":["com.littlebigfun.addressable-importer"],"secret":"..."}

Let's publish another package (com.bastianblokland.enumgenerator) for testing.

$ npm --registry=my-registry publish
...
+ [email protected]

Logs show that the return code is 201, the publish is successful. The NotFoundError is harmless for new package. Notice the publish job is executed by verdaccio instance 0 (the log prefix tells).

0|verdaccio  |  info <-- 127.0.0.1 requested 'PUT /com.bastianblokland.enumgenerator'
0|verdaccio  |  error-=- s3: [S3PackageManager writeTarball headObject] { NotFoundError: no such package available
0|verdaccio  |  http <-- 201, user: openupm(156.236.113.121 via 127.0.0.1), req: 'PUT /com.bastianblokland.enumgenerator', bytes: 1683542/53

The added package is verified in S3.

$ space cp s3://openupm/verdaccio/verdaccio-s3-db.json -
{"list":["com.littlebigfun.addressable-importer","com.bastianblokland.enumgenerator"],"secret":"..."}

$ space ls s3://openupm/verdaccio/
                           PRE com.bastianblokland.enumgenerator/
                           PRE com.littlebigfun.addressable-importer/
2019-11-27 01:08:59        162 verdaccio-s3-db.json

Now the buggy part, let's curl the package list, twice. Notice that only the second call return the new added package.

# first pass - wrong
$ curl https://my-registry/-/verdaccio/packages
[
  {
    "name": "com.littlebigfun.addressable-importer",
    ...
  }
]
# second pass - correct
$ curl https://my-registry/-/verdaccio/packages
[
  {
    "name": "com.bastianblokland.enumgenerator",
    ...
  },
  {
    "name": "com.littlebigfun.addressable-importer",
    ...
  }
]

Logs show that the second correct curl result is from verdaccio instance 0, the one just executed the publish command. The incorrect curl result is from verdaccio instance 1. We can run it for multiple times, the result is the same. The verdaccio instance 1 never return the new added package.

1|verdaccio  |  info <-- 127.0.0.1 requested 'GET /-/verdaccio/packages'
1|verdaccio  |  http <-- 200, user: null(156.236.113.121 via 127.0.0.1), req: 'GET /-/verdaccio/packages', bytes: 0/4070
0|verdaccio  |  info <-- 127.0.0.1 requested 'GET /-/verdaccio/packages'
0|verdaccio  |  http <-- 200, user: null(156.236.113.121 via 127.0.0.1), req: 'GET /-/verdaccio/packages', bytes: 0/7787

This behavior seems implying that verdaccio has some sort of local cache in memory of package list (verdaccio-s3-db.json). So until I restart verdaccio instance 1, there's no way to notify verdaccio instance to refresh the cache. I haven't check the source code yet, so it is just my guessing. But if this is true, it means verdaccio is not scalable, can only run with one instance. Well this isn't my expectation when discussing with @juanpicado on verdaccio/verdaccio#1459 (comment), where I ask for the the behaviour of how to handle a shared package list in cluster env.

I need some time to read getLocalDatabase method of https://github.com/verdaccio/verdaccio/blob/dbf20175dc68dd81e52363cc7e8013e24947d0fd/src/lib/storage.ts, to figure it out. But please guide me if you think there's something obvious I missed.

To Reproduce
You will need a simliar deployment - two instances of verdaccio managed by pm2, s3 backend. Nginx isn't necessary.

Expected behavior
All verdaccio instances should return the latest package list right after new package added (or removed).

Configuration File (cat ~/.config/verdaccio/config.yaml)

storage: ./storage
plugins: ./plugins
max_body_size: 200mb
listen: 0.0.0.0:4873

server:
  keepAliveTimeout: 60

middlewares:
  audit:
    enabled: true

web:
  enable: true

auth:
  htpasswd:
    file: ./htpasswd
    max_users: -1

packages:
  '@*/*':
    # scoped packages
    access: $all
    publish: $authenticated
    unpublish: $authenticate

  '**':
    access: $all
    publish: $authenticated
    unpublish: $authenticated

store:
  aws-s3-storage:
    bucket: openupm
    region: sfo2
    endpoint: ...
    accessKeyId: ...
    secretAccessKey: ...
    s3ForcePathStyle: true
    keyPrefix: 'verdaccio/'
    tarballACL: public-read
    tarballEdgeUrl: ...

convert_to_local_tarball_url: false

Environment information

verdaccio: 4.3.4 (modified: verdaccio/verdaccio#1580)
s3-plugin: 8.4.2 (modified: #249)
My modifications are made for #250, which is not related to this bug.

Debugging output

• $ NODE_DEBUG=request verdaccio display request calls (verdaccio <--> uplinks)
• $ DEBUG=express:* verdaccio enable extreme verdaccio debug mode (verdaccio api)
• $ npm -ddd prints:
• $ npm config get registry prints:

Additional context

Node security wg is planning to create a dedicated repo for the known vulnerabilities, see
nodejs/security-wg#359 (comment)

Using that sources instead of redirecting to upstream npm could be a nice option for verdaccio-audit.

They appear normally in the Web UI, but don't exist when using this endpoint (example.com/-/all).

Hi

I am receiving following error on Verdaccio 4.8.1 with aws-s3-storage plugin

error--- unexpected error: Please reduce your request rate.
SlowDown: Please reduce your request rate.
at Request.extractError (/opt/verdaccio/node_modules/aws-sdk/lib/services/s3.js:837:35)
at Request.callListeners (/opt/verdaccio/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
at Request.emit (/opt/verdaccio/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
at Request.emit (/opt/verdaccio/node_modules/aws-sdk/lib/request.js:688:14)
at Request.transition (/opt/verdaccio/node_modules/aws-sdk/lib/request.js:22:10)
at AcceptorStateMachine.runTo (/opt/verdaccio/node_modules/aws-sdk/lib/state_machine.js:14:12)
at /opt/verdaccio/node_modules/aws-sdk/lib/state_machine.js:26:10
at Request.<anonymous> (/opt/verdaccio/node_modules/aws-sdk/lib/request.js:38:9)
at Request.<anonymous> (/opt/verdaccio/node_modules/aws-sdk/lib/request.js:690:12)
at Request.callListeners (/opt/verdaccio/node_modules/aws-sdk/lib/sequential_executor.js:116:18)

most probably caused by S3 throttling.

I'd like to have configurable exponential backoff to be able to work-around errors like this.

--
Max

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Datasource	Name	Replacement PR?
npm	@babel/plugin-proposal-class-properties
npm	@babel/plugin-proposal-export-namespace-from
npm	@babel/plugin-proposal-json-strings
npm	@babel/plugin-proposal-nullish-coalescing-operator
npm	@babel/plugin-proposal-numeric-separator
npm	@babel/plugin-proposal-object-rest-spread
npm	@babel/plugin-proposal-optional-chaining
npm	@babel/polyfill
npm	@types/marked
npm	babel-eslint

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• fix(deps): update all core dependencies (@babel/cli, @babel/core, @babel/node, @babel/plugin-proposal-decorators, @babel/plugin-proposal-function-sent, @babel/plugin-proposal-throw-expressions, @babel/plugin-transform-async-to-generator, @babel/plugin-transform-classes, @babel/plugin-transform-runtime, @babel/preset-env, @babel/preset-typescript, @babel/register, @babel/runtime, @types/lodash, @verdaccio/config, @verdaccio/core, @verdaccio/types, debug, snyk, typedoc)

Ignored or Blocked

These are blocked by an existing closed PR and will not be recreated unless you click a checkbox below.

• chore(deps): replace dependency babel-eslint with @babel/eslint-parser 7.11.0

Detected dependencies

• Check this box to trigger a request for Renovate to run again on this repository

Is your feature request related to a problem? Please describe.

The intention is to improve the performance of tarball files distribution by leveraging S3 integrated CDN service, like Amazon CloudFront.

Prerequisites

• Tarball files need be granted anonymous read permission, so the CDN service can fetch tarball files directly from S3.
  • PR #249: set ACL of tarball files
• The tarball serving endpoint (https://hostname/-/tarball.tgz) needs return a HTTP redirect (302) to the CDN URL.
  • PR #320 add types
  • PR verdaccio/verdaccio#1688 redirect tarball URL based on config.tarball_url_redirect

https://firebase.google.com/docs/storage/web/start

Example:

https://github.com/verdaccio/verdaccio-memory
https://github.com/verdaccio/local-storage

Create a package that contains a common Eslint (and Prettier) config for every Verdaccio project.

The syntax to include this config in .eslintrc files is:

{
  "extends": ["@verdaccio"]
}

We should allow users to customize the imported configs by enabling or disabling them with some options. This includes a different approach to @verdaccio/babel-preset-core because here we can add or remove Eslint rules directly.

There are some upgrades blocked due to our time. I want to share a table with the package, dependency to upgrade and, the current and target version.

I'm going to close the current PRs opened by @dependabot, but I will mention this issue to follow up.

Also, I'm investigating some of them to make this upgrade faster 😉

Describe the bug
8d7ca5b, the 8.4.1 release breaks plugins/aws-s3-storage, and perhaps other, due to @verdaccio/[email protected] and @verdaccio/[email protected] do not exist.

To Reproduce
Steps to reproduce the behavior:

1. Checkoust monorepo
2. Go to plugins/aws-s3-storage
3. npm install

npm ERR! code ETARGET
npm ERR! notarget No matching version found for @verdaccio/babel-preset@^8.4.1.
npm ERR! notarget In most cases you or one of your dependencies are requesting
npm ERR! notarget a package version that doesn't exist.
npm ERR! notarget
npm ERR! notarget It was specified as a dependency of 'aws-s3-storage'
npm ERR! notarget

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/verdaccio/.npm/_logs/2019-11-23T14_42_18_694Z-debug.log

My reason:

We use fs.exists at least twice in our source code, this has been deprecated, we should upgrade it.

https://nodejs.org/api/fs.html#fs_fs_exists_path_callback

I want to use this instead native streams. For core/streams

https://www.nearform.com/blog/welcome-readable-stream-3/

The idea is export ReadTarball and WriteTarball streams which are extended from PassThrough.

At Verdaccio we use directly the stream library, but I have the intention to force any plugin to use this library so we can use stream pipeline across all the ecosystem.

I want to add some debug capabilities to this module as well.

The Google Cloud plugin has the following message on the Readme:

⚠️⚠️ This plugin is experimental and might be unstable. It requires further testing. ⚠️⚠️

Is it known what further testing is required / how finished this plugin is?

Thanks

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior:

• npm install -g verdaccio verdaccio-google-cloud

Expected behavior
GCP Storage Bucket as Storage

**Configuration File **

storage: ./storage
plugins: ~/.npm-global/lib
web:
  title: NPM Registry
  gravatar: false
auth:
  htpasswd:
    file: ./htpasswd
    max_users: -1
security:
  api:
    jwt:
      sign:
        expiresIn: 60d
        notBefore: 1
  web:
    sign:
      expiresIn: 7d
      notBefore: 1
uplinks:
  npmjs:
    url: https://registry.npmjs.org/
packages:
  '@*/*':
    access: $authenticated
    publish: $authenticated
    unpublish: $authenticated
    proxy: npmjs
  '**':
    access: $authenticated
    publish: $authenticated
    unpublish: $authenticated
    proxy: npmjs
server:
  keepAliveTimeout: 60
middlewares:
  audit:
    enabled: true
logs:
  - { type: stdout, format: pretty, level: http }
listen: 'https://localhost:4430'
https:
  key: /home/cno/.config/verdaccio/verdaccio-key.pem
  cert: /home/cno/.config/verdaccio/verdaccio-cert.pem
  ca: /home/cno/.config/verdaccio/verdaccio-csr.pem
store:
  google-cloud:
   projectId: project-01 || env (GOOGLE_CLOUD_VERDACCIO_PROJECT_ID)
   kind: someRandonMetadataDatabaseKey
   bucket: my-bucket-name
   keyFilename: /path/project-01.json || env (GOOGLE_CLOUD_VERDACCIO_KEY)
   resumable: true

Debugging output

• $ NODE_DEBUG=request verdaccio display request calls (verdaccio <--> uplinks)

warn --- config file  - /home/cno/testing.yaml
 error--- plugin not found. try npm install verdaccio-google-cloud
(node:12738) UnhandledPromiseRejectionWarning: Error:
        verdaccio-google-cloud plugin not found. try "npm install verdaccio-google-cloud"
    at /home/cno/.npm-global/lib/node_modules/verdaccio/build/lib/plugin-loader.js:110:13
    at Array.map (<anonymous>)
    at loadPlugin (/home/cno/.npm-global/lib/node_modules/verdaccio/build/lib/plugin-loader.js:62:37)
    at LocalStorage._loadStorePlugin (/home/cno/.npm-global/lib/node_modules/verdaccio/build/lib/local-storage.js:853:47)
    at LocalStorage._loadStorage (/home/cno/.npm-global/lib/node_modules/verdaccio/build/lib/local-storage.js:838:26)
    at new LocalStorage (/home/cno/.npm-global/lib/node_modules/verdaccio/build/lib/local-storage.js:47:27)
    at Storage.init (/home/cno/.npm-global/lib/node_modules/verdaccio/build/lib/storage.js:64:25)
    at _default (/home/cno/.npm-global/lib/node_modules/verdaccio/build/api/index.js:126:17)
    at startVerdaccio (/home/cno/.npm-global/lib/node_modules/verdaccio/build/lib/bootstrap.js:48:22)
    at init (/home/cno/.npm-global/lib/node_modules/verdaccio/build/lib/cli.js:76:35)
(node:12738) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 2)
(node:12738) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

• $ DEBUG=express:* verdaccio enable extreme verdaccio debug mode (verdaccio api)
• $ npm -ddd prints:

npm info it worked if it ends with ok
npm verb cli [ '/usr/bin/node', '/usr/bin/npm', '-ddd' ]
npm info using [email protected]
npm info using [email protected]
npm verb config Skipping project config: /home/cno/.npmrc. (matches userconfig)

Additional context
Plugin is installed globally, but never found by verdaccio

Improve our Eslint config after adding some projects to this monorepo, as it is difficult to create a common settings if we don't use it.

After first publish of @verdaccio/babel-preset and @verdaccio/eslint-config we should use in two projects to look how we should improve them and test how we feel with monorepo.

The pacakges proposed are the described in #8 by @juanpicado:

• lock-file
• streams

After that, we can follow with

• htpasswd
• local-storage

You can advice that we are adding coverage scripts with codecov but ignoring them on CI. This is because there are some ways to manage Coverage in monorepos and we haven't taken care about it until we had completed some initial steps.
We are going to use Codecov to complete this, as it has a great support for monorepos

From #13, I would like to separate in different tasks the import of packages. I think the best separation is by the type of packages.

The purpose of this issue is track the import of tool packages, composed by:

• @verdaccio/eslint-config from verdaccio/eslint-config-verdaccio
• eslint-plugin-verdaccio from verdaccio/eslint-plugin-verdaccio
• @verdaccio/babel-preset from verdaccio/babel-preset

Is your feature request related to a problem? Please describe.
I'm using Verdaccio with the verdaccio-aws-s3-storage plugin in a dockerized setup (ECS Fargate). Since the bucket, key-prefix and region attributes don't support environment variables, I cannot deploy the same image in different environments.

Describe the solution you'd like
The solution could be simple as trying assume that the value corresponds to an environment variable, read that variable and use it as the value. If the environment is not set (empty) then fall back to current behaviour and just use the value as it is.
This behaviour would mimic how other Verdaccio plugins behave (e.g.: verdaccio-github-oauth-ui)

Describe alternatives you've considered
The alternative solution could be publishing -dev suffixed Docker images.

Additional context

All plugins and core packages were moved. Next step is remove them from master, write a note and allow a new branch 9.x for bug fixes and required patches until verdaccio 4 is not longer supported.

Originally posted by @juanpicado in #386 (comment)

Currently we don't comply with the current eslint rules we are encouraging for all organization projects. We need to clean this up.

✖ 198 problems (0 errors, 198 warnings)
  0 errors and 3 warnings potentially fixable with the `--fix` option.

plugin: google-cloud

Roadmap to integrate core packages

Stage 1

• eslint-config @verdaccio/eslint-config
• babel-preset @verdaccio/babel-preset

Stage 2

• streams @verdaccio/streams
• file-lock @verdaccio/file-lock
• types @verdaccio/types
• http-error @verdaccio/http-error

Stage 3

• local-storage @verdaccio/local-storage
• htpasswd-auth @verdaccio/htpasswd-auth (I'd suggest this renaming, but we can discuss it) cc: @ayusharma

After Phase 1 some packages should be published to be integrated into Stage 2 packages #9, if all goes well we should move to Phase 2, we repeat the process integrating Stage 1,2 packages into Stage 3, if all goes well we will move Stage 3 into core packages.

All packages tagged as @sergiohgz described here #8 (comment) with (v4.0.0-alpha.0?).

Final Stage

• Migrate all core packages into Verdaccio main project

I'd like to download files inside a package, similarly to how unpkg.com does for the public npm registry.
this would be trivial to implement if verdaccio had an option to store package contents uncompress in s3 (besides the .tgz file),

Describe the solution you'd like
have a config option to ask verdaccio to also store package contents in s3 uncompressed in a subfolder. ideally only for specific package namespaces.

Describe alternatives you've considered
this might be doable with an AWS lambda, subscribed to the creation of .tgz files, that at that time decompress and stores the files next to it.

Is your feature request related to a problem? Please describe.

I'd love to have @babel/plugin-syntax-top-level-await to be used

https://babeljs.io/blog/2019/11/05/7.7.0

parser.parse(inputCode, {
  plugins: ["topLevelAwait"]
});

The types documentation proposes to import definitions from scoped package @verdaccio/types which are other than convention.

import type {ILocalData, LocalStorage, Logger, Config} from '@verdaccio/types';
// instead of 
import type {ILocalData, LocalStorage, Logger, Config} from 'verdaccio';

I understand that this is due to the desire to avoid publication by DefinitelyTyped.

Since npm 6 there is another solution - package aliasing. Simply:
npm i -D @types/verdaccio@npm:@verdaccio/types@latest
which download @verdaccio/types and install it as @types/verdaccio.

Unfortunately, this solution won't work because the definition file has hard-coded declared namespace:

declare module '@verdaccio/types' {
  // declarations
}

which, in fact, is not necessarily - Typescript will take it from package name.

I submit a proposal to remove this wrapper and add to the documentation an alternative installation method.

Describe the bug
Using it make fails the build.

I tried to update verdaccio-audit and prettier is not being found.

Expected behavior
Dependencies should be included in any project as devDependencies.

Screenshots
If applicable, add screenshots to help explain your problem.

➜ yarn test              
yarn run v1.17.3
$ npm run lint && jest
npm WARN lifecycle The node binary used for scripts is /var/folders/z9/wb47bj7j51n8p9h07y_11wch0000gn/T/yarn--1564387204819-0.5912425069863878/node but npm is using /Users/jpicado/.nvm/versions/node/v10.15.0/bin/node itself. Use the `--scripts-prepend-node-path` option to include the path for the node binary npm was executed with.

> [email protected] lint /Users/jpicado/projects/@verdaccio/verdaccio-audit
> eslint . --ext .js,.ts

Error: Cannot find module 'prettier'
Occurred while linting /Users/jpicado/projects/@verdaccio/verdaccio-audit/jest.config.js:3
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:581:15)
    at Function.Module._load (internal/modules/cjs/loader.js:507:25)
    at Module.require (internal/modules/cjs/loader.js:637:17)
    at require (internal/modules/cjs/helpers.js:22:18)
    at Program (/Users/jpicado/projects/@verdaccio/verdaccio-audit/node_modules/eslint-plugin-prettier/eslint-plugin-prettier.js:162:26)
    at listeners.(anonymous function).forEach.listener (/Users/jpicado/projects/@verdaccio/verdaccio-audit/node_modules/eslint/lib/util/safe-emitter.js:45:58)
    at Array.forEach (<anonymous>)
    at Object.emit (/Users/jpicado/projects/@verdaccio/verdaccio-audit/node_modules/eslint/lib/util/safe-emitter.js:45:38)
    at NodeEventGenerator.applySelector (/Users/jpicado/projects/@verdaccio/verdaccio-audit/node_modules/eslint/lib/util/node-event-generator.js:251:26)
    at NodeEventGenerator.applySelectors (/Users/jpicado/projects/@verdaccio/verdaccio-audit/node_modules/eslint/lib/util/node-event-generator.js:280:22)
npm ERR! code ELIFECYCLE
npm ERR! errno 2
npm ERR! [email protected] lint: `eslint . --ext .js,.ts`
npm ERR! Exit status 2

We need a package to centralize the error handling to be able to share with all plugins.

ErrorCode
https://github.com/verdaccio/verdaccio/blob/master/src/lib/utils.js#L392
API_ERROR this might either part of '@verdaccio/http-error'; or '@verdaccio/constants';
https://github.com/verdaccio/verdaccio/blob/master/src/lib/constants.js#L68

Eg: On verdaccio-htpasswd we have the following

 if (!this.users[user]) {
     return cb(new Error('User not found'));
 }

• We are using a native Error object
• Error messages depends of plugin developer which create inconsistence over all plugins

I want to create the following approach in any plugin.

import ErrorCode, {API_ERROR} from '@verdaccio/http-error';
...
 if (!this.users[user]) {
    return cb(ErrorCode.getNotFound(API_ERROR.USER_NOT_FOUND));
 }

In that way plugin developers can reuse the same messages and error handling is easier in the main project.

I created a PR to DefinetlyTyped to add the types needed to remove // @ts-ignore, following the definition shown in activedirectory2 documentation.

Originally posted by @sergiohgz in #64 (comment)

From #13, I would like to separate in different tasks the import of packages. I think the best separation is by the type of packages.

The purpose of this issue is track the import of core packages, composed by:

• @verdaccio/types from verdaccio/flow-types
• @verdaccio/file-locking from verdaccio/file-locking
• @verdaccio/streams from verdaccio/streams
• @verdaccio/readme from verdaccio/readme
• @verdaccio/commons-api from verdaccio/commons-api

Is your feature request related to a problem? Please describe.
The plugin is focused on create new JSX rules. With this purpose, the plugin is only required by @verdaccio/ui-theme, so it should not be provided by @verdaccio/eslint-config and the project could be linted with our common config.

Describe the solution you'd like

1. Remove eslint-plugin-verdaccio as dependency in @verdaccio/eslint-config
2. Add @verdaccio/eslint-config as devDependency in eslint-plugin-verdaccio
3. Clean settings from eslint-plugin-verdaccio and fix files with new config

Describe alternatives you've considered NA

Additional context
Remember this will be a Breaking Change in @verdaccio/eslint-config, as it won't provide eslint-plugin-verdaccio anymore, so users needs to add explicitly

I'm new to verdaccio, and this could be user-error.
When running "npm audit" in my dev-project, it works fine if the registry is set to "https://registry.npmjs.org"
I get the following error when running "npm audit" after running "npm set registry https://" -
**npm ERR! code ENOAUDIT
npm ERR! audit Your configured registry () does not support audit requests, or the audit endpoint is temporarily unavailable.

npm ERR! A complete log of this run can be found in:
npm ERR! /Users/**
This error is new, and wasn't there before I upgraded verdaccio to the latest release, so not sure what I'm missing

Is your feature request related to a problem?
Currently, the s3 plugin adds all packages (proxy or none proxy) flat out in the root of s3 bucket.
Describe the solution you'd like
Ideally, the storage: custom_folder for every package regex should also work for the s3.

'@*/*':
access: $all
publish: $all
proxy: npmjs
storage: custom_folder

this would cache the packages to s3/bucket/custom_folder

Rationale
we want to have our private packages persistently in S3, and cache npmjs packages there as well. But be able to clear the whole cache at regular intervals without touching our own private packages. Having these two separated makes maintenance easier.

Alternatives
For now we have been setting cache:false for npmjs uplinks to avoid polluting our s3 instance

Additional context
I would be able to make a PR on this, but I have a couple of questions in this regard:

• Do plugins get instantiated per package section? if thats the case the key prefix in aws-s3-storage can be utilized to achieve the goal
• currently, when we are setting the store to aws-s3-storage, local storage seems to be automatically turned off. How does this happen architecturally? Is s3-plugin turning the local storage off? Or is this simply a bug?

Is your feature request related to a problem? Please describe.

require.requireActual('babel/polyfill');

Is deprecated

Describe the solution you'd like
Find a new API that solve this.

Describe alternatives you've considered

No alternative yet.

Additional context

Vision of this repository

This repository aims to help us to reduce the maintenance cost we are paying for having a modularity approach, furthermore, we are aware we can do more in terms of modularity to achieve a more complete plugin system at Verdaccio 5.

Plugin ecosystem is important at Verdaccio, it creates a variety of alternatives and we want to help the developers to achieve the goal to ship a plugin.

Why we need this?

We have a big codebase with several projects and repositories. Maintain all of these repositories is a hard work that stops us evolving them.

We thought in to create some monorepos to maintain these projects, unify their architecture and align technologies like Babel, Eslint or Prettier.

The new repository architecture will be:

• core: Core packages are those are shipped within verdaccio or a shareable module for plugins

  • types: Typescript and Flow Types
  • file-locking
  • streams
  • readme
  • Future packages
    • @verdaccio/http-errors : http-erros is a module we use widely in plugins, we want this module create consistency in error messages across all plugins

• plugins: Any plugin is under the Verdaccio organization umbrella and maintenance

  • @verdaccio/local-storage
  • verdaccio-htpasswd
  • verdaccio-google-cloud
  • verdaccio-aws-s3-storage
  • verdaccio-memory: This repo will be migrated from
  • verdaccio-audit
  • verdaccio-auth-memory

• tools: Tools holds dependencies aims to be devDependencies or another sort of tools.

  • eslint-config-verdaccio: With package name @verdaccio/eslint-config, it will keep the main Eslint and Prettier settings for every Verdaccio project
  • eslint-plugin-verdaccio: Custom eslint rules
  • babel-preset: this contains a customizable preset to reuse in our projects

Packages are not going to be part of this monorepo

• verdaccio/verdaccio: Verdaccio project should remain as it is, we want fast deployments, since the Dockerimage is embedded within the project we want all dependencies ready to be bundled and shipped.
• verdaccio/ui: User Interface has a big amount of devDependencies, it is extremely complex in testing and we might use e2e testing at some point, the development is different of the others repositories.
• Plugins Generator
  • generator-verdaccio-plugin

Getting started with Development

• Define the stack (at this point we know we want to use Lerna, the current master uses yarn as PoC.
• Lerna setup and scripts
• Deployment flow: This involves setup CircleCi
• Use Verdaccio for E2E 😎 (of course we have to)
• Define labels and bots for simplifying maintenance
• 🤔 what' else? (please comment below)

I want to remark this was an @sergiohgz idea, in 2017 we had a lunch ☕️ at Madrid and a chat about this, at that point the code base was hard to unify. Today the situation is different, we have created several modules and migrate almost all listed above repositories to a similar codebase and stack, this will make migration really simple.

This is a task in the long term, we want to migrate repositories progressively, I'd suggest starting for smaller ones and test deeply the deployment.

Describe the bug
node-fetch is at v2.6.0 and should be updated to v2.6.1

To Reproduce
Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Docker || Kubernetes (please complete the following information):

• Docker verdaccio tag: [e.g. verdaccio:beta]
• Docker commands [e.g. docker pull ...]
• Docker Version [e.g. v18.05.0-ce-rc1]

Configuration File (cat ~/.config/verdaccio/config.yaml)

Debugging output

• $ NODE_DEBUG=request verdaccio display request calls (verdaccio <--> uplinks)
• $ DEBUG=express:* verdaccio enable extreme verdaccio debug mode (verdaccio api)
• $ npm -ddd prints:
• $ npm config get registry prints:

Additional context
Add any other context about the problem here.