siddharthkp / cost-of-modules Goto Github PK

Just wondering if there is any plans to support mono-repos (or just nested node modules in general) ?

Hi,
I have been trying out this module and every time I run it, I am getting the following output.
Any idea what might be wrong?
Node: v6.9.1
npm: 3.10.8
yarn: 0.16.1

│ name                                    │ children     │ size  │
├─────────────────────────────────────────┼──────────────┼───────┤
│ babel-cli                               │ 181          │ NaNM  │
├─────────────────────────────────────────┼──────────────┼───────┤
│ underscore                              │ 0            │ NaNM  │
├─────────────────────────────────────────┼──────────────┼───────┤
│ babel-loader                            │ 13           │ NaNM  │
├─────────────────────────────────────────┼──────────────┼───────┤
│ babel-plugin-syntax-flow                │ 0            │ NaNM  │
├─────────────────────────────────────────┼──────────────┼───────┤
│ babel-plugin-transform-builtin-extend   │ 0            │ NaNM  │
├─────────────────────────────────────────┼──────────────┼───────┤
│ babel-plugin-transform-flow-strip-types │ 0            │ NaNM  │
├─────────────────────────────────────────┼──────────────┼───────┤
│ babel-preset-es2015                     │ 38           │ NaNM  │
├─────────────────────────────────────────┼──────────────┼───────┤
│ babel-preset-react                      │ 6            │ NaNM  │
├─────────────────────────────────────────┼──────────────┼───────┤
│ babel-preset-stage-0                    │ 30           │ NaNM  │
├─────────────────────────────────────────┼──────────────┼───────┤
│ babel-register                          │ 5            │ NaNM  │
├─────────────────────────────────────────┼──────────────┼───────┤

Error: Cannot parse config file: 'C:\DEVCONNECTOR\config\default.json': SyntaxError: Unexpected end of JSON input
at Config.util.parseFile (C:\DEVCONNECTOR\node_modules\config\lib\config.js:794:11)
at C:\DEVCONNECTOR\node_modules\config\lib\config.js:601:26
at Array.forEach ()
at Config.util.loadFileConfigs (C:\DEVCONNECTOR\node_modules\config\lib\config.js:600:16)
at new Config (C:\DEVCONNECTOR\node_modules\config\lib\config.js:116:27)
at Object. (C:\DEVCONNECTOR\node_modules\config\lib\config.js:1441:31)
at Module._compile (internal/modules/cjs/loader.js:1158:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:1178:10)
at Module.load (internal/modules/cjs/loader.js:1002:32)
at Function.Module._load (internal/modules/cjs/loader.js:901:14)
at Module.require (internal/modules/cjs/loader.js:1044:19)
at require (internal/modules/cjs/helpers.js:77:18)
at Object. (C:\DEVCONNECTOR\config\db.js:2:16)
at Module._compile (internal/modules/cjs/loader.js:1158:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:1178:10)
at Module.load (internal/modules/cjs/loader.js:1002:32)
[nodemon] app crashed - waiting for file changes before starting...

It would be nice to have an option to ignore optionalDependencies, which apparently are included with the invoked npm install --production that gets triggered.

Here's the output for a project that I'm working on:

> cost-of-modules --less --no-install

Calculating...


┌──────────────┬───────────────┬──────────┐
│ name         │ children      │ size     │
├──────────────┼───────────────┼──────────┤
│ @uber        │ 2458          │ 1347.22M │
├──────────────┼───────────────┼──────────┤
│ @uber        │ 873           │ 1334.27M │
├──────────────┼───────────────┼──────────┤
│ @uber        │ 274           │ 1326.62M │
├──────────────┼───────────────┼──────────┤
│ @uber        │ 90            │ 1324.39M │
├──────────────┼───────────────┼──────────┤
│ @uber        │ 158           │ 1314.70M │
├──────────────┼───────────────┼──────────┤
│ @uber        │ 7             │ 1314.66M │
├──────────────┼───────────────┼──────────┤
│ @uber        │ 11            │ 1314.62M │
├──────────────┼───────────────┼──────────┤
│ @uber        │ 10            │ 1314.62M │
├──────────────┼───────────────┼──────────┤
│ @uber        │ 15            │ 1314.62M │
├──────────────┼───────────────┼──────────┤
│ @uber        │ 12            │ 1314.62M │
├──────────────┼───────────────┼──────────┤
│ + 39 modules │               │          │
├──────────────┼───────────────┼──────────┤
│ 49 modules   │ 1032 children │ 1400.34M │
└──────────────┴───────────────┴──────────┘

https://yarnpkg.com/features/pnp

👋 would you accept at PR that adds a flag to only output the total size? I want to run this module from in a CI task and fail if the total size goes about a certain number. Couldn't find any module that does this out of the box.

👋 I'm using this package together with SvelteKit. Since it uses a compiler + bundler approach, most dependencies are put in devDependencies because they get bundled during build and are not required at runtime. The --include-dev flag works great, but there are also some dependencies that aren't very useful to see, like typescript for example - nothing can be done about this dependency. So it would be great if it was possible to add --ignore=typescript,eslint to remove those packages from the list to reduce clutter.

Hi, I've tried to run cost-of-modules on our company project and our private dependency size was counted as NaNM

First of all I love this utility, thanks for making!

I am getting the following error:

SyntaxError: Unexpected end of JSON input
    at JSON.parse (<anonymous>)
    at getDependencyTree (/Users/me/.config/yarn/global/node_modules/cost-of-modules/lib/helpers.js:64:17)
    at Object.getRootDependencies (/Users/me/.config/yarn/global/node_modules/cost-of-modules/lib/helpers.js:73:26)
    at Object.<anonymous> (/Users/me/.config/yarn/global/node_modules/cost-of-modules/lib/index.js:26:32)
    at Module._compile (module.js:569:30)
    at Object.Module._extensions..js (module.js:580:10)
    at Module.load (module.js:503:32)
    at tryModuleLoad (module.js:466:12)
    at Function.Module._load (module.js:458:3)
    at Function.Module.runMain (module.js:605:10)

My package.json is:

{
  "name": "gtest",
  "version": "1.0.0",
  "main": "index.js",
  "license": "MIT",
  "devDependencies": {
    "gulp": "gulpjs/gulp#4.0",
    "gulp-cli": "^1.3.0"
  }
}

I am using npm 5.0.0, and I suspect that the issue relates to npm ls --json printing errors after printing the json (note that sync-exec channels both STDOUT and STDERR and returns the combination).

You should probably use:

const {spawnSync} = require('child_process')
const proc = spawnSync('npm', ['ls', '--json'])
console.log(JSON.parse(proc.stdout.toString()))
// ignore proc.stderr

Erroneous "JSON" output (note STDERR at end):

{
  "name": "gtest",
  "version": "1.0.0",
  "problems": [
    "invalid: [email protected] /Users/me/Desktop/gtest/node_modules/gulp",
    "extraneous: [email protected] /Users/me/Desktop/gtest/node_modules/glob-watcher",
    "extraneous: [email protected] /Users/me/Desktop/gtest/node_modules/undertaker",
    "extraneous: [email protected] /Users/me/Desktop/gtest/node_modules/vinyl-fs",
    "extraneous: [email protected] /Users/me/Desktop/gtest/node_modules/node-pre-gyp"
  ],
  "dependencies": {
    "...": "..."
  }
}
npm ERR! invalid: [email protected] /Users/me/Desktop/gtest/node_modules/gulp
npm ERR! extraneous: [email protected] /Users/me/Desktop/gtest/node_modules/glob-watcher
npm ERR! extraneous: [email protected] /Users/me/Desktop/gtest/node_modules/undertaker
npm ERR! extraneous: [email protected] /Users/me/Desktop/gtest/node_modules/vinyl-fs
npm ERR! extraneous: [email protected] /Users/me/Desktop/gtest/node_modules/node-pre-gyp

After running cost-of-modules on one of the projects, the largest dependency turned out to be babel-preset-es2015 with the size of 320.52M :

Although after running cost-of-modules on the clean repo with only babel-preset-es2015 dependency the size was mere 14.55M :

I guess something is wrong with the way size is calculated.

You have this listed on your todo list: "Check if there are any dev dependencies that are specified as dependencies (tricky)"

I have a package, lint-deps that would be able to do this once I get the refactor pushed up (it's finished but I need to do unit tests). As it is currently, lint-deps is really geared around CLI usage, but the refactor will make it easier to use via API, so you would be able to run it on each package in the dependency tree to get a report back.

just a thought, in case you're interested in this (fwiw, after using lint-deps for a while I've found that libs have unused deps way more often than I would have thought)

Hi there,

Excited to be integrating this tool, so thank you for your work. I found while playing out with a clone of this repo that npm test results in two failing tests on master, without my having written any new code.

   1. 0prod › get size for node_modules
   [ { name: '@whitneyit/data-fn', size: 7 },
  { name: 'date-time', size: 4 },
  { name: 'is', size: 56 },
  { name: 'node.extend' === [ { name: '@whitneyit/data-fn', size: 7 },
  { name: 'date-time', size: 4 },
  { name: 'is', size: 55 },
  { name: 'node.extend'
      { name: 'node.extend'
        Test.fn (fixtures/tests.js:41:11)
        _combinedTickCallback (internal/process/next_tick.js:67:7)
        process._tickCallback (internal/process/next_tick.js:98:9)


   2. 1dev › get size for node_modules
   [ { name: '@whitneyit/data-fn', size: 7 },
  { name: 'camelcase', size: 5 },
  { name: 'date-time', size: 4 },
  { name: 'is', s === [ { name: '@whitneyit/data-fn', size: 7 },
  { name: 'camelcase', size: 5 },
  { name: 'date-time', size: 4 },
  { name: 'is', s
      { name: 'is', s
        Test.fn (fixtures/tests.js:41:11)
        _combinedTickCallback (internal/process/next_tick.js:67:7)
        process._tickCallback (internal/process/next_tick.js:98:9)

This makes it hard to know where to start when contributing.

Thanks for your work and your time!

Edit: Running on El Capitan OSX 10.11.6 and node v.6.2.1, was also able to reproduce on my work machine, which has similar specs.

--
Brekk

I have just tried and it seems it prints out the size of the entire package, which is not very usefull, as not everything from a package is really used.

Could we add information of sizes from ever package which really end up in the build?

In my app (based on angular-seed), the runtime dependencies are installed by bower install which is triggered by npm script postinstall, and not detected by cost-of-modules. bower.json declares the runtime dependencies, and .bowerrc declares the directory where the dependencies are installed (the default directory is bower_components/).

I realize that this request may lead to a vortex of unreachable goals, such as a deep understanding of the impact of specific npm scripts, obscure options in bower, or supporting other package managers. Perhaps each of those can be prioritized and tackled independently. AFAIK, using bower for runtime dependencies is a popular practice.

edit: and fwiw, I'm sorry if I'm getting hung up on a language nuance. I know that "cost" technically implies size here. But given the debates about this topic atm, I think that could just be clarified in a brief sentence up front, to avoid misunderstandings. Specifically because by having a tool to identify "bloat", you're making some kind of statement about the impact of having lots of modules. It would be great to see why that "bloat" actually matters. hope that makes sense

This just shows the "size" of modules, which these days is obviously negligible and completely transparent to devs. Do you have an information on how size of modules, or the number of files negatively correlates to "cost"?

For instance, anything that describes how these things cause pain for developers, or makes it harder to program or maintain modules would be great. It would also be great to see a comparison of the cost of using highly unit testable and maintainable modules compares to the cost of under-tested monolithic modules that needless re-create the same code that is already well-tested elsewhere. Also, given project A, which has 10 "microlibs", each with a single responsibility (which has the disadvantage of them being in separate node_modules folder), versus project B, which has the same code as project A, maybe even in separate files, but in one node_modules folder), what is the average level of code coverage in "project B" versus "project A"? What is the average number of total outstanding bugs per SLOC in project B versus project A?

Other more specific questions:

• what is the impact to a library when a monolithic library they depend on stops being maintained?
• what is the impact to that library when a microlibs with a single purpose stops being maintained?
• what is the impact of having extra readme's or license files in node_modules?

I'd love to see some actual facts and data in the readme about this!

Is it normal for this script to blow away dev dependencies? I think when it runs npm install --production, for some reason, any dev dependencies in node_modules are gone.

It would be so good if this lib could show sizes of modules in browserify bundles. (or maybe it can already- if so how..?)

Ran cost-of-modules in a node project directory. Got this error:

Just tried it today on Your OS is Windows 10 64-bit

cost-of-modules --no-install --less

Calculating...


┌─────────────────────────────┬──────────────┬───────┐
│ name                        │ children     │ size  │
├─────────────────────────────┼──────────────┼───────┤
│ react-datepicker            │ 4            │ NaNM  │
├─────────────────────────────┼──────────────┼───────┤
│ sticky-table-headers        │ 0            │ NaNM  │
├─────────────────────────────┼──────────────┼───────┤
│ d3                          │ 0            │ NaNM  │
├─────────────────────────────┼──────────────┼───────┤
│ envify                      │ 21           │ NaNM  │
├─────────────────────────────┼──────────────┼───────┤
│ extract-text-webpack-plugin │ 9            │ NaNM  │
├─────────────────────────────┼──────────────┼───────┤
│ flux-standard-action        │ 5            │ NaNM  │
├─────────────────────────────┼──────────────┼───────┤
│ history                     │ 5            │ NaNM  │
├─────────────────────────────┼──────────────┼───────┤
│ http-server                 │ 19           │ NaNM  │
├─────────────────────────────┼──────────────┼───────┤
│ keypress.js                 │ 0            │ NaNM  │
├─────────────────────────────┼──────────────┼───────┤
│ lodash                      │ 0            │ NaNM  │
├─────────────────────────────┼──────────────┼───────┤
│ + 26 modules                │              │       │
├─────────────────────────────┼──────────────┼───────┤
│ 36 modules                  │ 238 children │ 0.00M │
└─────────────────────────────┴──────────────┴───────┘

Hi,

The npm audit command returns the warning:

┌───────────────┬──────────────────────────────────────────────────────────────┐
│ Low           │ Prototype Pollution                                          │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Package       │ lodash                                                       │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Patched in    │ >=4.17.5                                                     │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Dependency of │ cost-of-modules [dev]                                        │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Path          │ cost-of-modules > cli-table2 > lodash                        │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ More info     │ https://nodesecurity.io/advisories/577                       │
└───────────────┴──────────────────────────────────────────────────────────────┘

Migrate to cli-table3 (merging PR #52 ?) should do the trick.

Hi @siddharthkp, thank you for good package.
I think your package shouldn't be bound to command line interface and can be decomposed to API and CLI, to allow users write their own interfaces.
What do you think about this?

I use yarn to install my packages and having cost-of-modules run npm install --production is kind of annoying. I'd like to be able to say: cost-of-modules --no-install to skip that part.

$ npm test

> [email protected] test D:\Repositories\cost-of-modules
> npm run build && npm run build-test && npm run ava

> [email protected] build D:\Repositories\cost-of-modules
> babel src -d dist

src\helpers.js -> dist\helpers.js
src\index.js -> dist\index.js

> [email protected] build-test D:\Repositories\cost-of-modules
> babel test/src -d test/dist

test\src\0prod.js -> test\dist\0prod.js
test\src\1dev.js -> test\dist\1dev.js
test\src\fixtures\dev-test-data.js -> test\dist\fixtures\dev-test-data.js
test\src\fixtures\helpers.js -> test\dist\fixtures\helpers.js
test\src\fixtures\test-data.js -> test\dist\fixtures\test-data.js
test\src\fixtures\tests.js -> test\dist\fixtures\tests.js

> [email protected] ava D:\Repositories\cost-of-modules
> ava test/dist/*.js -s --no-cache --verbose

Making sure dependencies are installed
npm install --production

npm WARN [email protected] No repository field.

up to date in 0.076s

  - 0prod » setup was complete
  × 0prod » get size for node_modules [ { name: '@siddharthkp/empty', size: 0 },
  { name: 'camelcase', size: 4 },
  { name: 'date-time', size: 3 },
  { name: 'once', === [ { name: '@siddharthkp/empty', size: 1 },
  { name: 'date-time', size: 4 },
  { name: 'once', size: 5 },
  { name: 'time-zone',
  √ 0prod » get root dependencies (989ms)
  √ 0prod » attach nested dependencies (1.1s)
  √ 0prod » get all dependencies
  - 0prod » show results

Making sure dependencies are installed
npm install

npm WARN [email protected] No repository field.

up to date in 0.079s

  - 1dev » setup was complete
  × 1dev » get size for node_modules [ { name: '@siddharthkp/empty', size: 0 },
  { name: 'camelcase', size: 4 },
  { name: 'date-time', size: 3 },
  { name: 'once', === [ { name: '@siddharthkp/empty', size: 1 },
  { name: 'camelcase', size: 5 },
  { name: 'date-time', size: 4 },
  { name: 'once',
  √ 1dev » get root dependencies (1s)
  √ 1dev » attach nested dependencies (987ms)
  √ 1dev » get all dependencies
  - 1dev » show results

  2 tests failed [10:42:38]
  4 tests todo

  1. 0prod » get size for node_modules
  AssertionError: [ { name: '@siddharthkp/empty', size: 0 },
    { name: 'time-zone',
        Test.fn (fixtures/tests.js:41:11)
        _combinedTickCallback (internal/process/next_tick.js:141:11)
        process._tickCallback (internal/process/next_tick.js:180:9)

  2. 1dev » get size for node_modules
  AssertionError: [ { name: '@siddharthkp/empty', size: 0 },
    { name: 'once',
        Test.fn (fixtures/tests.js:41:11)
        _combinedTickCallback (internal/process/next_tick.js:141:11)
        process._tickCallback (internal/process/next_tick.js:180:9)

npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] ava: `ava test/dist/*.js -s --no-cache --verbose`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] ava script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     C:\Users\orias\AppData\Roaming\npm-cache\_logs\2017-12-23T08_42_38_361Z-debug.log
npm ERR! Test failed.  See above for more details.

you can find the complete debug log here

This is odd, but I wonder if it's because my project has no production dependencies (only dev). There are definitely modules in node_modules though. Any ideas?

Is there a grunt plugin? Could not find one at npm.

If there is none would you mind, me making one?

Have you thought about creating a website so that someone could link to the report to show the cost of a module?

For example, bundlephobia shows you the webpack size if you included the module. See Example.

It would be great to do something like cost-of-modules.com/[email protected] and see that the result is 24 MB.

from https://twitter.com/PmaContacto/status/804703698257608704

Hi, the Warning image at the end of ReadMe is Not Found.

I don't know, how important that is to you, but I think that the sizes that are reported are sometimes smaller than the actual sizes. For example

> mkdir test
> cd test/
> npm init -y 
Wrote to /home/nknappmeier/tmp/test/package.json:
[...]

> npm install --save thought
npm WARN prefer global [email protected] should be installed with -g
[email protected] /home/nknappmeier/tmp/test
└─┬ [email protected] 
  ├── [email protected] 
[...]

> cost-of-modules

Making sure dependencies are installed
npm install --production

npm WARN [email protected] No description
npm WARN [email protected] No repository field.

Calculating...


┌───────────┬──────────────┬───────┐
│ name      │ children     │ size  │
├───────────┼──────────────┼───────┤
│ thought   │ 118          │ 8.63M │
├───────────┼──────────────┼───────┤
│ 1 modules │ 113 children │ 8.43M │
└───────────┴──────────────┴───────┘

> du -hs .
13M	.

While cost-of-modules only reports 8.43M of this project (that has only a package.json and the thought-dependency), while du -hs reports 13M. Maybe it does not take into account, that even a very small file uses 4kb of space, depending on the filesystems block-size.

Part of a standard CI flow is to verify that a package size did not cross a certain threshold. Right now due to the way the results are outputted it's cumbersome to extract and assert the final package size value.

Suggested solution:
Adding a new command line option, e.g. assert-smaller if the resulting package size is higher than the given number then fail with exit code different than 0.

Reference: https://travis-ci.org/siddharthkp/cost-of-modules/jobs/265179931

> ava test/dist/*.js -s --no-cache

module.js:491
    throw err;
    ^

Error: Cannot find module '../../../lib/helpers'

For anyone that faces this issue, try updating to Node 16 and run the command again. You can just use it for this command, and then go back to your version!

There is an issue with this particular line:

// src/helper.js: 61:71
var getDependencyTree = function getDependencyTree() {
    var result = syncExec('npm ls --json ' + productionModifier);
    return JSON.parse(result.stdout).dependencies;
};

The problem is that syncExec uses a child_process.exec to execute the npm command, and depending on the size of the output, it might break due to the restrictions of the buffer size.

I played with this a bit and there's a way to make this package work with yarn, so if someone wants to work on it, I just want to leave what code I tried:

diff --git a/src/helpers.js b/src/helpers.js
index c157a54..97eabd8 100644
--- a/src/helpers.js
+++ b/src/helpers.js
@@ -59,8 +59,20 @@ let setup = (includeDev) => {
     Adds them with --include-dev
 */
 let getDependencyTree = () => {
-  let result = syncExec(`npm ls --json ${productionModifier}`);
-  return JSON.parse(result.stdout).dependencies;
+  let result = syncExec(`yarn list --json`);
+  const transform = (packages) => {
+    const out = {};
+    packages.forEach((x) => {
+      const names = x.name.split("@");
+      const name = names[0] || "@" + names[1];
+      out[name] = { name };
+      if ("children" in x) {
+        out[name].dependencies = transform(x.children);
+      }
+    });
+    return out;
+  };
+  return transform(JSON.parse(result.stdout).data.trees);
 }; 

p.s. this doesn't work for my project because it has the same issue as in #66
p.p.s. I'm trying out analyze-module-size for now

Hi, what is the current status of this project? Is it still maintained?

This is a featrue 🙋

How about being able to check the size of all global modules?