astur / check-npm-dependents Goto Github PK

I'm using this to sort through hundreds of thousands of NPM packages, and it will give a bad name error for a package in one run but resolve properly in another. Any ideas why this might be happening?

It got through 40k packages before starting to do this.

There are two closely related, but separate steps that this package currently performs all in one go:

1. Fetches a page on npm
   

   check-npm-dependents/index.js

   Lines 5 to 6 in 126eeca

   	const response = await scra(`https://www.npmjs.com/package/${name}`);
   	if(response.statusCode !== 200) return -1;

2. Parses the fetched page to return a number
   

   check-npm-dependents/index.js

   Lines 7 to 10 in 126eeca

   	const $ = load(response.body);
   	const depLink = $('a[href="?activeTab=dependents"]');
   	if(depLink.length !== 1) return -1;
   	return +depLink.text().replace(/[^0-9]/g, '');

The latter is what I am most interested in having an abstraction for. HTTP requests, on the other hand, are unfortunately a very leaky abstraction. I would prefer to have control over retries, caching semantics, etc. BYOC (Bring Your Own Client), if you will.

I imagine this being implemented as a .parse(html) method or similar.

When things go wrong, this package currently returns -1.

Since these represent unrecoverable errors, I would like an error to be thrown instead. Otherwise, I will have to write something like this around every invocation to avoid running the business logic when the data is unavailable:

if (dependents === -1) {
    throw new Error(`Unable to determine dependents for package: "${name}"`);
}
// ... business logic