A source code of an Apify actor that finds and reports broken links on a website. Unlike other SEO analysis tools, it also reports broken URL #fragments.
Home Page: https://apify.com/jancurn/find-broken-links
License: Apache License 2.0
I downloaded this repo, ran npm i and then apify run and I get:
The new version of Apify CLI uses a new format of the "apify.json" file. Your file will be automatically updated
to the new format. Yes
I chose 'Yes' and then I get a fatal error:
Error: Cannot find module 'underscore'
Require stack:
- /Users/torletsky/Sites/domain7/actor-find-broken-links-master/main.js
at Function.Module._resolveFilename (internal/modules/cjs/loader.js:815:15)
at Function.Module._load (internal/modules/cjs/loader.js:667:27)
at Module.require (internal/modules/cjs/loader.js:887:19)
at require (internal/modules/cjs/helpers.js:74:18)
at Object.<anonymous> (/Users/torletsky/Sites/domain7/actor-find-broken-links-master/main.js:1:11)
at Module._compile (internal/modules/cjs/loader.js:999:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:1027:10)
at Module.load (internal/modules/cjs/loader.js:863:32)
at Function.Module._load (internal/modules/cjs/loader.js:708:14)
at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:60:12) {
code: 'MODULE_NOT_FOUND',
requireStack: [
'/Users/torletsky/Sites/domain7/actor-find-broken-links-master/main.js'
]
}
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] start: `node main.js`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] start script.
How do I proceed, please?
better explain how to know the URL is broken, is this the broken link?
{
"url": "https://tvo.org/current-affairs",
"normalizedUrl": "https://tvo.org/current-affairs",
"httpStatus": null,
"errorMessage": null,
"fragment": "",
"fragmentValid": false,
"crawled": false
},
the example dataset is an old example with only a KV store
now there is anything in the KV store as described in the readme, there is only output dataset:
https://my.apify.com/tasks/hVQYNQVysl3nX4bP2#/runs/oH3FaE4wkumgG6reG
do you need proxies to run the actor? what is the consumption?
Here is the kind of errors, I can see:
ERROR: BasicCrawler: handleRequestFunction failed, reclaiming failed request back to the list or queue {"url":"https://www.humancoders.com/formations/securite-des-applications-web.pdf","retryCount":1}
2020-07-03T12:04:43.666Z Error: net::ERR_ABORTED at https://www.humancoders.com/formations/securite-des-applications-web.pdf
2020-07-03T12:04:43.666Z at navigate (/home/node/node_modules/puppeteer/lib/FrameManager.js:103:37)
2020-07-03T12:04:43.667Z at <anonymous>
2020-07-03T12:04:43.667Z at process._tickCallback (internal/process/next_tick.js:189:7)
2020-07-03T12:04:43.668Z -- ASYNC --
2020-07-03T12:04:43.668Z at Frame.<anonymous> (/home/node/node_modules/puppeteer/lib/helper.js:144:27)
2020-07-03T12:04:43.669Z at Page.goto (/home/node/node_modules/puppeteer/lib/Page.js:587:49)
2020-07-03T12:04:43.669Z at Page.<anonymous> (/home/node/node_modules/puppeteer/lib/helper.js:145:23)
2020-07-03T12:04:43.669Z at PuppeteerCrawler.gotoFunction (/home/node/node_modules/apify/build/puppeteer_crawler.js:30:53)
2020-07-03T12:04:43.670Z at PuppeteerCrawler._handleRequestFunction (/home/node/node_modules/apify/build/puppeteer_crawler.js:308:48)
2020-07-03T12:04:43.670Z at <anonymous>
2020-07-03T12:04:43.671Z at process._tickCallback (internal/process/next_tick.js:189:7)
For example, when crawling blog.apify.com, some pages are visited multiple-times since they have different URL params:
https://blog.apify.com/contact-information-scraper-7104cb0df25e?source=post_recirc---------1------------------
https://blog.apify.com/contact-information-scraper-7104cb0df25e?source=collection_home---4------8-----------------------
We could add an input option to list URL parameters that should be ignored when deciding if URL is unique.
Sometimes you also want to find links that still work, but have changed over time, so that you can fix those.
Or maybe generate two reports?
So that if you enter apify.com, it will also crawl blog.apify.com etc.
Subject says it all
So that people can easily schedule the actor to send them a report every day, or week.
When inspecting a website, if you insert its URL with www. in the beginning, like https://www.apify.com/, but the internal links don't have the www. prefix, like https://apify.com/about, they do not get enqueued for recursive inspection, even though they are technically nested pages of the main page. We should consider adding some logic that deals with this and maybe ignores the www. prefix at the start of the URL when considering whether to recursively check a link, or not.
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.