A Python project that captures all the links, recursively, from any websites sitemap.xml file and checks whether the links are 200OK or are broken. It also sends an email notification upon completion of the checks.

A simple website URL and link checker. It crawls through a given sitemap for every accessible URL, checks that the response is 200 (OK). It then procedes to check every link in the html of that page.

This program does not create a sitemap.xml for you, so for that you can a website such as https://www.xml-sitemaps.com/.

Note: Make sure to have Python pre-installed on your system before moving forward.

There are few packages and libraries that you'll be required to install. These packages, for your benifit, have been added as lineitems within the requirements.txt file.

To install the required packages use pip: pip install -r requirements.txt

Copy the contents of the conf/conf.ini.example file into conf/conf.ini in the same directory.

Modify your config file with the details for your site:

For sites that are protected behind a username and password, you can authenticate by providing the username and password in the config.

WARNING These are stored in plain text, so the right priviledges should be granted to keep them as secure as possible.

To run the script, ensure you have python (min <2.7) installed and run: python3 linkChecker.py

You can pass the config file name as a Command line argument, this is useful for multiple sites with a config for each site. ie:

python3 linkchecker.py mysite.ini

python3 linkchecker.py mysecondsite.ini

Enter the Broker-Link-Crawler directory, and run: python3 main.py

From here you can enter or browse for the filename of the XML sitemap, and click enter.

If your site has http authentication, then you will be asked to enter the username and password for the site. These details are not stored.

The script will carry out the test on every url, and then output a report of all the broken links found.

Mailtrap(https://mailtrap.io/) is a fake SMTP server for development teams to test, view and share emails sent from the development and staging environments without spamming.

WARNING These are stored in plain text, so the right priviledges should be granted to keep them as secure as possible.

If all has been set correctly, you can login to the mailtrap.io website and can see the emails coming thru.