While parsing habr got few links from script field.

• Cutout script, code, textaea, style before runing parser.

Short Description

Add way to check exernal url if local fails.

Scenario

1. CI checks local website after generating static artefacts on domain localhost:8080
2. dev) localhost:8080/old/path fails
3. prod) domain.com/docs, /old/path redirected to domain.com/docs/new/feature
4. so in case of local fail, we suppose to ask prod to check same url (somehow)

Research possibility to use parsers instead regular expression for links extraction.

This needs to be implemented, maybe in some other form.

important to have

• python 3.5 support
• fast serving for local files

if you type

deadlinks --help

Usage Example doesn't have new lines.

Scenario

• One of the links on the page has trailing slash, but deadlinks, removes it (trailing slash).
• Webserver return 404 if trailing slash not available.

Tests case

deadlinks https://devdocs.magento.com/ -n 10 -s all -p "blob/master" -p "issues/new"  -d "jira"  | tee devdocs.txt

RuntimeError: Set changed size during iteration

Using sets wasn't so great idea.

https://github.com/butuzov/deadlinks/runs/2073524533?check_suite_focus=true

Create Formula for Brew

Checkout ansible and youtube-dl as examples of python packages distributed via homebrew

Test for different possible URL decoding bugs.

Case

1. get page at https://domain.com/path
2. redirection in action https://domain.com/path -> https://domain.com/path/
3. get links at /path (link1 and link2)
4. create new links (https://domain.com/link1 and https://domain.com/link2), while real url https://domain.com/path/link1 and https://domain.com/path/link2

> deadlinks localhost:1313 -s all
Usage:
  deadlinks <URL> [OPTIONS]
Try "deadlinks --help" for help.

Error: URL localhost:1313 is not valid

Its better for users to, catch this error and prepand "http" silently.

Few Blockers before moving to next version supported.

1. reppy - seomoz/reppy#122
2. new flow (golden master)
3. CI update (switch to github actions or other)
4. time to do work on this

I need to limit the app by default to stay within the documentation path, and maybe allow to index the whole website with some knob.

Scenario

1. Ask to index localhost/docs/projectname/ (which simply serves static files within $root/docs/projectname folder).
2. All links found that points to the localhost domain are valid, but only if they also starts with /docs/projectname/
3. If not, links declare to be external.

TODO

• Implement
• Introduce cli args
• Tests
• Documentation

This program is great! But it runs very slow when it comes to too many deadlinks.
All the threads are just stuck for a very long time. It'll take hours to finish the progress.
So, I wonder if we can set a timeout argument, let's say, 10s, would greatly improve the experience!

• docs
• release pipeline
• changelog
• pipy package
• brew tap
• docker image

App needs to have default user_agent, and way to redefine it.

Pros:

• robots.txt comlience - #8

Cons:

• None

need it.

Issue requested

Maybe it's will be useful to create a ci option to run commands before running crawler.

So, for example, deadlinks will be able to run some webserver to server HTML from directory.

Program should still return report if SIGINT received.

maybe it worth time to create github action for deadlinks?

"mailto" links are not ignored.

TODO

• Implement
• Tests

• release pipeline doens't reacts on new tag that comes from github
• automade docs publishig from docs to @docs
• automate bew formula bump

Google is not accessible in some places
So Maybe you can add proxy function.
Thanks

Need more tests for prerelease (end to end) against dev brew and dev package.

In Addition to the #30 it would be great to server netlify redirections.

Files now doesn't support trailing slash redirections.

https://hynek.me/articles/simple-python-azure-pipelines/

Goal

• Obtain deadlinks package name.
• Build travis-ci deploy pipeline for development branch
• Build travis-ci deploy pipeline for master branch

Pros:

• It's sitemap full of URLs.

Cons:

• will require a better mechanism of starting URL for the app.

The bot created this issue to inform you that pyup.io has been set up on this repo.
Once you have closed it, the bot will open pull requests for updates as soon as they are available.

implement

link with "\n" is vlid from browser standpoint.

We need to respect robots.txt

1. It's not supposed to exists for local dev.
2. It can prevent of DoS'ing website.

It's a complex issue, of which domain we should respect (ideally - all).

• (at least) documentation domain.
• (at most) all others.

Pros:

• It's right thing to do.

Cons:

• Slower indexation/crawling.

Need to show somehow current progress of indexation.

This is draft of idea to have localization checked. Idea come true after founds at kubernetes.io

deadlinks kubernetes.io/ko -n10 -s ignored | \ 
        grep kubernetes.io/docs | \
        awk '{print $4}' | \
        sed 's/.io\/docs/.io\/ko\/docs/g' | \
        xargs -I {} sh -c "deadlinks {} -single -short"

Somehow internal files are slow to index.

We need ability to run agains multiple pathes, and then compare results.

• http://localhost:8080/
• http://localhost:8080/en/stable

Maybe its can be done in a separate project, not deadlinks itself, but it's very useful while checking documentation.

deadlinks deserves better tagline

Current variations:

• (github/header) CLI app checker for dead links in the generated html documentation https://pypi.org/project/deadlinks
• (github/readme) deadlinks is a simple cli tool to check your documentation/website for deadlinks.
• (pypi) CLI/API for links liveness checking.
• (brew) CLI/API for links liveness checking

Check for htttp://example.org/ -> http://example.org (ignored)

In the case of domains, it shouldn't happen.