Verify links in awesome projects
awesome_bot checks for valid URLs in a file, it can be used to verify pull requests updating a README.
$ gem install awesome_bot
Usage: awesome_bot [file or files]
awesome_bot [options]
-f, --files [files] Comma separated files to check
-a, --allow [errors] Status code errors to allow
--allow-dupe Duplicate URLs are allowed
--allow-ssl SSL errors are allowed
--allow-redirect Redirected URLs are allowed
--allow-timeout URLs that time out are allowed
--base-url [base url] Base URL to use for relative links
-d, --request-delay [seconds] Set request delay
-t, --set-timeout [seconds] Set connection timeout
--skip-save-results Skip saving results
-w, --white-list [urls] Comma separated URLs to white list
-
You can check multiple files (comma separated or
*pattern, look below for details). -
By default, duplicate URLs or any status code other than
200are flagged as failures.- Use option
--allow-dupeto allow duplicates. - Use option
--allow-redirectto allow redirects. - Use option
--allowto allow specific status code errors. - Use option
--white-list(-wfor short) to prevent links from being flagged:-w domain1.com/post/article,domain2.comwhite listsdomain1.com/post/articleand all links matchingdomain2.com.
- Use option
$ awesome_bot README.md
> Checking links in README.md
Links found: 56, 37 unique
01. https://github.com/sindresorhus/awesome
02. http://i.giphy.com/urvsFBDfR6N32.gif
03. https://travis-ci.org/dkhamsing/awesome_bot.svg
# ...
37. https://github.com/dkhamsing
Checking URLs: ✓✓✓→?✓→✓→→✓✓→✓✓✓→✓✓✓✓✓✓✓✓✓✓✓→✓✓✓✓✓→✓✓
Issues :-(
> Links
1. [L007] 301 https://travis-ci.org/dkhamsing/awesome_bot.svg → https://api.travis-ci.org/dkhamsing/awesome_bot.svg
2. [L008] 302 https://badge.fury.io/rb/awesome_bot → http://rubygems.org/gems/awesome_bot
# ...
> Dupes
1. [L03] https://github.com/sindresorhus/awesome
2. [L05] http://i.giphy.com/urvsFBDfR6N32.gif
# ...$ awesome_bot README.md --allow-dupe --allow-redirect -w rubydoc,giphy
# allow redirects, dupes and white list all links matching rubydoc and giphy
$ awesome_bot README.md,README-zh.md
# check links in 2 files
$ awesome_bot docs/*.md
# check all markdown files in docs/ directory
$ awesome_bot README.md --allow-timeout -t 5
# speed up validation by setting a timeout of 5 seconds per link request and allowing timeouts
$ awesome_bot README.md --allow 403,429
# allow status code errors 403 and 429
# --allow 301 would be similar to --allow-redirect
$ awesome_bot README.md --base-url https://github.com/IDR/idr-notebooks/blob/master/
# will check relative links using the base url provided(master) $ git branch
* master
(master) $ git checkout -b new-branch
Switched to a new branch 'new-branch'
(new-branch) $ touch new-readme.md && echo 'https://github.com/dkhamsing' >> new-readme.md
(new-branch) $ git add new-readme.md
(new-branch) $ git commit -m 'Testing'
[new-branch ef47336] Testing
1 file changed, 1 insertion(+)
create mode 100644 new-readme.md
(new-branch) $ git diff master.. --name-only | grep '.md' | xargs awesome_bot
> Checking links in new-readme.md
Links to check: 1
1. https://github.com/dkhamsing
Checking URLs: ✓
No issues :-)
Wrote results to ab-results-new-readme.md.jsonirb(main):001:0> require 'awesome_bot'
=> true
irb(main):002:0> content = File.read 'README.md'
=> "..."
irb(main):003:0> result = AwesomeBot.check content
=> #<AwesomeBot::Result:0x007fdde39f4408 @links=...>
# AwesomeBot Result with success, statuses_issues, dupes and more
irb(main):004:0> puts result.success ? 'No errors' : ':-('
:-(More information at rubydoc.
Does your GitHub README contain a lot of links? awesome_bot can help you validate them when a pull request is created (or a commit is pushed). It is used by:
- https://github.com/tiimgreen/github-cheat-sheet
- https://github.com/enaqx/awesome-react
- https://github.com/ziadoz/awesome-php
- https://github.com/vsouza/awesome-ios
- https://github.com/alebcay/awesome-shell
- https://github.com/matteocrippa/awesome-swift
and more.
Tips
- Use the keyword
[ci skip]in your commit title/message to skip verification. - Use Danger.
To use awesome_bot with Travis CI, connect your repo and create a .travis.yml file.
language: ruby
rvm: 2.2
before_script: gem install awesome_bot
script: awesome_bot README.mdTo turn off email notifications, add the lines below
notifications:
email: falseIf you prefer Circle CI, it'll work too. Connect your repo and create a circle.yml file.
machine:
ruby:
version: 2.2.0
test:
pre:
- gem install awesome_bot
override:
- awesome_bot README.mdCircle CI, Codeship, and Semaphore CI support running tests without adding a file to the repo (a public configuration file can however help others contribute).
# Codeship
Setup
rvm use 2.2.0 --install
gem install awesome_bot
Test
awesome_bot README.md
# Semaphore CI
Language: Ruby
Ruby version: 2.2
Databases for: don't generate
Setup:
gem install awesome_bot
awesome_bot README.md
To add the Travis CI build status badge above to your project, use the following code
[](https://travis-ci.org/<username>/<project>)
i.e.
[](https://travis-ci.org/dkhamsing/awesome_bot)
As it happens, the default code snippet provided contain a redirect so adding a badge could fail your status 😭.. one way to fix this is to white list travis-ci, i.e.
- awesome_bot README.md --white-list travis-ci
You can also add a badge for other CI tools, check out shields.io.
Integrate awesome_bot with Danger and have results reported back to the pull request).
Here's the step in your Dangerfile:
# Check links
require 'json'
results = File.read 'ab-results-README.md-markdown-table.json'
j = JSON.parse results
if j['error']==true
fail j['title']
markdown j['message']
endThis project is available under the MIT license. See the LICENSE file for more info.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent