lassebunk / human_power Goto Github PK

Is it possible to add arbitrary rules? e.g. Crawl-delay:

User-agent: msnbot
Crawl-delay: 1

http://www.bing.com/blogs/site_blogs/b/webmaster/archive/2009/08/10/crawl-delay-and-the-bing-crawler-msnbot.aspx

Content-Type:text/html; charset=utf-8
Should be
Content-Type:text/plain; charset=utf-8

Is there any way to have different configurations for different Rails environments?

if Rails.env.production?
  disallow :none
else
  disallow :all
end

Error:

NoMethodError (undefined method `env' for HumanPower::Rails:Module):
  config/robots.rb:1:in `block in robots'
  lib/silverpop_middleware.rb:13:in `call'
  lib/health_check_middleware.rb:10:in `call'

Is there an easy way in the DSL to write write rules for all user agents instead. So instead of...

googlebot do
 ...
end

...or...

user_agent[:googlebot, :bingbot] do
... 
end

...is there an equivalent of the hard-coded phrase Use_agent: * that would allow any user agent at the top of robots.txt ?

I got an app that is responsible for both the website and an API
The ApplicationController is for website only, with many before_action
Some before_action are not suitable only for HumanPower::Rails::RobotsController.
Currently I have a workaround to make skip the before_action for action robots.
Is it possible to change the superclass for HumanPower::Rails::RobotsController?

Easier then reading commits (and find the last version)