woodyrew / metalsmith-robots Goto Github PK

Hi,

When using metalsmith-robots and the frontmatter options, it is creating a robots.txt file but using a .hbs extension instead of .html, eg:

Disallow: subpage/index.hbs

Is there a way of making this plugin use a .html file name?

Thanks.

Settings public: true or private: true in a file's frontmatter add that page to the allow or disallow option without slash "/" in the beginning. Which lead to incorrect interpretation of its value by different crawler.

In Goolge Robots.txt Specifications you can find rule that [path] must start with "/".
Also with Yandex Robots.txt analysis you can check that without "/" it marks such rule as error - "Rule doesn't start with / or *".

I see that we can include the filenames '404.html', however can we include a path like /assets/img for example?