Old version documentation should not appear on Seach engines about rook.github.io HOT 8 CLOSED

@travisn removed 👍

from rook.github.io.

We dealt with this in Crossplane as well - google was very slow/unreliable to update their indexing, even when older docs were explicitly deleted from the site entirely.

See some discussion here: crossplane/docs#107 (comment)

Now, as part of our release process, we use Google Search Console to request a removal for the docs version that is now stale (unsupported) according to our release policies.

e.g. On https://search.google.com/search-console/removals, we add a removal for Starts with: https://crossplane.io/docs/v1.7.

from rook.github.io.

@travisn how about closing this issue for now as we have a workaround?

from rook.github.io.

An idea that I have is to create a robots.txt during release docs build, then in some way to symlink that latest-release robots.txt to the root of the website that it is picked up accordingly.

Not sure if the symlinking part works, though we can generally test the robots.txt "theory" by manually adding a robots.txt now. @travisn Anything against adding a robots.txt directly to the gh-pages branch to see if it would technically even help with the issue?

from rook.github.io.

An idea that I have is to create a robots.txt during release docs build, then in some way to symlink that latest-release robots.txt to the root of the website that it is picked up accordingly.

Not sure if the symlinking part works, though we can generally test the robots.txt "theory" by manually adding a robots.txt now. @travisn Anything against adding a robots.txt directly to the gh-pages branch to see if it would technically even help with the issue?

Sounds great to add the robots.txt directly to the branch and test this out for now.

from rook.github.io.

@jbw976 @travisn I don't have access to the Google Search Console of rook.io, does one of you have access to it?

from rook.github.io.

Ok I found access to the Google search console for rook.io and submitted a request to remove index on 1.8 and 1.9 docs. The request is still in progress, so let's check back soon if the search is finding the newer docs.

from rook.github.io.

@rathpc Google search is also finding your test publish site for the rook docs. How about taking it down? For example, one of the top test search results was here: https://rathpc.github.io/rook.github.io/docs/rook/v1.4/k8s-pre-reqs.html

from rook.github.io.