Bad requests with GTM about heritrix3 HOT 4 CLOSED

Hi @damien-git, you should probably drop the Internet Archive a note (mailto:[email protected]), as they may be able to tune the behaviour of their crawler.

In general, I personally do not recommend Heritrix users use the speculative JavaScript extractor at all. It seems to cause more trouble than it's worth.

I quite like the idea of tuning the crawl via robots.txt but we should probably look at deprecating or improving the ExtractJS or KnowledgableExtractorJS processors first.

If we can find our which extractor they are using that might help.

from heritrix3.

I've run a variant of ExtractorJS for years, that lets me filter out the links it discovers using a set of regular expressions. These are applied before the links are turned into full URLs, making it a bit easier to target common false positives in JS libraries than it would be if we are doing the filtering in the scope. You also don't risk catching any URLs extracted via other (more reliable) means.

Looking at the above, I should probably filter out any links extracted via ExtractorJS containing "gtm."

from heritrix3.

We (Akamai) are seeing a similar issue with sites that have our mPulse product enabled, which includes JavaScript in the page's HTML that looks like this:

var a=["ak.bpcip","ak.cport","..."];

This results in our customer's websites getting crawled by numerous crawlers on each page for those 20+ elements of the array, e.g.:

http://website/foo/bar/ak.bpcip
http://website/foo/bar/ak.cport
... etc

from heritrix3.

+1

I am using google tag manager and crawler is making many requests with "/gtm.js"

from heritrix3.