Exclude does not seem to work for crawl about sitediff HOT 3 CLOSED

Can confirm that the exclude setting does not work properly.
Excluding URLs via RegEx should be straightforward but RegEx's tested valid with rubular.com don't seem to match with sitediff when crawling.

e.g. a URL with a path of "/de-en/products/" is still crawled even with an exclude value of \/de-en\/.*?$

from sitediff.

When you first run init, you can include the exclude command line parameter. Then, when you crawl, it will exclude the paths represented in the RegEx.

@mmunz You'll want to use a regex of .*\.pdf

@friggingee You'll want to use a regex of ^\/de\-en\/.*?$

from sitediff.

@kirk-brown-ew I tried that. Using your suggested regex as a yaml file setting:

settings:
  exclude: "^\/de\-en\/.*?$"

yields the same result, namely that sites starting with "/de-en/..." are still being visited/cached/processed.

The actual solution lies in the syntax of the exclude setting, namely that it needs to be written like it's being generated by sitediff init instead of what the documentation suggests.

Using an exclude value of !ruby/regexp /^\/de\-en\/.*?$/ yields the desired result of all links starting with "de-en" being excluded.
On this note, the regex I used, \/de-en\/.*?$ works just fine, as expected, as well, if used with the prefixed syntax.

from sitediff.