set AllowedDomains as the Limit way may be better about colly HOT 5 CLOSED

@asciimoo use release version to keep it in old version, master branch only for development and release version for stable in different stage, like go-redis/redis does, but this will take you more time

from colly.

@cctse I agree, but it would require API change to be effective, because glob expressions have to be compiled and we have to track manually when the AllowedDomains list changes. Perhaps we could add a SetAllowedDomainsGlobs(domainGlobs string...) error function to the Collector for this purpose.

from colly.

@cctse as an alternative solution you can use Collector.URLFilters:

c.URLFilters = []*regexp.Regexp{
	regexp.MustCompile(`^https?://[a-z]+\.httpbin\.org/`)
}

Full example: http://go-colly.org/docs/examples/url_filter/

I think this functionality covers your needs and maybe adding domain globs too would be redundant. What do you think?

from colly.

@asciimoo thanks, this can works well, I'm using glob check before visit a link in onResponse. I'm still thinking add a function like SetAllowedDomainsGlobs may be better

from colly.

It would require less boilerplate than URLFilters but the feature it provides is only a subset of URLFilters's functionality. I think it is more important to keep the API clean and small than optimizing for edge cases by adding new functions without new functionality. Feel free to reopen if you still disagree.

from colly.