User-Agent about goquery HOT 4 CLOSED

Ah ok, sorry for the confusion, I didn't see where the user-agent could come into play in goquery. Well this is really just a helper function, you can't customize anything on the request as it is.

What you can do (though not tested, but along those lines) is this:

req, err := http.NewRequest("GET", url, nil)
if err != nil {
// handle error
}
req.Header.Set("User-Agent", ua)
res, err := http.DefaultClient.Do(req)
if err != nil {
// handle error
}
defer res.Body.Close()
root, err := html.Parse(res.Body)
if err != nil {
  return
}
d := NewDocumentFromNode(root)

from goquery.

@raichu
hello,
Maybe this code can help you.

req, _ := http.NewRequest("GET", link, nil)
req.Header.Add("Content-Type", "application/x-www-form-urlencoded")
req.Header.Add("user-agent", "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36")
if resp, err := http.DefaultClient.Do(req); err == nil {
if doc, err := goquery.NewDocumentFromResponse(resp); err == nil {
}
}`

from goquery.

I think you're on the wrong project page, you must be referring to gocrawl https://github.com/PuerkitoBio/gocrawl ?

If so, yes you can configure the user-agent (both the user-agent of the crawler when requesting robots.txt and the user-agent used to request pages). It's on the Crawler.Options field. (https://github.com/PuerkitoBio/gocrawl#options)

from goquery.

I'm referring to this line specifically

from goquery.