hu17889 / go_spider Goto Github PK

can't load package: package github.com/hu17889/go_spider: no buildable Go source files in D:\Server\gopath\src\github.com\hu17889\go_spider

what's wrong?

我现在想获取html里面的script内容要怎么获取，并且获取完后我想过滤

The items of the PageItems are map[stirng]string, but many times we want to save some struct. It is not very convenient.

taskname有什么用处

spide如下页面出现乱码

http://www.autohome.com.cn/grade/carhtml/B_photo.html

抽取出来的中文全是乱码，怎么解决？

go get github.com/PuerkitoBio/goquery出现需要 golang.org/x/net/html ，这个找不到了，怎么解决呀？

Content-Type:application/json 会乱
Content-Type:application/json; charset=utf-8 不会乱

https://github.com/Jack-Cherish/python-spider

https://github.com/GoJerry/JSCrack 特别是这个项目

你好，这个可以用来收集域名吗？去重的那一步应该在哪里做呢

It'd be great if you could use a local cache or enable a cache mode for development. When working on spiders, it's both considerate and smart to use local caches of documents to keep from pulling thousands of documents from the nets.

自定义的程序处理爬虫过程processor；
自定义的结构体中如有某个属性存在资源竞争，需要定义sync.Mutex，加锁吗?

我将并发数调到20 ，但是发现同一个请求被重复爬取，且大多数请求未被处理

如果获取在js执行之后的页面数据

hmm... Why U set connectByHttp(p *page.Page, req *request.Request) as unexported function in downloader package? And without this function ,how can I get the response after request?

golang.org/x/net/html/charset
这个包，目前这个地址没有了，怎么获取？

charset不支持的gb2312，能否允许自定义解析的编码，然后添加gb2312的支持

It'd be great if there was a way to store some metadata on Page via AddUrl, this would make the page processor easier to implement as opposed to doing some switching on urls or other http parameters. I think just a map of string to string would be fine.

spider.go第144行闭包go函数丢了参数名，导致req成为引用传参！

scheduler_queue.go

func (this *QueueScheduler) Push(requ *request.Request) {
//assert count() ==MAX
this.queue <- requ
}

错误日志如下

2017/10/25 17:04:53 /go_proj/src/github.com/hu17889/go_spider/core/downloader/downloader_http.go:238: [ERROR] Get https://xxxxxxx: x509: certificate signed by unknown authority

在connectByHttp 函数中对client添加跳过证书验证，即可爬取https页面，没有找相关功能的函数设置

比如这个url https://www.repian110.com/film6/48765/48765.js

应该是gbk编码的文件，里面有一个圆点，httpdownloader自动转码后变成乱码，而iconv转gbk到utf8是正常的。