gsh199449 / spider Goto Github PK

View Code? Open in Web Editor NEW

986.0 122.0 485.0 14.78 MB

A configurable web spider with a easy-to-use web console

License: GNU General Public License v3.0

Java 85.45% JavaScript 14.55%

spider gatherplatform web-console text-mining cralwer

spider's People

Contributors

Stargazers

Watchers

Forkers

jixiang0537 wongsir guoyu07 xyz1o2 fengwuze guoplus jerrycshen jibaro nilportugues tempbottle code4craft tiagobarreto lynchuan xj626852095 thaer899 qjyzwlz asnaper liinnux jerry4pan 771724316 gantoday-spider dyhack wangchenyang geekcheng playguys fysoft2006 okuc phantomes msnvip iwen-pengh igdp javajiao watemei chu888chu888 wesleychuforever 9527y lindominic davidmr001 shiyangbinxixi minsifansi huangxiaofeng10047 nylqd bruceq robin-niu keshecn fashtimedotcom dystudio zerolugithub leexyan wuxianyuzhou it-starver syngain neji1983 0xqq qingkongabc markryan47 chltkgg vincentchen kyoucr guoweihdu yzmin1106 blackzw ippxie woniu001 huangshaoze qiankunshe flywencww circlestriker useryu lqnasa qzw1210 shxinding seasonouc asd13606410359 halking billho yuanych royzeng xinghan winsomexjs dlhn hysgithub ouyanghuangzheng zhangqiaoqi588 konglingjuanyi 7758285 vimskk xiaokekehaha simonlu9 liuxiaoqiang kenchen1101 zzylekang jieke-tao ozhiganov liyan726 binrui frankzhuo superhj1987 sunshineit francescoalb

spider's Issues

建议idea工程相关的文件要ignore

现在工程里clone下来会有idea的工程配置文件，建议添加到gitignore里面

启动时 analyzer [index_ansj] not found for field [content]

analyzer [index_ansj] not found for field [content]

能否曾加一个批量发布采集功能？

很感谢您的这个产品，昨天我试了一下。有个建议希望在爬虫列表那里增加一个勾选框，这样方便批量发布采集任务，这样不用每次都要单独进入编辑模板才能发布。谢谢。

owenr 你好请问 webmagic 如何和 spring 整合在一起使用？

抓取任务列表中的【已抓取数量】和网站列表中的【资讯数】不一致

task在触发reachMax或者exceedRatio停止之后，CommonSpider onSuccess方法中log打印的【有效页面数】和抓取任务列表中的【已抓取数量】以及网站列表中的【资讯数】不一致均不一致

log

爬虫ID5e21c6bd-4878-413c-b0fa-a46a5c3376ac已处理31个页面,有效页面6个,最大抓取页数10,reachMax=false,exceedRatio=true,退出.

已抓取数量

任务名称已抓取数量抓取状态
www.163.com 9 STOP

资讯数

网站域名资讯数
www.163.com 7

defaultCategory字段无法使用

"categoryReg": "",
"categoryXPath": "",
"defaultCategory": "体育",
//之前以为前2项不指定，就会默认分类为"体育"。试了下不是这样。如上指定规则，最后得到的category是空

owner 我现在想做列表的爬虫抓取和列表中链接信息页数据抓取

http://cs.58.com/zhaozu/pn1/pve_1092_1/?PGTID=0d30000d-0019-e4b5-3eff-f37b576ebd78&ClickID=1
你好这个是58求租页的抓取
http://cs.58.com/zhaozu/27966561338700x.shtml?psid=175333501193992205461587821&entinfo=27966561338700_0&iuType=p_1&PGTID=0d30000d-0019-e7ad-0e33-37b8a66da82f&ClickID=1
这个是其中一个信息页的抓取
求帮忙实现哦

org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'commonSpider' defined in URL [file:/C:/Java/IdeaProjects/GatherPlatform/spider-master/target/spider/WEB-INF/classes/mvc-dispatcher-servlet.xml]: Cannot resolve reference to bean 'commonWebpageDAO' while setting bean property 'commonWebpageDAO'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'commonWebpageDAO' defined in file [C:\Java\IdeaProjects\GatherPlatform\spider-master\target\spider\WEB-INF\classes\com\gs\spider\dao\CommonWebpageDAO.class]: Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.gs.spider.dao.CommonWebpageDAO]: Constructor threw exception; nested exception is NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{y3YLXLvBRXGPiHARqVc2qQ}{127.0.0.1}{127.0.0.1:9300}]]

java.lang.IllegalArgumentException: Non-positive period.
	java.util.Timer.schedule(Unknown Source)
	com.gs.spider.gather.commons.CommonSpider.<init>(CommonSpider.java:364)
	sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
	sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
	java.lang.reflect.Constructor.newInstance(Unknown Source)

存储模板时NPE

在编辑模板页面编辑好后，点保存总是会出现/panel/commons/editSpiderInfo，每次都必现。

java.lang.NullPointerException
	at com.gs.spider.dao.SpiderInfoDAO.getByDomain(SpiderInfoDAO.java:126)
	at com.gs.spider.dao.SpiderInfoDAO.index(SpiderInfoDAO.java:51)
	at com.gs.spider.service.commons.spiderinfo.SpiderInfoService.lambda$index$2(SpiderInfoService.java:58)
	at com.gs.spider.model.utils.ResultBundleBuilder.bundle(ResultBundleBuilder.java:25)
	at com.gs.spider.service.commons.spiderinfo.SpiderInfoService.index(SpiderInfoService.java:58)
	at com.gs.spider.controller.commons.spiderinfo.SpiderInfoController.save(SpiderInfoController.java:87)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

大数据量导出存在内存溢出隐患

如果数据量过大，很有可能造成程序崩溃。

WebMagic已经升级到0.6.0

webmagic升级咯，咱们这个是不是也跟进一下呢？