- 我是幸明,一名爬虫工程师与交易工程师;
- 忙碌而无为是件多么无趣的事情;
- 壮年不过三十载,希望能留下一些东西;
liuxingming / sinaspider Goto Github PK
View Code? Open in Web Editor NEW新浪微博爬虫(Scrapy、Redis)
新浪微博爬虫(Scrapy、Redis)
您好,我最近遇到scrapy分布式处理的问题,对这个分布式处理总是处理不正确,您可以详细介绍下scrapy分布式中的注意和说明吗 十分感谢
现在WAP登录也需要输入验证码了,楼主能否写个获取验证码的模块,或者有其他什么方法?
开启了debug模式,但是每次都抓不到,get后的链接如: http://weibo.cn/5876375670/info
都指向了一个有验证码的登录页面
话说不是一直在往里add和pop么。为啥会运行结束了,大概跑了不到一小时。
报错如下
2017-04-29 11:04:01 [scrapy.core.engine] ERROR: Error while obtaining start requests
Traceback (most recent call last):
File "/Library/Python/2.7/site-packages/scrapy/core/engine.py", line 127, in _next_request
request = next(slot.start_requests)
File "/Users/voidwalker/Downloads/爬虫和web程序/SinaSpider/Sina_spider1/Sina_spider1/spiders/spiders.py", line 64, in start_requests
ID = self.scrawl_ID.pop()
KeyError: 'pop from an empty set'
出现这个报错是新浪封了IP吗
单机版里面parse3回掉没找到FollowsItem["follows"] FansItem["fans"]的赋值,但是程序运行的结果里面确实是有值,不太理解这块
我用的国外的VPS 在虚拟机上测试是可以用的 但是在VPS上的时候就会单次获取不成功就直接Unhandled error in Deferred: 应该怎么解决
URLError:
2016-08-26 16:22:18 [boto] ERROR: Unable to read instance data, giving up
2016-08-26 16:22:18 [requests.packages.urllib3.connectionpool] INFO: Starting new HTTPS connection (1): login.sina.com.cn
2016-08-26 16:22:18 [requests.packages.urllib3.connectionpool] DEBUG: "POST /sso/login.php?client=ssologin.js(v1.4.15) HTTP/1.1" 200 None
Unhandled error in Deferred:
2016-08-26 16:22:18 [twisted] CRITICAL: Unhandled error in Deferred:
红圈的地方没有"location"
这个关键词。
200的response如下:
{'Proc_Node': ['web359.mweibo.bx.sinanode.com'], 'Set-Cookie': ['_T_WM=339ff7f97fb5c147c8e465d4e9eb1aaa; expires=Fri, 07-Apr-2017 03:27:31 GMT; path=/; domain=.weibo.cn; httponly', 'WEIBOCN_FROM=deleted; expires=Thu, 01-Jan-1970 00:00:01 GMT; path=/; domain=.weibo.cn'], 'Lb_Node': ['layer7-002.mweibo.hk.sinanode.com'], 'Vary': ['Host,Accept-Encoding'], 'X-Log-Uid': ['6039348406'], 'Server': ['Tengine'], 'Date': ['Wed, 08 Mar 2017 03:27:31 GMT'], 'Content-Type': ['text/html; charset=utf-8'], 'Age': ['3']}
另外
raise IgnoreRequest
会让scrapy直接忽略这个语法错误,不利于debug。
请教大神们,代码出现以下报错是什么原因,redis启动了,weibo账号也能读取,但获取cookie之前检查redis中是否存在账号就会报错。。。。
/anaconda/bin/python SinaSpider-master/Sina_spider3/launch.py
2017-02-09 18:55:25 [scrapy] INFO: Scrapy 1.0.5 started (bot: ['Sina_spider3'])
2017-02-09 18:55:25 [scrapy] INFO: Optional features available: ssl, http11, boto
2017-02-09 18:55:25 [scrapy] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'Sina_spider3.spiders', 'CONCURRENT_REQUESTS': 1, 'SPIDER_MODULES': ['Sina_spider3.spiders'], 'BOT_NAME': ['Sina_spider3'], 'SCHEDULER': 'Sina_spider3.scrapy_redis.scheduler.Scheduler', 'REDIRECT_ENABLED': False, 'DOWNLOAD_DELAY': 10}
2017-02-09 18:55:25 [scrapy] INFO: Enabled extensions: CloseSpider, TelnetConsole, LogStats, CoreStats, SpiderState
2017-02-09 18:55:25 [Sina_spider3] DEBUG: Reading URLs from redis list 'Sina_spider3:start_urls'
Unhandled error in Deferred:
2017-02-09 18:55:26 [twisted] CRITICAL: Unhandled error in Deferred:
2017-02-09 18:55:26 [twisted] CRITICAL:
Traceback (most recent call last):
File "/anaconda/lib/python2.7/site-packages/twisted/internet/defer.py", line 1299, in _inlineCallbacks
result = g.send(result)
File "/anaconda/lib/python2.7/site-packages/scrapy/crawler.py", line 71, in crawl
self.engine = self._create_engine()
File "/anaconda/lib/python2.7/site-packages/scrapy/crawler.py", line 83, in _create_engine
return ExecutionEngine(self, lambda _: self.stop())
File "/anaconda/lib/python2.7/site-packages/scrapy/core/engine.py", line 68, in init
self.downloader = downloader_cls(crawler)
File "/anaconda/lib/python2.7/site-packages/scrapy/core/downloader/init.py", line 69, in init
self.middleware = DownloaderMiddlewareManager.from_crawler(crawler)
File "/anaconda/lib/python2.7/site-packages/scrapy/middleware.py", line 56, in from_crawler
return cls.from_settings(crawler.settings, crawler)
File "/anaconda/lib/python2.7/site-packages/scrapy/middleware.py", line 34, in from_settings
mw = mwcls.from_crawler(crawler)
File "/Users/yang/Documents/Code/SinaSpider-master/Sina_spider3/Sina_spider3/middleware.py", line 40, in from_crawler
return cls(crawler.settings, crawler)
File "/Users/yang/Documents/Code/SinaSpider-master/Sina_spider3/Sina_spider3/middleware.py", line 36, in init
initCookie(self.rconn, crawler.spider.name)
File "/Users/yang/Documents/Code/SinaSpider-master/Sina_spider3/Sina_spider3/cookies.py", line 66, in initCookie
if rconn.get("%s:Cookies:%s--%s" % (spiderName, weibo[0], weibo[1])) is None: # 'SinaSpider:Cookies:账号--密码',为None即不存在。
KeyError: 0
Process finished with exit code 0`
self.server.setbit(self.key + str(uid / 4000000000), uid % 4000000000, 1)
key = "dupefilter:%s" % int(time.time())
这里key不唯一,去重应该是失败的。
请楼主确认。
在爬取了部分数据后,我发现有些微博的点赞、转发、评论数是错误的,会达到几亿的级别。
借楼主地用一下,我建一个QQ群537549079,大家可以加入讨论一下这个爬微博数据的问题,我现在用分布式的版本,但是有很多问题,希望大家能够一起交流解决问题。
各位大神,为什么我运行出现下面的错误啊?
Traceback (most recent call last):
File "C:\Python27\lib\site-packages\scrapy\core\engine.py", line 126, in _next_request
request = next(slot.start_requests)
File "C:\Users\chinchilla77\SinaSpider\Sina_spider1\Sina_spider1\spiders\spiders.py", line 22, in start_requests
ID = self.scrawl_ID.pop()
KeyError: 'pop from an empty set'
新手,求帮助:(
Sina_spider1.items这个是怎么来的?
from scrapy.spider import CrawlSpider
from scrapy.selector import Selector
from scrapy.http import Request
from Sina_spider1.items import InformationItem, TweetsItem, FollowsItem, FansItem
Traceback (most recent call last):
File "<pyshell#37>", line 1, in
from Sina_spider1.items import InformationItem, TweetsItem, FollowsItem, FansItem
ImportError: No module named Sina_spider1.items
运行出错是什么情况
/System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7 /Applications/PyCharm.app/Contents/helpers/pydev/pydevd.py --multiproc --qt-support --client 127.0.0.1 --port 59058 --file /data/apps/codes/weibo/Begin.py
warning: Debugger speedups using cython not found. Run '"/System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python" "/Applications/PyCharm.app/Contents/helpers/pydev/setup_cython.py" build_ext --inplace' to build.
pydev debugger: process 8313 is connecting
Connected to pydev debugger (build 163.10154.50)
/data/apps/codes/weibo/Sina_spider1/spiders/spiders.py:4: ScrapyDeprecationWarning: Module `scrapy.spider` is deprecated, use `scrapy.spiders` instead
from scrapy.spider import CrawlSpider
2017-02-13 19:40:30 [scrapy] INFO: Scrapy 1.2.2 started (bot: Sina_spider1)
2017-02-13 19:40:30 [scrapy] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'Sina_spider1.spiders', 'SPIDER_MODULES': ['Sina_spider1.spiders'], 'LOG_LEVEL': 'INFO', 'DOWNLOAD_DELAY': 2, 'BOT_NAME': 'Sina_spider1'}
2017-02-13 19:40:31 [scrapy] INFO: Enabled extensions:
['scrapy.extensions.logstats.LogStats',
'scrapy.extensions.telnet.TelnetConsole',
'scrapy.extensions.corestats.CoreStats']
Get Cookie Success!( Account:17012010793 )
Get Cookies Finish!( Num:1)
2017-02-13 19:40:31 [scrapy] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
'Sina_spider1.middleware.UserAgentMiddleware',
'Sina_spider1.middleware.CookiesMiddleware',
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
'scrapy.downloadermiddlewares.retry.RetryMiddleware',
'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
'scrapy.downloadermiddlewares.chunked.ChunkedTransferMiddleware',
'scrapy.downloadermiddlewares.stats.DownloaderStats']
2017-02-13 19:40:31 [scrapy] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
'scrapy.spidermiddlewares.referer.RefererMiddleware',
'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
'scrapy.spidermiddlewares.depth.DepthMiddleware']
2017-02-13 19:40:31 [scrapy] INFO: Enabled item pipelines:
['Sina_spider1.pipelines.MongoDBPipleline']
2017-02-13 19:40:31 [scrapy] INFO: Spider opened
2017-02-13 19:40:31 [scrapy] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2017-02-13 19:40:32 [scrapy] ERROR: Error downloading <GET http://weibo.cn/5235640836/follow>
Traceback (most recent call last):
File "/Library/Python/2.7/site-packages/twisted/internet/defer.py", line 1299, in _inlineCallbacks
result = result.throwExceptionIntoGenerator(g)
File "/Library/Python/2.7/site-packages/twisted/python/failure.py", line 393, in throwExceptionIntoGenerator
return g.throw(self.type, self.value, self.tb)
File "/Library/Python/2.7/site-packages/scrapy/core/downloader/middleware.py", line 43, in process_request
defer.returnValue((yield download_func(request=request,spider=spider)))
File "/Library/Python/2.7/site-packages/scrapy/utils/defer.py", line 45, in mustbe_deferred
result = f(*args, **kw)
File "/Library/Python/2.7/site-packages/scrapy/core/downloader/handlers/__init__.py", line 65, in download_request
return handler.download_request(request, spider)
File "/Library/Python/2.7/site-packages/scrapy/core/downloader/handlers/http11.py", line 60, in download_request
return agent.download_request(request)
File "/Library/Python/2.7/site-packages/scrapy/core/downloader/handlers/http11.py", line 285, in download_request
method, to_bytes(url, encoding='ascii'), headers, bodyproducer)
File "/Library/Python/2.7/site-packages/twisted/web/client.py", line 1631, in request
parsedURI.originForm)
File "/Library/Python/2.7/site-packages/twisted/web/client.py", line 1408, in _requestWithEndpoint
d = self._pool.getConnection(key, endpoint)
File "/Library/Python/2.7/site-packages/twisted/web/client.py", line 1294, in getConnection
return self._newConnection(key, endpoint)
File "/Library/Python/2.7/site-packages/twisted/web/client.py", line 1306, in _newConnection
return endpoint.connect(factory)
File "/Library/Python/2.7/site-packages/twisted/internet/endpoints.py", line 788, in connect
EndpointReceiver, self._hostText, portNumber=self._port
File "/Library/Python/2.7/site-packages/twisted/internet/_resolver.py", line 174, in resolveHostName
onAddress = self._simpleResolver.getHostByName(hostName)
File "/Library/Python/2.7/site-packages/scrapy/resolver.py", line 21, in getHostByName
d = super(CachingThreadedResolver, self).getHostByName(name, timeout)
File "/Library/Python/2.7/site-packages/twisted/internet/base.py", line 276, in getHostByName
timeoutDelay = sum(timeout)
TypeError: 'float' object is not iterable
2017-02-13 19:40:32 [scrapy] INFO: Closing spider (finished)
2017-02-13 19:40:32 [scrapy] INFO: Dumping Scrapy stats:
{'downloader/exception_count': 1,
'downloader/exception_type_count/exceptions.TypeError': 1,
'downloader/request_bytes': 1003,
'downloader/request_count': 1,
'downloader/request_method_count/GET': 1,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2017, 2, 13, 11, 40, 32, 108493),
'log_count/ERROR': 1,
'log_count/INFO': 7,
'scheduler/dequeued': 1,
'scheduler/dequeued/memory': 1,
'scheduler/enqueued': 1,
'scheduler/enqueued/memory': 1,
'start_time': datetime.datetime(2017, 2, 13, 11, 40, 31, 780254)}
2017-02-13 19:40:32 [scrapy] INFO: Spider closed (finished)
Process finished with exit code 0
以上是运行的时候的错误堆栈,大神快出现。
Traceback (most recent call last):
File "Begin.py", line 3, in
cmdline.execute("scrapy crawl sinaSpider".split())
File "/usr/local/lib/python2.7/dist-packages/scrapy/cmdline.py", line 108, in execute
settings = get_project_settings()
File "/usr/local/lib/python2.7/dist-packages/scrapy/utils/project.py", line 60, in get_project_settings
settings.setmodule(settings_module_path, priority='project')
File "/usr/local/lib/python2.7/dist-packages/scrapy/settings/init.py", line 285, in setmodule
self.set(key, getattr(module, key), priority)
File "/usr/local/lib/python2.7/dist-packages/scrapy/settings/init.py", line 260, in set
self.attributes[name].set(value, priority)
File "/usr/local/lib/python2.7/dist-packages/scrapy/settings/init.py", line 55, in set
value = BaseSettings(value, priority=priority)
File "/usr/local/lib/python2.7/dist-packages/scrapy/settings/init.py", line 91, in init
self.update(values, priority)
File "/usr/local/lib/python2.7/dist-packages/scrapy/settings/init.py", line 317, in update
for name, value in six.iteritems(values):
File "/usr/local/lib/python2.7/dist-packages/six.py", line 599, in iteritems
return d.iteritems(**kw)
AttributeError: 'list' object has no attribute 'iteritems'
你好,之前爬虫都没有问题,后来有一次DOWNLOAD_DELAY设为0了,爬了半个小时就出现302错误,错误如下:
Redirecting (302) to GET http://m.weibo.cn/security from GET http://weibo.cn/2139359753/fans
Redirecting (302) to GET http://m.weibo.cn/security from GET http://weibo.cn/attgroup/opening?uid=2139359753
一开始以为是ip被封了,但是好几天都没好,而且直接用浏览器访问
http://weibo.cn/2139359753/fans
又都能打开,请问是什么问题呢?
即使用4g 抓也只有3、4k 每秒的速度。
大约40个账号,设置如下:
DOWNLOAD_DELAY = 2 # 间隔时间
CONCURRENT_REQUESTS_PER_DOMAIN = 100
CONCURRENT_REQUESTS = 70
求解答!
其实可以不用进行验证码操作,受作者启发,可以先登录weibo.com
的无验证码入口(微博账号安全里设为常登陆地点可以免验证码),然后直接在phontomjs
模拟打开weibo.cn
,weibo.cn
会是登录状态,这时候获取cookies便可。
由于我自己实现了,代码如下,仅供参考:
def init_phantomjs_driver():
headers = {
'Cookie': 'YF-Ugrow-G0=b02489d329584fca03ad6347fc915997; SUB=_2AkMvgPj2dcPxrAFYnPgWyGvkZYpH-jycVZEAAn7uJhMyOhgv7nBSqSVOKynW2PbhU4768kfRGZgNPwXeRA..; SUBP=0033WrSXqPxfM72wWs9jqgMF55529P9D9WWEFXHsNpvgJdQjr1GM.e765JpVF020SKM7e0571hMc', # 未登录时weibo.com的cookie
}
for key, value in headers.items():
webdriver.DesiredCapabilities.PHANTOMJS['phantomjs.page.customHeaders.{}'.format(key)] = value
useragent = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.110 Safari/537.36'
webdriver.DesiredCapabilities.PHANTOMJS['phantomjs.page.settings.userAgent'] = useragent
# local path refer phantomjs
driver = webdriver.PhantomJS(executable_path='xxxxxxxphantomjs路径xxxxxxx')
driver.set_window_size(1366, 768)
return driver
browser = weibo_auto_handle.init_phantomjs_driver()
browser.get("http://weibo.com")
time.sleep(3)
failure = 0
while "微博-随时随地发现新鲜事" == browser.title and failure < 5:
failure += 1
username = browser.find_element_by_name("username")
pwd = browser.find_element_by_name("password")
login_submit = browser.find_element_by_class_name('W_btn_a')
username.clear()
username.send_keys(account['usn'])
pwd.clear()
pwd.send_keys(account['pwd'])
login_submit.click()
time.sleep(5)
# if browser.find_element_by_class_name('verify').is_displayed():
# logging.error("Verify code is needed! (Account: %s)" % account)
if "我的首页 微博-随时随地发现新鲜事" in browser.title:
browser.get('http://weibo.cn/')
cookie = dict()
if "我的首页" in browser.title:
for elem in browser.get_cookies():
cookie[elem["name"]] = elem["value"]
# p2 = persist_iics.Persist()
# p2.save_account_cookies(accounts[0][0], cookie, datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S"))
logging.error('Account cookies updated! (Account_id: %s)' % account['usn'])
return cookie
刚开始我运行了D:\SinaSpider-master\Sina_spider1\Begin.py(我的路径)没有任何反应,也没有报错
于是我尝试了在命令行进行运行,代 码都没有动 因为之前有人分享过将twisted改到16以下,于是我也试了,改成了Twisted-16.5.0,可还是不行,跪求大神指点
却出了错误,报错如下:
D:\SinaSpider-master\Sina_spider1>scrapy crawl sinaSpider -s LOG_LEVEL=ERROR
D:\SinaSpider-master\Sina_spider1\Sina_spider1\spiders\spiders.py:4: ScrapyDeprecationWarning: Module scrapy.spider
is deprecated, use scrapy.spiders
instead
from scrapy.spider import CrawlSpider
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/5235640836/follow>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/5235640836/fans>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/attgroup/opening?uid=5235640836>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/5235640836/profile?filter=1&page=1>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/5676304901/follow>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/5676304901/fans>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/attgroup/opening?uid=5676304901>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/5676304901/profile?filter=1&page=1>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/5871897095/follow>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/5871897095/fans>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/attgroup/opening?uid=5871897095>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/5871897095/profile?filter=1&page=1>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/2139359753/follow>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/2139359753/fans>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/attgroup/opening?uid=2139359753>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/2139359753/profile?filter=1&page=1>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/5579672076/follow>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/5579672076/fans>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/attgroup/opening?uid=5579672076>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/5579672076/profile?filter=1&page=1>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/2517436943/follow>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/2517436943/fans>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/attgroup/opening?uid=2517436943>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/2517436943/profile?filter=1&page=1>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/5778999829/follow>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/5778999829/fans>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/attgroup/opening?uid=5778999829>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/5778999829/profile?filter=1&page=1>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/5780802073/follow>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/5780802073/fans>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/attgroup/opening?uid=5780802073>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/5780802073/profile?filter=1&page=1>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/2159807003/follow>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/2159807003/fans>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/attgroup/opening?uid=2159807003>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/2159807003/profile?filter=1&page=1>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/3378940452/follow>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/3378940452/fans>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/attgroup/opening?uid=3378940452>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/3378940452/profile?filter=1&page=1>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/1885080105/follow>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/1885080105/fans>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/attgroup/opening?uid=1885080105>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/1885080105/profile?filter=1&page=1>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/5778836010/follow>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/5778836010/fans>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/attgroup/opening?uid=5778836010>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/5778836010/profile?filter=1&page=1>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/5762793904/follow>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/5762793904/fans>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/attgroup/opening?uid=5762793904>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/5762793904/profile?filter=1&page=1>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/5722737202/follow>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/5722737202/fans>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/attgroup/opening?uid=5722737202>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/5722737202/profile?filter=1&page=1>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/3105589817/follow>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/3105589817/fans>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/3105589817/profile?filter=1&page=1>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/attgroup/opening?uid=3105589817>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/5882481217/follow>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/5882481217/fans>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/attgroup/opening?uid=5882481217>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/5882481217/profile?filter=1&page=1>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/5831264835/follow>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/5831264835/fans>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/5831264835/profile?filter=1&page=1>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/attgroup/opening?uid=5831264835>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/1756807885/follow>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/1756807885/fans>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/attgroup/opening?uid=1756807885>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/1756807885/profile?filter=1&page=1>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/3637185102/follow>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/3637185102/fans>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/attgroup/opening?uid=3637185102>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/3637185102/profile?filter=1&page=1>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/2717354573/follow>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/2717354573/fans>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/2717354573/profile?filter=1&page=1>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/attgroup/opening?uid=2717354573>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/1934363217/follow>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/1934363217/fans>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/attgroup/opening?uid=1934363217>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/1934363217/profile?filter=1&page=1>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/5336500817/fans>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/5336500817/follow>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/attgroup/opening?uid=5336500817>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/5336500817/profile?filter=1&page=1>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/1431308884/fans>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/1431308884/follow>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/attgroup/opening?uid=1431308884>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/1431308884/profile?filter=1&page=1>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/5818747476/follow>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/5818747476/fans>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/attgroup/opening?uid=5818747476>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/5073111647/follow>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/5818747476/profile?filter=1&page=1>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/5073111647/fans>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/5073111647/profile?filter=1&page=1>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/attgroup/opening?uid=5073111647>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/5398825573/follow>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/5398825573/fans>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/5398825573/profile?filter=1&page=1>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/attgroup/opening?uid=5398825573>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/2501511785/follow>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/2501511785/fans>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/attgroup/opening?uid=2501511785>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
2017-04-24 16:01:07 [scrapy.core.scraper] ERROR: Error downloading <GET http://weibo.cn/2501511785/profile?filter=1&page=1>
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "D:\SinaSpider-master\Sina_spider1\Sina_spider1\middleware.py", line 19, in process_request
cookie = random.choice(cookies)
File "c:\python27\lib\random.py", line 275, in choice
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
IndexError: list index out of range
返回错误信息“为了你的安全,请输入验证码“,应该怎么解决?
mongodb结果表中的Follows表和Fans表,_id后面的的1 10 100 101 ...... 这些列是什么意思?后面跟的都是关注人和粉丝的ID吗?
老是出现这个问题,要崩溃了。。 stackoverflow 和网上其他地方的方法都用了。。但是没用啊
加了代理池 配了分布式 但是突然发现淘宝没有卖微博账号了 求助那里可以购买账号
你好。
我使用SinaSpider1,连续运行24小时后速度会降为原来的1/2,这是为什么呢?
在Information表中有的Id拥有几千粉丝,但是存储的时候只能存几个,应该是翻页有限制,翻了几页之后就翻不下去了。不知道有没有什么比较好的解决方案。谢谢:)
我在单机跑 Sina_spider2 的时候报错,已经成功获得Cookie,并加载中间件和pipelines ,但是Spider刚open就立刻close了。Sina_spider2能不能再一台机器上跑?Tried to stop a LoopingCall that was " AssertionError: Tried to stop a LoopingCall that was not running.
SCHEDULER = 'scrapy_redis.scheduler.Scheduler'
SCHEDULER_PERSIST = True
SCHEDULER_QUEUE_CLASS = 'scrapy_redis.queue.SpiderPriorityQueue'
REDIE_URL = None
REDIS_HOST = 'localhost'
REDIS_PORT = 6379
......
2016-07-02 08:15:15 [scrapy] INFO: Enabled item pipelines:
['Sina_spider2.pipelines.MongoDBPipleline']
2016-07-02 08:15:15 [scrapy] INFO: Spider opened
2016-07-02 08:15:15 [scrapy] INFO: Closing spider (shutdown)
Unhandled error in Deferred:
2016-07-02 08:15:15 [twisted] CRITICAL: Unhandled error in Deferred:
Traceback (most recent call last):
File "/Users/georgezou/Documents/Coding/github/SinaSpider/Sina_spider2/Sina_spider2/commands/crawlall.py", line 37, in run
self.crawler_process.crawl(spidername, **opts.spargs)
File "/Library/Python/2.7/site-packages/scrapy/crawler.py", line 163, in crawl
return self._crawl(crawler, *args, **kwargs)
File "/Library/Python/2.7/site-packages/scrapy/crawler.py", line 167, in _crawl
d = crawler.crawl(*args, **kwargs)
File "/Library/Python/2.7/site-packages/twisted/internet/defer.py", line 1274, in unwindGenerator
return _inlineCallbacks(None, gen, Deferred())
--- <exception caught here> ---
File "/Library/Python/2.7/site-packages/twisted/internet/defer.py", line 1126, in _inlineCallbacks
result = result.throwExceptionIntoGenerator(g)
File "/Library/Python/2.7/site-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator
return g.throw(self.type, self.value, self.tb)
File "/Library/Python/2.7/site-packages/scrapy/crawler.py", line 87, in crawl
yield self.engine.close()
File "/Library/Python/2.7/site-packages/scrapy/core/engine.py", line 100, in close
return self._close_all_spiders()
File "/Library/Python/2.7/site-packages/scrapy/core/engine.py", line 340, in _close_all_spiders
dfds = [self.close_spider(s, reason='shutdown') for s in self.open_spiders]
File "/Library/Python/2.7/site-packages/scrapy/core/engine.py", line 298, in close_spider
dfd = slot.close()
File "/Library/Python/2.7/site-packages/scrapy/core/engine.py", line 44, in close
self._maybe_fire_closing()
File "/Library/Python/2.7/site-packages/scrapy/core/engine.py", line 51, in _maybe_fire_closing
self.heartbeat.stop()
File "/Library/Python/2.7/site-packages/twisted/internet/task.py", line 202, in stop
assert self.running, ("Tried to stop a LoopingCall that was "
exceptions.AssertionError: Tried to stop a LoopingCall that was not running.
2016-07-02 08:15:15 [twisted] CRITICAL:
Traceback (most recent call last):
File "/Library/Python/2.7/site-packages/twisted/internet/defer.py", line 1126, in _inlineCallbacks
result = result.throwExceptionIntoGenerator(g)
File "/Library/Python/2.7/site-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator
return g.throw(self.type, self.value, self.tb)
File "/Library/Python/2.7/site-packages/scrapy/crawler.py", line 87, in crawl
yield self.engine.close()
File "/Library/Python/2.7/site-packages/scrapy/core/engine.py", line 100, in close
return self._close_all_spiders()
File "/Library/Python/2.7/site-packages/scrapy/core/engine.py", line 340, in _close_all_spiders
dfds = [self.close_spider(s, reason='shutdown') for s in self.open_spiders]
File "/Library/Python/2.7/site-packages/scrapy/core/engine.py", line 298, in close_spider
dfd = slot.close()
File "/Library/Python/2.7/site-packages/scrapy/core/engine.py", line 44, in close
self._maybe_fire_closing()
File "/Library/Python/2.7/site-packages/scrapy/core/engine.py", line 51, in _maybe_fire_closing
self.heartbeat.stop()
File "/Library/Python/2.7/site-packages/twisted/internet/task.py", line 202, in stop
assert self.running, ("Tried to stop a LoopingCall that was "
AssertionError: Tried to stop a LoopingCall that was not running.
我想问一下 你单机的时候跑一天不会被封iP么?
也没看到你用代理
你有遇到封IP的情况么
我正在发愁呢
不过我和你的唯一区别就在没用agent
哥们你好,我一运行就提示ImportError: cannot import name CrawlSpider,想问一下CrawlSpider模块是系统自带的吗?还是自己写的,要是自己写的话能不能分享一下?
比如我注意到SinaSpider2里拆分为info和tweet两个爬虫去实现,但是用户info不需要每次都更新(只有微博数、粉丝数改变的频率比较高),而tweet需要每隔一段时间看看有没有更新,怎么将不同爬虫设置不同的爬取频率?
新手,请勿见怪,见笑
http://weibo.cn/%s/follow(简称一)这个是关注列表页面的url,请问如何得到的。
我自己查看只发现http://m.weibo.cn/page/tpl?containerid=1005052026034205_-_FOLLOWERS(简称二)这个url。
对一进行访问,发现可以通过302跳转到二,请问如何得到一那个url
谢谢!!
我把SinaSpider3的验证码识别那一块接入到SinaSpider1 中,经常会出现中文验证码,在敲入中文验证码后会验证失败。求解。
虽然说现在微博对爬取的频率做了限制,不过站在微博的角度思考,人家也很纠结,又想让搜索引擎爬取到数据,又要防止其他爬虫对服务器带来压力。
本人尝试过将爬虫对 UA 改为诸如百度爬虫之类的,可以在不模拟登录的情况下,高频率爬取到很多数据。可以把这条建议追加到 README 中。
最后得到的数据只有原始的start url的数据,没有爬他follower的数据啊。
安装完python2.7之后,
sudo pip install scrapy==1.0.5
sudo pip install pymongo
sudo pip install pyasn1
sudo pip install scrapy-redis
这四个库一定要装上,尤其是**# scrapy限定1.05版本**,最新版的是会报错的。
如果在单机模式玩分布式版本的话需要改settings.py:
settings.py中加上
DUPEFILTER_CLASS = "scrapy_redis.dupefilter.RFPDupeFilter"
不然会报
Failed to instantiate dupefilter class '%s': %s", 'scrapy.dupefilters.RFPDupeFilter
为什么没有见到爬取的坐标信息,是把这个删了吗
爬虫一开始就遇到这个问题,老是说downloading网址的时候出错,有没有人遇到过同样问题的
I wonder if the customized CookiesMiddleware bellow will change the cookie per spider. That is, since normally serveral requests are needed for each spider to crawl the target will it (randomly) change the cookie(we know that we provide many cookies for the project containing several spiders) during the this process for each spider?
class CookiesMiddleware(object):
""" 换Cookie """def process_request(self, request, spider): cookie = random.choice(cookies) request.cookies = cookie
I'm tailoring your code for my weibo.com crawler. Weibo.com's anti-crawler is way powerful than that of weibo.cn and m.weibo.cn.
踩过的坑:
a. scrapy版本不对,安装时不要安装自带的,而是用sudo pip install scrapy==1.0.5。如果已经安装自带的,就用sudo pipi uninstall scrapy卸载后,再用sudo pip install scrapy==1.0.5安装。
b.安装mongo后,安装pymongo,然后就可以运行了,不要管之前报的错误。之前看似跟mongo无关的错误,在安装pymongo包后都会消失!
错误类似于:
twisted] CRITICAL: Unhandled error in Deferred:
Traceback (most recent call last):
File "c:\python27\lib\site-packages\scrapy\cmdline.py", line 150, in _run_comm
and
cmd.run(args, opts)
File "c:\python27\lib\site-packages\scrapy\commands\crawl.py", line 57, in run
self.crawler_process.crawl(spname, **opts.spargs)
等等。
c.mongo的管理工具mongoBooster下载地址:http://mongobooster.com/downloads
ITEM_PIPELINES = {"Sina_spider2.pipelines.MongoDBPipleline": 300}
我在公司抓,大概一天只能抓10万条微博,实在是太慢了。
SinaSpider3的cookies文件已经更新了,请问SinaSpider1的cookies文件什么时候能更新一下?或者请问哪个小伙伴有可以输入验证码的SinaSpider1?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.