饭否爬虫, 根据用户ID搜集.支持词云展示,发贴时间统计,关键词搜索,数据库更新等功能。
- python3.6
- mysql5.7
- PyMySQL
- fire
-
Create database "spider"
-
Config headers
- Sign in Fanfou
- Open chrome F12, input URL: https://fanfou.com/[UserID]
- Click Network(top side)->fanfou.com(left side)->Request Headers
- Copy the corresponding parameters to self.headers
-
Configure self.table_name used to store the data
-
User ID can be found in the URL of the profile page
python FanFouSpider.py start [UserID] --table=[TableName]
python FanFouSpider.py key_word [TableName] [KeyWord]
# dump to csv
python FanFouSpider.py dump [TableName]
# draw charts, find them in ./output/
python analytics.py draw [TableName]