Git Product home page Git Product logo

bookspider's Introduction

Book Spider

概况

  1. 包含一个Scrapy爬虫项目
  2. 包含一个Django站点, 用来显示书籍
  3. 爬虫与站点的数据模型相关联
  4. 爬虫实现了以下站点的内容收集:
网站域名 爬虫名称
www.86696.cc douluo
  1. 站点已完成功能:
  • 用户登录
  • 用户书架
  • 评论系统
  • 书籍书签的增加与删除
  • 起点推荐排名获取
  • 手机端样式的适配
  • 按书籍名称搜索
  • 按作者浏览
  • 分类浏览
  • 排行榜
  • 点击排行
  • 收藏排行
  • 推荐排行
  1. 未完成事项:
  • 书目整理
  • 投票系统
  • 书籍更新内容获取方式

安装使用

  1. 安装PostgreSQL(>=9.4)
  2. 安装Python 2.7
  3. 安装Pip
  4. 使用pip安装Scrapy(1.0.) Django(1.9.)
  5. clone本项目
  6. cd booksite && python setup.py develop
  7. 配置Django项目的 local_settings.py 文件,位于:PROJECT_DIR/booksite/booksite,配置数据库, 如:
	DATABASES = {
		'default': {
			'ENGINE': 'django.db.backends.postgresql',
			'NAME': 'bookspider',
			'USER': 'spider',
			'PASSWORD': 'admin',
			'HOST': '127.0.0.1',
		}
	}
  1. 生成数据库 python manage.py syncdb
  2. 进入目录 PROJECT_DIR/bookspider
  3. 使用Scrapy进行抓取, scrapy crawl "爬虫名称"

bookspider's People

Contributors

tkliuxing avatar

Watchers

 avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.