Git Product home page Git Product logo

1688_spider's Introduction

爬虫项目部署

author:素心

一、项目简介

该项目是获取某网站商品的信息,并将其以API的形式返现。

二、环境

  • CentOS 7.3
  • MySQL 5.7.27
  • Python 3.6.5
  • Nginx

二、源码简介

.
├── back # 备用数据
│   └── back.txt
├── config.ini # 项目主配置文件
├── image # 图片文件
│   ├── demo.png
│   └── static
│       └── image
│           └── demo.png
├── log # 日志文件
│   ├── detail.log
│   └── spider.log
├── requirements.txt # pip环境
├── tools # 项目扩展文件
│   ├── add_ua.py # User-Agent池维护
│   ├── API.py # 项目API文件
│   ├── config.ini # 扩展配置文件
│   ├── db.py # 数据库有关的操作
│   ├── detail.py # 获取商品详情
│   ├── __init__.py # 路径更改文件
│   ├── parse.py # 数据提取规则
│   ├── spider.py # 商品搜索接口
│   └── trans.py # 翻译扩展,因对泰项目,故需要翻译
├── update # 查询参数更新脚本
│   ├── demo.png 
│   ├── get_query.py
│   └── ghostdriver.log
└── web.py # Flask文件

三、项目部署

1. CentOS 7.3 安装

2. Python 3.6.5安装

参考

3. MySQL 5.7.27安装

参考

4. UWSGI安装

pip install uwsgi -i https://pypi.douban.com/simple

创建配置文件(在当前目录下创建uwsgi.ini文件)

[uwsgi]
socket = 本机IP以及端口
chdir = /www
wsgi-file = /www/web.py
callable = app
processes = 4
threads = 2
pythonpath = /www

5.代码部署

四、注意事项

启动项目
systemctl restart redis systemctl restart mysqld nohup python -u web.py > log/pro.log 2&1 &

1688_spider's People

Contributors

xinxinxiangyin09 avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.