Scrapy Crawler

Summary

this is appstore crawler by python Scrapy
basic setup plus added progress bar etc.
the result saved as csv file with icon image

Setup

brew install pyenv
brew install pipenv

pipenv install
pipenv install --dev

Run

pipenv shell
cd appstore_crawler
export LANG=en_US.utf8  # to avoid ParserError

run one category

# set category
CATEGORY=photo-video

# this example will crawl urls includes /genre/ios-{category}
# i.e. https://apps.apple.com/us/genre/ios-photo-video/id6008

scrapy crawl appstore \
    -a category=${CATEGORY} \
    -s JOBDIR=crawls/${CATEGORY}-1  # update this number or delete the dir

or run all categories

scrapy crawl appstore \
    -s JOBDIR=crawls/appstore-1 \
    -o csvdata/appstore.csv

you can stop the spider safely at any time (by pressing Ctrl-C)
and resume it later by issuing the same command
- the csv will be appended
c.f. https://doc.scrapy.org/en/latest/topics/jobs.html

Read icon image

from PIL import Image
import numpy as np

icon = Image.open('appstore_crawler/icondata/photo-video/1006639052.png')
img = np.array(icon, 'f')

print(img.dtype)   # dtype('float32')
print(img.shape)   # (246, 246, 3)
# NOTE: the order is RGB (c.f. OpenCV is BGR)

Read CSV result

import pandas as pd

df = pd.read_csv('appstore_crawler/csvdata/photo-video.csv')

df.head()

df.columns
Index(['id', 'category', 'name', 'subtitle', 'url', 'date_published',
       'rating_value', 'rating_count', 'rating_ratio', 'price_category',
       'price', 'price_currency', 'has_in_app_purchases', 'author_name',
       'author_url', 'description'],
      dtype='object')

# show rating ranking for example
df.sort_values(
    ['rating_value', 'rating_count'], ascending=False
    )[['rating_value', 'rating_count', 'name']].head()

Debug scrapy

# for example Evernote
scrapy shell https://apps.apple.com/us/app/evernote/id281796108
# another example Dropbox
scrapy shell https://apps.apple.com/us/app/dropbox/id327630330

(reference) command log of initial setup

export PIPENV_VENV_IN_PROJECT=true
pipenv --python 3.6
pipenv install scrapy tqdm python-dateutil pandas
pipenv run pip list
pipenv graph

pipenv shell
scrapy startproject appstore_crawler
cd appstore_crawler
scrapy genspider appstore example.com

# check
pipenv update --outdated

# do update
pipenv update
pipenv update --dev
pipenv clean

# re-crete virtual env
pipenv --rm
pipenv install
pipenv install --dev

kenfj / scrapy-crawler Goto Github PK

scrapy-crawler's Introduction

Scrapy Crawler

Summary

Setup

Run

Read icon image

Read CSV result

Debug scrapy

(reference) command log of initial setup

Reference

Scrapy

App Store

scrapy-crawler's People

Contributors

Watchers

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent