Git Product home page Git Product logo

flickr-scrape's Introduction

Scrape flickr

Installation

  1. Clone the repo

  2. Create a virtualenvironment

virtualenv env
source env/bin/activate
  1. Install requirements

pip install -r requirements.txt

Usage

Get an API key from Flickr and make a file called credentials.json which has the following text in it (replace the credentials with your own):

{"KEY":"YOUR_API_KEY", "SECRET":"YOUR_API_SECRET"}

To scrape for a particular search term:

python scraper.py --search "SEARCH TERM" --bbox "minimum_longitude minimum_latitude maximum_longitude maximum_latitude"

To scrape for a particular group:

python scraper.py --group "GROUP URL"

Where GROUP URL is something like https://www.flickr.com/groups/scenery/pool/

You can also add a lat/lng coordinates to specify a geographic bounding box:

python scraper.py --search "SEARCH TERM" --bbox "minimum_longitude minimum_latitude maximum_longitude maximum_latitude"

Large-sized (1024px width) will be downloaded by default. You can download the original images by passing the flag --original.

Limit the number of pages of results downloaded by passing --max-pages N where N is pages of 500 results each. Specify the start page with --start-page M.

flickr-scrape's People

Contributors

antiboredom avatar dependabot[bot] avatar genekogan avatar ptd006 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

flickr-scrape's Issues

KeyError (line 14)

I get an "KeyError" after inserting my flickr api key and running the script.

Parameters to FlickrAPI does not include the "relevant" parameter

Hello, I tried to use your flickr-scrape code and was having trouble getting images that related to the search queries I was providing. I assume flickr api has changed and so now a "relevant" parameter is needed to provide results relevant to your search query. Here is the only change needed for it to work again (I removed content_type as well although that shouldn't have an effect):
Old
params = { 'content_type': '7', 'per_page': '500', 'media': 'photos', 'format': 'json', 'advanced': 1, 'nojsoncallback': 1, 'extras': 'media,license,realname,%s,o_dims,geo,tags,machine_tags,date_taken' % ('url_o' if original else 'url_l'), #url_c,url_l,url_m,url_n,url_q,url_s,url_sq,url_t,url_z', 'page': page, 'api_key': KEY }
New
params = { 'sort': 'relevance', 'privacy_filter': 1, 'per_page': '500', 'format': 'json', 'advanced': 1, 'nojsoncallback': 1, 'extras': 'media,license,realname,url_o, url_k, url_h, url_l, url_c,o_dims,tags,machine_tags', #url_c,url_l,url_m,url_n,url_q,url_s,url_sq,url_t,url_z', 'page': page, 'api_key': KEY }

I hope you find this is useful,

error with line 7, in <module>

Any idea why I'm getting this error?

Traceback (most recent call last):
File "scraper.py", line 7, in
import requests

Thanks

No module named tqdm

Getting this issue:

Traceback (most recent call last):
File "scraper.py", line 8, in
from tqdm import tqdm
ImportError: No module named tqdm
Devs-MacBook-Pro:flickr-scrape devethanvalladares$ python scraper.py --group "https://www.flickr.com/groups/52481568@N00/pool/page117" --max-pages 5 --start-page 1
Traceback (most recent call last):
File "scraper.py", line 8, in
from tqdm import tqdm
ImportError: No module named tqdm

Any solutions?

json.decoder.JSONDecodeError

Hello,
I created the credential.json file with my credentials, but after running the scraper, I received this error:
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 1 column 9 (char 8)

Could anyone help me with this? Thanks!

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.