Git Product home page Git Product logo

pokemon-scraper-wework-eng-11-17's Introduction

Pokemon Scraper

Objectives

  • Set up an SQLite database
  • Scrape and save data into your database
  • Use data to make ruby objects

Overview

In this lab, you will:

  1. Set up your schema
  2. Scrape data
  3. Insert it into your database
  4. Build out methods to manipulate your data.

This mirrors a very "real world" task.

  1. Set up a local database schema
  2. Find something on the internet that you want to capture to your local database (copyright etc. permitting) and "scrape it"
  3. For each interesting thing found on the internet...
  4. Insert its information into your local database's schema by writing methods
  5. ...and then write additional methods to your classes so that you can easily work with your local database

Schema Provided

We have provided a db/schema_migration.sql file that will run the SQL statements to set up your database in db/pokemon.db. You can initialize the database file with touch db/pokemon.db.

Alert: Make sure that your database is located in the db directory!

We can tell sqlite3 to execute a migration script upon a new database with a command like: sqlite3 path/to/database.db < source_of_sql_data.

Scraper and Scrape-Content Provided

In this example, we're going to take information from a web page about those adorable Pokémon. Instead of having you write a network code that retrieves a Pokémon catalog and scrapes it, we're going to provide you a "scraper" class in lib/scraper.rb and it's going to "scrape" a local HTML file (pokemon_index.html). It shouldn't be too hard to see that with just a little bit of extra code we could scrape a "live" web page.

Your scraper should not do the work of inserting rows into the database. Its role is to process the file, it should "hand-off" the creation responsibilities to the Pokemon class. It's appropriate for the scraper to say Pokemon.create... but it's not appropriate for the scraper to open up a connection to the database.

The goal of an ORM class is to "glue" the scraper to the database.

Speaking of ORM classes...

The Pokemon Class

The Pokemon class (lib/pokemon.rb) is responsible for saving, adding, removing, or changing anything about each Pokémon.

It should be handed "raw" Pokémon data from the scraper and handle the ORM work to put the data into the database. It's probably most appropriate to

Create any methods you think you need to make working with the Pokémon data easier.

Notice that #initialize requires keyword arguments.

A Note On Inserting Into the Database

When you use SQL to insert into a database, you write out the values by hand and insert them into the database. However, when you insert your Pokémon into the database you don't want to insert them into the query via string interpolation because of potential dangerous consequences. Instead, we can do two things:

It is generally recommended to use parameters. Check out this Stack Overflow question for a good example.

View Objectives: on Learn.co and start learning to code for free.

pokemon-scraper-wework-eng-11-17's People

Contributors

tsiege avatar maxwellbenton avatar pletcher avatar bhollan avatar morgvanny avatar fislabstest avatar drakeltheryuujin avatar changamanda avatar dependabot[bot] avatar sgharms avatar jjseymour avatar annjohn avatar victhevenot avatar ruchiramani avatar babyshoes avatar ahimmelstoss avatar octosteve avatar sarogers avatar peterbell avatar lizbur10 avatar joshuabamboo avatar irmiller22 avatar howardbdev avatar gj avatar febbraiod avatar fs-lms-test-bot avatar aturkewi avatar

Watchers

 avatar  avatar  avatar  avatar Joe Cardarelli avatar Sara Tibbetts avatar  avatar Cernan Bernardo avatar Sophie DeBenedetto avatar  avatar Antoin avatar  avatar  avatar Nicole Kroese  avatar Lisa Jiang avatar  avatar

Forkers

btsk18

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.