- Use a web API to seed a database with realistic data
In the last lesson, we learned the importance of having seed data in our database. We also learned how to generate randomized data using Faker. Sometimes though, it's preferable to have more realistic data to work with. We could create that data by hand, but there's also a whole lot of structured data out there on the internet for us to use.
One other way to get more realistic data into our database is to seed data from a JSON API. There are many public JSON APIs out there that you can use to access data that other developers have put together on a variety of topics. This Public APIs repository has a great curated list broken down by category, so depending on what kind of data you're looking for, you may be able to find an API here.
For our application, we'll be using the Dungeons and Dragons API as an example. This API doesn't require any authorization (no API key is required), so it's easy to set up.
In this application, we have a migration for one table, spells
:
# db/migrate/20210718144445_create_spells.rb
class CreateSpells < ActiveRecord::Migration[6.1]
def change
create_table :spells do |t|
t.string :name
t.integer :level
t.string :description
end
end
end
And a corresponding Spell
class that inherits from Active Record:
# app/models/spell
class Spell < ActiveRecord::Base
end
This lesson is set up as a code-along, so make sure to fork and clone the lesson. Then run these commands to set up the dependencies and set up the database:
bundle install
bundle exec rake db:migrate
The first step in working with an API is to understand what endpoints it has
available, and how the response is structured. For our purposes, we'll be
using the spells endpoint from the Dungeons and Dragons API to
generate data for a spells
table in our database.
Based on their documentation, we can see a list of all the spells at this endpoint: https://www.dnd5eapi.co/api/spells, and making a request to the endpoint for one single spell, like https://www.dnd5eapi.co/api/spells/acid-arrow, will return a JSON object formatted like this:
{
"index": "acid-arrow",
"name": "Acid Arrow",
"desc": [
"A shimmering green arrow streaks toward a target within range and bursts in a spray of acid. Make a ranged spell attack against the target. On a hit, the target takes 4d4 acid damage immediately and 2d4 acid damage at the end of its next turn. On a miss, the arrow splashes the target with acid for half as much of the initial damage and no damage at the end of its next turn."
],
"higher_level": [
"When you cast this spell using a spell slot of 3rd level or higher, the damage (both initial and later) increases by 1d4 for each slot level above 2nd."
],
"range": "90 feet",
"components": ["V", "S", "M"],
"material": "Powdered rhubarb leaf and an adder's stomach.",
"ritual": false,
"duration": "Instantaneous",
"concentration": false,
"casting_time": "1 action",
"level": 2,
"attack_type": "ranged"
}
So in order to seed our database from this API, we'll need to use Ruby to:
- Make a request to the spells endpoint for the API
- Parse the JSON response into a Ruby hash
- Find the data in that hash that we want to save to our database
- Use Active Record to save the data to the database
To make the request to the API in Ruby, we'll use the
Rest Client gem. This library simplifies the process of making
network requests in Ruby. We've already got this gem in our Gemfile, so let's
experiment with it in our rake console
:
response = RestClient.get "https://www.dnd5eapi.co/api/spells/acid-arrow"
# => <RestClient::Response 200 "{\"index\":\"a...">
spell_hash = JSON.parse(response)
# => {"index"=>"acid-arrow",
# "name"=>"Acid Arrow",
# "desc"=>
# ["A shimmering green arrow streaks toward a target within range and bursts in a spray of acid. Make a ranged spell attack against the target. On a hit, the target takes 4d4 acid damage immediately and 2d4 acid damage at the end of its next turn. On a miss, the arrow splashes the target with acid for half as much of the initial damage and no damage at the end of its next turn."], ...
spell_hash["name"]
# => "Acid Arrow"
spell_hash.keys
# spell_hash
# => ["index",
# "name",
# "desc",
# "higher_level",
# "range",
# "components",
# "material", ...
Awesome! With just a couple lines of code, we were able to make a GET request to the API and parse the response into a Ruby hash.
The process of using the Rest Client gem in Ruby should feel similar to using
fetch
in JavaScript: we make a request using a URL and HTTP verb, which
returns a response object; to work with the response object, we parse it from a
JSON string to a Ruby hash. One key difference is that this code happens
synchronously in Ruby rather than asynchronously. Our program has to wait
for the response before running the next line of code.
Once we have the response data as a Ruby hash, we can interact with it like we would with any other Ruby hash, and access data using bracket notation.
Now that we know how to access the data we need, let's write some Ruby code in
the seeds.rb
file that will communicate with the API and persist data to our
database. Add this code to the seeds.rb
file:
puts "Seeding spells..."
# these are the spells we want to add to the database
spells = ["acid-arrow", "animal-messenger", "calm-emotions", "charm-person"]
# iterate over each spell
spells.each do |spell|
# make a request to the endpoint for the individual spell:
response = RestClient.get "https://www.dnd5eapi.co/api/spells/#{spell}"
# the response will come back as a JSON-formatted string.
# use JSON.parse to convert this string to a Ruby hash:
spell_hash = JSON.parse(response)
# create a spell in the database using the data from this hash:
Spell.create(
name: spell_hash["name"],
level: spell_hash["level"],
description: spell_hash["desc"][0] # spell_hash["desc"] returns an array, so we need to access the first index to get just a string of the description
)
end
puts "Done seeding!"
As you can see, we're making requests for several different spells from the API,
and using the response to create new records in our spells
table with Active
Record.
Working with this JSON API and (many others) mean you need to be comfortable
working with Ruby hashes and arrays, since many API developers will structure
their data as nested hashes and arrays. If you're ever unsure how to get the
data out of the response, try using binding.pry
and experiment in the console.
Now that we've got the new code in the seeds.rb
file, we can run it:
bundle exec rake db:seed
And enter rake console
to explore the new data:
Spell.last
# => #<Spell:0x00007fee74b38138
# id: 4,
# name: "Charm Person",
# level: 1,
# description:
# "You attempt to charm a humanoid you can see within range. It must make a wisdom saving throw, and does so with advantage if you or your companions are fighting it. If it fails the saving throw, it is charmed by you until the spell ends or until you or your companions do anything harmful to it. The charmed creature regards you as a friendly acquaintance. When the spell ends, the creature knows it was charmed by you.">
Success! We've taken the data from the API and used it to populate our database.
Run learn test
now to pass the test and complete this lesson.