Git Product home page Git Product logo

weather_api_etl's Introduction

ETL Process

  1. Crawl data from Open Weather API
  2. Load data into SQLite relational database
  3. Schedule process to run every 3 seconds

Requirements


✔️ Python --version 3.7 or above to run the project

✔️ Python libraries:

  • from asyncio.windows_events import NULL
  • import sqlite3 (included in the standard library since Python 2.5).
  • import schedule
  • import requests
  • import json (built-in module)

✔️ Environment: Visual Studio Code, with listed extensions:

  • Python (to run Python source)
  • SQLite Viewer (to view SQLite tables)

✔️ Project Package Management: Poetry


SOURCE FOLDER

  • src/main.py: run this file, which contains whole process
  • src/database.py: contains database operations
  • src/utilities.py: contains file-database processing functions
  • schema/Schema.jpg: database schema image
  • schema/response_description.txt: API data description
  • result/weather.json: response value from API, saved in JSON format
  • result/weather.db: SQLite database

HOW TO RUN

  1. Install Python 3.x version, VS Code (https://code.visualstudio.com/), listed above libraries and extensions
  2. In Terminal of VS Code, run the command to install all required packages
>> pip install poetry
>> poetry install
  1. Run main.py file
    • "Enter file name (i.e: weather.json)": give JSON file a name, this file will store crawled data
    • "Enter database name (i.e: weather.db)": give SQLite database a name
    • "Enter location": city name (i.e: Hanoi, Moscow, New York, Tokyo, Havana, Seoul, Saigon, Paris, Berlin,..)
    • "Enter unit": metric or imperial
  2. If there is no error, the program will produce 2 new files (auto-generate): .json and .db (with crawled data)

weather_api_etl's People

Contributors

luthevy avatar

Watchers

 avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.