This repository contains a scraper built with Python and integrated with GitHub Actions to fetch RSS feeds from BBC multiple times a day. The scraper utilizes the power of GitHub Actions to automate the scraping process.

• Fetches 16 public RSS feeds from BBC. (feeds/rss.csv)
• Runs the scraper 6 times a day to keep the news articles up to date.
• Utilizes GitHub Actions for automation and scheduling.
• Saves scraped data to a MongoAtlas cluster. Additionally, saves polarity/sentiment of each article fetched as well.

The basic project structure is as follows:

root\
  ├─ .github\
  │  └─ workflows\
  │     └─ actions.yml
  ├─ .gitignore
  ├─ feeds\
  │  └─ rss.csv
  ├─ resources\
  │  └─ nltk_data\
  │     └─ sentiment\
  │        ├─ vader_lexicon.xml
  │        └─ vader_lexicon.zip
  ├─ src\
  │  ├─ getcontent.py
  │  ├─ getsentiment.py
  │  ├─ loader.py
  │  ├─ mongoconnect.py
  │  └─ scraper.py
  ├─ main.py
  ├─ requirements.txt
  ├─ README.md
  └─ LICENSE

To run the BBC Scraper locally, you need to have the following prerequisites installed:

• Python 3.9/3.10
• pip (Python package installer)
• MongoDB (Local install)

To get started with the BBC Scraper, follow these steps:

1. Clone this repository to your local machine using the following command:

   $ git clone [email protected]:sarbhanub/bbc-scraper.git

2. Navigate to the project directory:

   $ cd bbc-scraper

3. Install the required Python dependencies:

   $ pip install -r requirements.txt

4. Run the scraper locally:

   $ python main.py

This will fetch articles present in the RSS feeds from BBC and will save it to MongoDB if you're running it locally on port 27017 (default).

The scraper is integrated with GitHub Actions, allowing you to automate the scraping process and schedule it to run multiple times a day and save it directly to a MongoAtlas cluster running on cloud. The workflow file can be found in the .github/workflows directory.

The default configuration is set to run the scraper every 3 hours (8 times a day). You can modify the schedule by editing the .github/workflows/actions.yml file.

This project is licensed under the MIT License. You are free to use, modify, and distribute the code as per the terms of the license.

Please note that scraping websites may be subject to legal restrictions or terms of service. Make sure to review and comply with any applicable terms and conditions when using this scraper. BBC Scraper is provided as-is and the developers are not responsible for any misuse or legal consequences resulting from its use.