milost / rssgrab Goto Github PK
View Code? Open in Web Editor NEWA tool for crawling rss feeds on a regular basis
A tool for crawling rss feeds on a regular basis
All grabbers should be recreated from their stored definitions during the start of the system.
We should be able to edit the definition of a grabber. For example, change in which intervals the grabber is executed. The only things that should be editable for now are
The definition of a Grabber should be stored in the database. Use the SmplConnPool to connect to the database.
We need a config file where we can specify things like:
... and maybe other things.
It would be interesting to show some simple stats like:
It should be possible to define a new Grabber and add it to the system.
We should be able to delete a specific grabber from the system.
Every grabber should know how often it should execute itself (every hour, once per day etc)
During the execution, a grabber should fetch his feed and start to download all relevant articles in the feed. Use the request package to download the articles. After successfully downloading the content of a page, it should store this content in the database.
We should be able to configure the database the grabber writes the data to. We could also do this in a YAML config file.
We should be able to Start and Stop a specific grabber. This issue does not directly mean start stop but rather ... schedule for execution or not.
A downloaded article can consist of multiple sites. We should be able to get the entire article by paginating to all the relevant sites and storing them too. To do this there needs to be some mechanism on how to discover that we need to paginate through multiple sites.
During the execution a grabber #5, we are only interested in discovering new articles. Instead of saving articles multiple times (because they are mentioned in multiple subsequent feed grabbing sessions) we only want to identify what's new in a feed that we crawl multiple times.
Therefore, we need a way to memorize what was in the feed the last time we crawled it. Using this information we can easily calculate the difference between the two crawls and only grab the articles that are new.
One possibility is to use a simple caching service like a simple hashmap, Beaker or Redis to store all the URLs that were contained in the feed during the previous crawl.
This issue is directly connected to Issue #5
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.