Git Product home page Git Product logo

orbweaver's Introduction

Orb Weaver

Google News RSS made simple and easy.


GitHub forks GitHub stars GitHub issues GitHub license


What is Orb Weaver?

Orb Weaver is an all-in-one tool to extract Google News RSS, get source links and also images for your own news personal project.

Features:

  • Headlines.
  • Geolocation.
  • Topics.
  • Search.
  • Language and country.
  • Cover images.
  • Sources.
  • Amount of articles to collect.

Installation:

    npm install @ivanbogaeb/orbweaver

Usage:

const {orbWeaver} = require('@ivanbogaeb/orbweaver');
/* import {orbWeaver} from '@ivanbogaeb/orbweaver'; */ // You can import the package the way you want!

let weaver = new orbWeaver(); // Initialize

weaver.location = 'AR'; // Sets news location
weaver.language = 'es'; // Sets news language
weaver.extractURLs = true; // Get articles source URL
weaver.images = true; // Get base64 image for each article (Ready for img or canvas)
weaver.verbose = true; // If you like to see how the process is done

console.log(weaver.getTopics()); // Returns array with all available topics
console.log(weaver.getLangCountryMap()); // Returns all the available languages and countries.

(async () => {
    /* Your async function here */
    try {
        let technology = await weaver.topic('Technology', 10); // Extract 10 technology articles

        weaver.images = false; // You can turn off features dynamically!
            
        let headlines = await weaver.headlines(5); // Get only 5 headline articles
        let geolocation = await weaver.geo('New York'); // All the New York articles available
        let search = await weaver.search('Elon Musk', 1); // Last Elon Musk article
     
        if (!technology.length){ // If the array is empty
            console.log("No news were found");
        };

        /* Read the data */
        technology.forEach(({title, origin, link, pubDate, image, related}) => {
            /* Do what you need with the data */
        });
    } catch (error){ // In the case you miss something when you request articles!
        console.log(error);
    };
})();

You can run multiple Orb Weavers at the same time, but keep in mind it's CPU and RAM usage.

Expected output:

weaver.topic() / weaver.headlines() / weaver.geo() / weaver.search()

    /* Successful output */
    [
        ...,
        {
            title: "This is a very interesting article";
            origin: "Myself, of course";
            link: "https://github.com/ivanbogaeb/orbweaver";
            pubDate: "Tue, 11 May 2022 07:00:00 GMT";
            image: "data:image/jpeg;base64,blablablablablabla...";
            related: [ /*Might or might not be empty */
                {
                    title: "Another very interesting article";
                    url: "https://ivanbogaeb.github.io";
                }
            ];
        },
        ...
    ]


    /* Error output */
    // Console output if something happened
    [] // Array or failed property will be empty

weaver.getTopics()

    /* Successful output */
    ['World', 'Nation', 'Business', 'Technology', 'Entertainment', 'Sports', 'Science', 'Health'];

weaver.getLangCountryMap()

    /* Successful output */
    {
        "af": { // You can directly access the language as property
        "language": "Afrikaans", // English translation
        "native": "Afrikaans", // Native
        "country": ["NA", "ZA"] // Countries that speak it
        },
        "agq": {
            "language": "Aghem",
            "native": "Aghem",
            "country": ["CM"]
        },
        "ak": {
            "language": "Akan",
            "native": "Akan",
            "country": ["GH"]
        },
        "smn": {
            "language": "Inari Sami",
            "native": "anarâškielâ",
            "country": ["FI"]
        },
        ...
    };

Things to consider:

  • Language codenames and countries are based on the ISO 2 Letter (Alpha-2 code, ISO 639-1) and ISO 3 Letter (Alpha-3 code, ISO 639-2) Standard Codes for the Representation of Names of Languages.
  • The country is tied to the language you select, so look up for it with getLangCountryMap() function.
  • Google News only returns articles that were written in the language and country you selected. If you are looking for Japanese news in English for example, you will get articles written in English, but not translated native news.
  • The final amount of articles depends on Google. If you ask for 200 articles, you will only get the ones currently available and nothing else, not the 200.
  • Gathering the original URL or Image is a slow process, you have to be patient.

Changelog:

  • 17 May 2022 - Readme update - Version 1.1.1
  • 12 May 2022 - Bug fixes and improvements - Version 1.1.0
  • 11 May 2022 - First Release - Version 1.0.0
  • Read more...

Credits:

License:

CC0 1.0 Universal

orbweaver's People

Stargazers

 avatar

Watchers

 avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.