How to scrap sites which requires login? about cheerio HOT 9 CLOSED

Try using osmosis module. It has login feature. But is limited to the given generic login selector. But it should work in most cases. You can do post request if you want.

Another option you have is to use a headless browser. Try using CasperJs or PhantomJs. It lets you manipulate the DOM and input some fields.

from cheerio.

Cheerio does not retrieve a HTML page for us(or login to access a webpage and scrap the site after), Cheerio is there to parse HTML data and allows us to scrap text, tag, attribute in the HTML page using selectors for example.

from cheerio.

Yah, so scraping sites requires some additional logic to request the page. The authentication should be part of the request logic.

Cheerio is meant to work exclusively with HTML, you give it HTML, you can mess around with the HTML, then you can output it.

The screencast in the README might be worth checking out if you haven't done so already. It shows how you might scrape a page using cheerio and Mikeal's request library. Doesn't cover authentication though.

from cheerio.

Hi Matthew,

Thank you for your reply. I came across many tutorials on scrapping websites with nodejs. But none of them explains how to scrap a site which requires login. If you can do that it will be really help full.

Thanks
koti

from cheerio.

@ekanna, to be honest - I'm not really sure myself. I believe you have to send the appropriate cookie information along with the header when you request the page.

This is probably what I would try: log into a page, grab the appropriate cookie from your browser, use the request library to set the cookie, then try it.

I could be way off though :-)

from cheerio.

@matthew, thank you. I will try using cheerio and request, if i am successful, i will let you know.

from cheerio.

any luck?

from cheerio.

any luck?

from cheerio.

@oseisaac there are a few options mentioned here https://stackoverflow.com/questions/8726079/how-can-i-scrape-sites-that-require-authentication-using-node-js. Hope it helps

from cheerio.