first,thanks jprichardson:) i appreciate you.

i want to get image title & url from a image file with scraping.

I've looked around so long, but never resolve.
this is a condition,not using "Google custom API".
I searched some way, but i couldn't resolve yet.

please help me!!

Very useful module, thank you for it!
How can I find videos only? I know that I have to add tbm=vid in GET like a
google.requestOptions = { qs : {'tbm' : 'vid'} };
But It is not working.

how to get more then 200 post of last 24 hours from google using GOOGLE NPM?

res.next => get undefined all time.

how to get next page data or more then 200 post?

An ability to choose in what tab you do the search request. It should be an easy adjunction to the main URL.

for the 'All' tab it's empty.
for the 'News' tab: &tbm=nws
for the 'Videos' tab: &tbm=vid
for the 'Shop' tab: &tbm=shop (though I don't think the links path is the same for this one, might not work)

When using the API as:
google(slackComment.toLowerCase().replace('.google ',''), function (err, next, links){
for (var i = 0; i < links.length; ++i) {
console.log(links[i].title + ' - ' + links[i].link) // link.href is an alias for link.link
console.log(links[i].description + "\n")
}
var botPayload = {
text : "google unavailable"
};
respond(res, userName, botPayload);
})

links variable is always empty.

I saw you can search in the last hour, day, month and year. Is there a possibility to search at a specific time frame ?

eg get search result from 1 January 2018 , at 16:30 till 2 January 2018 12:15 ? Or at least from 1 January 2018 till 2 January 2018 ?

Any idea how to do this?

I have found that in the google.js on line 44
URL = URL.indexOf('tbs=qdr:') >= 0 ? URL.replace(/tbs=qdr:[snhdwmy]\d*/, 'tbs=qdr:' + google.timeSpan) : URL.concat('&tbs=qdr:', google.timeSpan)
there is a google timeSpan but I am wondering how can I construct some time span to support options like this ?

I get this result when googling:

{ title: 'We Feel Fine / by Jonathan Harris and Sep Kamvar',
link: null,
description: 'www.wefeelfine.org/ - Cached - SimilarAn exploration of human emotion, in six movements by Jonathan Harris and Sep Kamvar. Open We Feel Fine. Book · Mission · Movements · Methodology ...‎We Feel - ‎We Feel Fine - ‎Movements - ‎Gallery' }
.
{ title: 'iFeeL',
link: null,
description: 'www.ifeel.com.my/ - Cached - Similarifeel 1月杂志Promo. 搜寻 Search. ifeel 1月特号出版啦！ Go Above 2013! ... Copyright � 2012 ifeel. All Rights Reserved. Scroll to Top. PRIVACY POLICY ...' }
.
{ title: 'I Feel London',
link: null,
description: 'www.ifeellondon.com/ - Cached - Similarabout. Navigate your city by mood.' }
.
{ title: 'I feel dizzy! - YouTube► 1:41www.youtube.com/watch?v=__dUlY0JpSc25 May 2012 - 2 min - Uploaded by TheDizzays8Waking up after anesthesia ORIGINAL VERSION http://www.ifeeldizzay.com.',
link: null,
description: '' }

Link is empty and the link is either in title or description.

playing with it and its a fun fast way to dive into google search api without setting it up.

i know its more of a cli tool but would it be valuable for exporting json?

When you search for hello lyrics, you get this empty result.

{ title: 'News for hello lyrics',
  link: null,
  description: '',
  href: null }

when use google u need rejectUnauthorized: false by default in your request's

Hi, is possible return also de body of the page, to do more scraping? like get the content on the left of the page of google?
can be done easly changing:

callback(null, nextFunc, links, body); //in line 74
callback(new Error('E......dy), null, null, null); // in line 76

Or add a object that include body & response

Cheers, and great job!

always responds with links: [] and no error

I see your using request for getting the source, any way you could pass in an object to request instead of just the URL?
That we you could set a proxy property on the google variable and use that property (if set) as the value for the proxy key in the request object.

I could send a pull request in a couple of days. just a little busy at the moment. that ok?

All searches are returning "null".
res == null

What happened?

Note: I'm using this module to make a Discord bot.

Using this inside company firewall may not work because I cannot go to https://www.google.com.

Can we have option for http here?

var URL = 'https://www.google.%s/search?hl=%s&q=%s&start=%s&sa=N&num=%s&ie=UTF-8&oe=UTF-8&gws_rd=ssl'

i need to get about 500.000 domains with dot something so i will be calling this a lot, i don't know i would be able to prevent the "abuse".

Thanks
Claudio

The Next string used to detect if the next page is present is hardcoded.
If you set a language different from en you can't go after the first page because the string to be used for detection changes with language.

Expose a config variable google.nextText for the Next string and warn user if language is not en and google.nextText string is the default one would be useful. Another option is to map the next string for every language.

What do you think?

I was just wondering if you've noticed search results ending after a certain number of pages, when it appears there are more pages of results available when performing the same query with your browser. I I can provide an example if it would be helpful.

Would it be possible to add a startAt option? E.g.

google.startAt = 100

to start from the 100th result?

Now that this module can google different Google versions (ru, eu, etc) and can support proxies, it makes sense to have a factory method.

when you google something about date, time, currency, weather, google will provide the answer for you, like

'what time is it?'
'usd eur'

what do you think about adding this kind of results?

Hi @jprichardson ,
First thanks for this awesome project.
But I have a small issue with this module in Meteor. When I try to use proxy in request option sometimes for some proxy my app restarted and crash completely with following error:

W20160607-14:55:35.706(4.5)? (STDERR) events.js:72
W20160607-14:55:35.707(4.5)? (STDERR)         throw er; // Unhandled 'error' event
W20160607-14:55:35.707(4.5)? (STDERR)               ^
W20160607-14:55:35.710(4.5)? (STDERR) Error: read ECONNRESET
W20160607-14:55:35.710(4.5)? (STDERR)     at errnoException (net.js:905:11)
W20160607-14:55:35.711(4.5)? (STDERR)     at TCP.onread (net.js:559:19)
=> Exited with code: 8

This is my code:

google.requestOptions = {
            proxy: proxy,
            timeout: 3000
        };
        google.resultsPerPage = 50;

        let syncGoogle = Meteor.wrapAsync(google);

        let googleResult;
        try {
            googleResult = syncGoogle(keyword);
            for (let i = 0; i < googleResult.links.length; ++i) {
                // some process on result.
            }
        }
        catch (error) {
            console.error("error: " + error);
        }

Can you help me to fix this?

Hey,
Thank you for this excellent module, Have been working for my app perfectly from the last 2 years.

But, few days ago, it stopped working and i am getting this error:

/Users/ayushnarula/Desktop/test.js:9
for (var i = 0; i < res.links.length; ++i) {
^

TypeError: Cannot read property 'length' of undefined
    at /Users/ayushnarula/Desktop/test.js:9:32
    at Request._callback (/Users/ayushnarula/node_modules/google/lib/google.js:57:7)
    at Request.self.callback (/Users/ayushnarula/node_modules/google/node_modules/request/main.js:122:22)
    at emitTwo (events.js:87:13)
    at Request.emit (events.js:172:7)
    at Request.<anonymous> (/Users/ayushnarula/node_modules/google/node_modules/request/main.js:661:16)
    at emitOne (events.js:82:20)
    at Request.emit (events.js:169:7)
    at IncomingMessage.<anonymous> (/Users/ayushnarula/node_modules/google/node_modules/request/main.js:623:14)
    at emitNone (events.js:72:20)  

Wondering if google had changed its html results structure

Thanks again,
Cheers

I want to skip some results that i already visited. How can i skip some results based on a condition in my code?

Hey there,

I am using your library to pull results from Google, but I can't return the links array outside of the callback:

_searchWithGoogle: function(search, comp) {
        var result = [];
        google(GOOGLE_SEARCH_QUERY_BASE_ALT + comp + '"', function (err, next, links){
            result = links;
        });
        return result;
},

How do I pull links outside? When I call this method, I always have undefined.

Thank you,

https://history.google.com/history/

Shouldn't this be hidden? Or in a deep mode of incognito?

Handle cookies so that you can retrieve more than 10 results per query. This should be possible with request since it has a cookie support.

Hello,

Congrats on your awesome tool, we're wondering if we can use some of your work inside our tool?

Disclaimer: We've just released a Serp API Tool, and also a Node.js library. We think we can use part of your code to improve a bit both. We know you've released your code under a MIT License, but we rather ask you directly. It's perfectly fine if you rather not.

Thanks!

Hi,

Google doesn't use anymore the TLD (Google.com, .fr, ...) in the ranking algorithm since last week.

Important consequences:

• Changing the var "google.lang" will still have an impact
• Changing the var google.tld won't have an impact

To get the Geographically AND Language based results:

• Google uses (IP from the client querying OR IF SET the parameters "Region Settings" in https://www.google.com/preferences ) AND language parameter set https://www.google.hr/preferences?#languages

If you want to get Google results like French with an IP based in Germany, 2 options:

• Having and IP address in Germany + set the language to German
• Changing the "Region Settings" AND the language

Awesome module but defnitly could use support for enabling / disabling safe search 😊

Hi
I'm beginner in git, nodejs and heroku.
Please give me a step by step instructions to install and use this package on heroku.

thanks.

Recently I found an application using this with vulnerabilities on install... upon review i realized it was google. After looking into the package.json and understanding the depth of changes, I realized anything using cheerio pre version 0.22.0 probably would be difficult to refactor.

I agree with #63. This library is unmaintained I am just going to refactor around an API with a key in that application to get the audit to be clean and use best practices.

                                                                             
┌──────────────────────────────────────────────────────────────────────────────┐
│                                Manual Review                                 │
│            Some vulnerabilities require your attention to resolve            │
│                                                                              │
│         Visit https://go.npm.me/audit-guide for additional guidance          │
└──────────────────────────────────────────────────────────────────────────────┘
┌───────────────┬──────────────────────────────────────────────────────────────┐
│ Moderate      │ Prototype Pollution                                          │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Package       │ lodash                                                       │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Patched in    │ >=4.17.11                                                    │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Dependency of │ google                                                       │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Path          │ google > cheerio > lodash                                    │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ More info     │ https://nodesecurity.io/advisories/782                       │
└───────────────┴──────────────────────────────────────────────────────────────┘
┌───────────────┬──────────────────────────────────────────────────────────────┐
│ Low           │ Prototype Pollution                                          │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Package       │ lodash                                                       │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Patched in    │ >=4.17.5                                                     │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Dependency of │ google                                                       │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Path          │ google > cheerio > lodash                                    │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ More info     │ https://nodesecurity.io/advisories/577                       │
└───────────────┴──────────────────────────────────────────────────────────────┘
found 2 vulnerabilities (1 low, 1 moderate)

Links to vulnerabilities:

• https://nodesecurity.io/advisories/782
• https://nodesecurity.io/advisories/577

Is it possible to extend this to pulling in data about a business's popular times? I've looked through the google API for places and don't think they provide this as part of the request process...knowing so little about node.js...I'm not sure if it can be done or not using your code?

Hello,
first of all, my problem was to search on google in a custom interval, which yours code just do for a defined period of time. So I changed your code for searching in custom period of time:
initialize google.timeSpan as:

//for custom date: pass object in the format
// {from: "mm/dd/yyyy",
// to:"mm/dd/yyyy"
// }
google.timeSpan = {
from: "07/16/2017",
to: "07/18/2017"
}

here is my code, in google.js, over the line 40:

if (google.timeSpan) {
if(typeof google.timeSpan == 'object'){
var min = google.timeSpan.from
var max = google.timeSpan.to
var it_= "&tbs="+querystring.escape('cdr:1,cd_min:'+min + ',cd_max:'+max + '&tbm=')
// tbs=cdr:1,cd_min:7/11/2017,cd_max:7/18/2017&tbm=
URL = URL.indexOf('tbs=qdr:') >= 0 ?
URL.replace(/tbs=qdr:[snhdwmy]\d*/, it_ )
: URL.concat(it_)
}
else{
URL = URL.indexOf('tbs=qdr:') >= 0 ? URL.replace(/tbs=qdr:[snhdwmy]\d*/, 'tbs=qdr:' + google.timeSpan) : URL.concat('&tbs=qdr:', google.timeSpan)
}

But now for my help request:
it's simples, it's not with the code. I make my request to Google and I try to wait at least 3-5 or more min for each request but soon or later google catch me and block my requests.

I looked on internet but there are a bunch of things, I think you guys can at least give directions or links for read, good practices and stuffes, anything helps.
Thanks

google.requestOptions = {
  proxy: 'http://12.41.141.10:8080',
  timeout: 30000,
  localAddress: '127.0.0.1',
  jar: true,
  headers: {
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
    'Accept-Encoding': 'gzip, deflate',
    'Accept-Language': 'en;q=0.5',
    'Cache-Control': 'max-age=0',
    'Connection': 'keep-alive',
    'DNT': 1,
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.75 Safari/537.36'
  }
}

If I use a proxy and some other headers, I just get this error:

Error: Error on response:Error: connect EADDRNOTAVAIL 12.41.141.10:8080 - Local (127.0.0.1:58302) : undefined
    at Request._callback (/Users/john/Work/www/myapp/node_modules/google/lib/google.js:98:16)
    at self.callback (/Users/john/Work/www/myapp/node_modules/request/request.js:186:22)
    at emitOne (events.js:96:13)
    at Request.emit (events.js:188:7)
    at Request.onRequestError (/Users/john/Work/www/myapp/node_modules/request/request.js:845:8)
    at emitOne (events.js:96:13)
    at ClientRequest.emit (events.js:188:7)
    at Socket.socketErrorListener (_http_client.js:310:9)
    at emitOne (events.js:96:13)
    at Socket.emit (events.js:188:7)
/Users/john/Work/www/myapp/scrape.js:25
  for (var i = 0; i < res.links.length; ++i) {
                         ^

TypeError: Cannot read property 'links' of null
    at /Users/john/Work/www/myapp/scrape.js:25:26
    at Request._callback (/Users/john/Work/www/myapp/node_modules/google/lib/google.js:98:7)
    at self.callback (/Users/john/Work/www/myapp/node_modules/request/request.js:186:22)
    at emitOne (events.js:96:13)
    at Request.emit (events.js:188:7)
    at Request.onRequestError (/Users/john/Work/www/myapp/node_modules/request/request.js:845:8)
    at emitOne (events.js:96:13)
    at ClientRequest.emit (events.js:188:7)
    at Socket.socketErrorListener (_http_client.js:310:9)
    at emitOne (events.js:96:13)

I need only two searches, when I wrote node program for it and execute, its not returning back on console, I need to press ctrl+c. This will create memory leak problem when you use it with expressjs

Can't figure out why I'm only getting undefined results when running the test?
Not even getting an error, just nothing.
This is what i ran --
var google = require( "./google/lib/google.js" );

google.resultsPerPage = 25;
var nextCounter = 0;

google( 'node.js best practices' , function( err , next , links )
{
if( err )
{
console.error( err );
}
console.log( links.length );
for( var i = 0 ; i < links.length ; i++ )
{
console.log( links[i].title + ' - ' + links[i].link );
console.log( links[i].description + "\n" );
}
if( nextCounter < 4 )
{
nextCounter+=1;
if( next )
{
next();
}
}
});

This project has a ton of potential as a simple, command line SEO tool. I think the following features would be really beneficial:

• Provide input files with keywords. Something like google -q file.csv (or .txt or whatever) and get results for a comma-separated list of keywords.
• Website rank mode. google -w www.somewebsite.com -q "query" (or -q input_file.txt). Outputs results for a specific website with a number ranking.
• Project mode. Use a local JSON file with a structure of your choice, combining the above (a website and a list of keywords) plus a list of TLDs and languages. Specified by input file: google -p mywebsite.json. Returns a nicely structured output for each language/TLD combination and for all the keywords. Could use the colors nom package (and/or others) for pretty output.

I'm not great with node, but if you like this idea and need help, I'll help in any way I can.

Also, I have been working on a frontend for a web-based open-source SEO tool, based on your package. If you are good with the backend stuff, we could work on it together. I'm thinking MVC with express, angularJS and mongoDB.

Searched term "google account"

Module stopped working completely and gives error:

Used the example code: for (var i = 0; i < res.links.length; ++i) {
==^
TypeError Cannot read property 'links' of null

Nothing I do will fix this. It seems completely broken. The search term "google account" was used after searching many things that did work, then boom the whole thing broken.

I'm experimenting with how many searches I can do a minute without getting the captcha. So far I'm up to 4 searches a minute.

How many are you all able to do? Any tricks?

Hi guys,

this module works really nice,

BUT BUT BUT: it STOPPS working pretty quick as google shows me a CAPTCHA.

HOW do you guys prevent that from happening?

I am planning to do around 1000 queries per day.

Has anyone tried out running thru proxies? There are services offering private proxies... ???

The proper way to get Google results is to use an API key. With this module, you'll quickly run into the Google captcha issue (Google will detect an unusual amount of traffic from an IP), without using an API key.

You want to query Google directly using an API key...
https://www.googleapis.com/customsearch/v1?key=<YOUR_GOOGLE_SEARCH_API_KEY>&q=Node JS

Get your Google Search API Key here: https://developers.google.com/custom-search/json-api/v1/overview

This is wayyyy faster too, you get raw JSON results directly from their server in one HTTP call. No time wasted on parsing HTML and if Google changes their HTML structure you are not dead in the water. Also, one less NPM repo in your codebase, better security.