Google rate limit kicking in about how2 HOT 12 OPEN

I think that there isn't a shared API key. This might be caused by someone in your network running lots of google api calls. (This probably makes more sense if you are in an office rather than at home).

from how2.

Have the same problem, after playing few minutes I always got an error "You are doing too many requests to Google. You need to wait a bit before trying again"

from how2.

1. The how2 module does not use the Google Search API as it (to the best of my knowlege) has been deprecated in favour of Google Custom Search, which does not suit the needs of the project, not least of all because it imposes severe rate limits.
2. The error actually stems from node-google, the dependency how2 uses for scraping google results, which (as it should) respects the rate limiting Google imposes on what it believes to be (correctly in this case) automated traffic. This appears to be in the neighbourhood of four queries per minute.
3. A realistic (and useful!) fix for this issue would probably involve writing npm modules that implement the same interface as node-google (or a consistent adapter for existing modules) but for scraping bing and duck-duck-go. If searches were then scheduled round-robin from how2, it could perhaps triple the rate limit (depending on their respective limits). If there is sufficient interest in that solution, I'd be happy to take a look at it.

Hope that helps!

from how2.

I didn't even managed to use this once. I always get the rate limit msg

from how2.

Yeah we need someone to write an api to scrape bing or duckduckgo

from how2.

I'm happy to take a look at implementing that in the coming week, if you'd like.

from how2.

@mkohlmyr awesome

from how2.

Pretty sure I have ddg / bing / google basically working with an -e flag for search engine selection. If it sounds interesting I could also add support for an environment variable to set the default.

Let me know if you want any changes to https://github.com/mkohlmyr/how2/commit/4b0cf93f9e87634f125d28c658a313a7e1517e23 before I make a PR. Forgot about the tests!

https://github.com/mkohlmyr/how2/commit/4b0cf93f9e87634f125d28c658a313a7e1517e23
+
https://github.com/mkohlmyr/how2/commit/db290793ddf01d2e6729b566a387d82445cf1f53

Let me know if you want any changes.

from how2.

Ping @santinic re commits above 📢

Happy to squash them if you feel that's cleaner, or if you want me to rename / restructure any of the changes.

from how2.

@mkohlmyr sorry for the delay. It looks really good ! I have two questions:

• I seem to get fewer answers with google compared with the node-google, do you know why?
• With bing, I get this:

|Cannot connect to search engine.
Error: Failed to locate valid search results on page for query: [object Object]```

But, it's amazing :) thanks for your work!

from how2.

• You're right, I didn't include the num querystring parameter for the google URL. What number would you like? node-google sets a default of 10 and a max of 100 I believe.
• I will take a look at that over the next few days, I did notice sometimes bing struggled to return answers for some queries.

I'll hit you back up by the end of the weekend probably with some updates.

from how2.

So I have done some investigating.

Your error is probably a rate limiting page that I was unaware of. It would be easy to improve the printing of the parameters, I could also create a special error message for this case by looking at the source it brings back.

FWIW it also looks like Bing has a tendency to return questions that have been closed by the user, moderator or have no answers, all of which would cause the magic search to reply with "cannot find any reasonable answer for your query". e.g. how2 reverse proxy nginx -e bing.

I have not had time to implement the changes I mentioned, but I will hopefully get to it later this week (after wednesday). If you don't want to wait, feel free to adapt the code I posted!

from how2.