postmodern / gscraper Goto Github PK

GScraper does not heavily use Mechanize. It can be dropped in favor of Net::HTTP and Nokogiri. We would only need to respect the Set-Cookie header.

You've referenced the WWW constant from xxxxx/gems/gscraper-0.2.4/lib/gscraper/gscraper.rb:156:in `web_agent', please
switch the "WWW" to "Mechanize". Thanks!

It would be cool if you could specify querying via: News, Images, Video, Maps, Shopping, Books, Blogs, Updates and Discussions ...

I can replicate this in irb by trying to resolve a page of a query with video results like "michael jackson":

irb(main):001:0> require 'gscraper'
=> true
irb(main):002:0> q = GScraper::Search.query(:query => "michael jackson")
irb(main):003:0> q.page(1)
=> NoMethodError: undefined method `inner_text' for nil:NilClass

It might be a good idea to treat li.g elements that also contain "videobox" in the class differently or just ignore them by default since it makes more sense not to make them increment the rank counter.

Using the version currently in github repo rather than RubyGems, because the version from RubyGems doesn't include :search_host.

https://gist.github.com/1365681

No matter how i try to query, it just returns the top result over and over.

Any guesses on whats going on?