sparklemotion / mechanize Goto Github PK
View Code? Open in Web Editor NEWMechanize is a ruby library that makes automated web interaction easy.
Home Page: https://www.rubydoc.info/gems/mechanize/
License: MIT License
Mechanize is a ruby library that makes automated web interaction easy.
Home Page: https://www.rubydoc.info/gems/mechanize/
License: MIT License
This is not a bug or anything - this is me doing something stupid I'm pretty sure ...
I just upgraded to
Now,
/Library/Ruby/Gems/1.8/gems/mechanize-0.9.3/lib/www/mechanize.rb:413:in `post_form'
is not posting form parameters.
I'm running this before my features (was working before upgrade)
Before do
visit TESTING_SERVER
fill_in('myuser_login', :with => "my.name")
fill_in('myuser_password', :with => "xxxxxxx")
puts response.body
click_button('Submit')
end
When I look at my rails logs of the app i'm testing - no params are sent (just the action and controller are shown). I can see that I'm on the right page - and the fields (looking at response body)- but they are not set - and they are not sent. Even when I put in another field that is already populated - the form fields are not posted.
Anyway, if anyone can unstupid me on this, I'd appreciate it :P
Where is that settable ?
Via
agent.get({:url => foo, :res_klass => X }
Doesn't seem to get set, or make any difference
This problem is easily solved by editing the gemspec file and removing all references to the subdirectory 'www'
Hey there,
Whenever i use OpenSSL after opening up a mechanize agent I get the following error:
OpenSSL::SSL::SSLError: SSL_read:: no start line
method sysread in protocol.rb at line 133
method rbuf_fill in protocol.rb at line 133
method timeout in timeout.rb at line 62
method timeout in timeout.rb at line 93
method rbuf_fill in protocol.rb at line 132
method read in protocol.rb at line 86
method read_chunked in http.rb at line 2232
method read_body_0 in http.rb at line 2207
method read_body in http.rb at line 2173
method handle in response_reader.rb at line 15
method handle in chain.rb at line 25
method fetch_page in mechanize.rb at line 490
method request in http.rb at line 1053
method reading_body in http.rb at line 2136
method request in http.rb at line 1052
method fetch_page in mechanize.rb at line 485
method post_form in mechanize.rb at line 413
method submit in mechanize.rb at line 344
at top level in mechanize_problem.rb at line 17
nolan-evanss-macbook-pro:ssl_problem nolanevans$ gem which mechanize
(checking gem mechanize-0.9.3 for mechanize)
/Library/Ruby/Gems/1.8/gems/mechanize-0.9.3/lib/mechanize.rb
nolan-evanss-macbook-pro:ssl_problem nolanevans$ ruby --version
ruby 1.8.6 (2008-08-11 patchlevel 287) [universal-darwin9.0]
Has anyone run into a problem like this?
You can see sample code here:
http://gist.github.com/295276
Cheers,
-Nolan
Error:
test_encoding_override_after_parser_was_initialized(TestPage):
Errno::ENOENT: No such file or directory - /var/tmp/portage/dev-ruby/mechanize-0.9.3-r1/work/all/mechanize-0.9.3/test/htdocs/tc_bad_charset.html
/var/tmp/portage/dev-ruby/mechanize-0.9.3-r1/work/all/mechanize-0.9.3/test/helper.rb:82:in initialize' /var/tmp/portage/dev-ruby/mechanize-0.9.3-r1/work/all/mechanize-0.9.3/test/helper.rb:82:in
open'
/var/tmp/portage/dev-ruby/mechanize-0.9.3-r1/work/all/mechanize-0.9.3/test/helper.rb:82:in request' /var/tmp/portage/dev-ruby/mechanize-0.9.3-r1/work/all/mechanize-0.9.3/lib/www/mechanize.rb:485:in
fetch_page'
/var/tmp/portage/dev-ruby/mechanize-0.9.3-r1/work/all/mechanize-0.9.3/lib/www/mechanize.rb:238:in get' ./test/test_page.rb:52:in
test_encoding_override_after_parser_was_initialized'
Error:
test_encoding_override_before_parser_initialized(TestPage):
Errno::ENOENT: No such file or directory - /var/tmp/portage/dev-ruby/mechanize-0.9.3-r1/work/all/mechanize-0.9.3/test/htdocs/tc_bad_charset.html
/var/tmp/portage/dev-ruby/mechanize-0.9.3-r1/work/all/mechanize-0.9.3/test/helper.rb:82:in initialize' /var/tmp/portage/dev-ruby/mechanize-0.9.3-r1/work/all/mechanize-0.9.3/test/helper.rb:82:in
open'
/var/tmp/portage/dev-ruby/mechanize-0.9.3-r1/work/all/mechanize-0.9.3/test/helper.rb:82:in request' /var/tmp/portage/dev-ruby/mechanize-0.9.3-r1/work/all/mechanize-0.9.3/lib/www/mechanize.rb:485:in
fetch_page'
/var/tmp/portage/dev-ruby/mechanize-0.9.3-r1/work/all/mechanize-0.9.3/lib/www/mechanize.rb:238:in get' ./test/test_page.rb:44:in
test_encoding_override_before_parser_initialized'
Error:
test_page_gets_charset_from_page(TestPage):
Errno::ENOENT: No such file or directory - /var/tmp/portage/dev-ruby/mechanize-0.9.3-r1/work/all/mechanize-0.9.3/test/htdocs/tc_charset.html
/var/tmp/portage/dev-ruby/mechanize-0.9.3-r1/work/all/mechanize-0.9.3/test/helper.rb:82:in initialize' /var/tmp/portage/dev-ruby/mechanize-0.9.3-r1/work/all/mechanize-0.9.3/test/helper.rb:82:in
open'
/var/tmp/portage/dev-ruby/mechanize-0.9.3-r1/work/all/mechanize-0.9.3/test/helper.rb:82:in request' /var/tmp/portage/dev-ruby/mechanize-0.9.3-r1/work/all/mechanize-0.9.3/lib/www/mechanize.rb:485:in
fetch_page'
/var/tmp/portage/dev-ruby/mechanize-0.9.3-r1/work/all/mechanize-0.9.3/lib/www/mechanize.rb:238:in get' ./test/test_page.rb:11:in
test_page_gets_charset_from_page'
I use ruby 1.8.7 (2008-08-11 patchlevel 72) [i386-mswin32] and mechanize (0.9.3), nokogiri (1.3.3)
SSL証明書が無効なサイトにつなぐと以下のエラーが起きます。
2009.08.20 15:50:47:error: can't modify frozen object
xxxxxxxxxxx/lib/ruby/1.8/net/https.rb:138:in verify_mode=' xxxxxxxxxxx/lib/ruby/1.8/net/https.rb:138:in
verify_mode='
xxxxxxxxxxx/gems/mechanize-0.9.3/lib/www/mechanize/chain/ssl_resolver.rb:26:in handle' xxxxxxxxxxx/gems/mechanize-0.9.3/lib/www/mechanize/chain.rb:30:in
pass'
xxxxxxxxxxx/gems/mechanize-0.9.3/lib/www/mechanize/chain/handler.rb:6:in handle' xxxxxxxxxxx/gems/mechanize-0.9.3/lib/www/mechanize/chain/connection_resolver.rb:73:in
handle'
xxxxxxxxxxx/gems/mechanize-0.9.3/lib/www/mechanize/chain.rb:30:in pass' xxxxxxxxxxx/gems/mechanize-0.9.3/lib/www/mechanize/chain/handler.rb:6:in
handle'
xxxxxxxxxxx/gems/mechanize-0.9.3/lib/www/mechanize/chain/request_resolver.rb:27:in handle' xxxxxxxxxxx/gems/mechanize-0.9.3/lib/www/mechanize/chain.rb:30:in
pass'
xxxxxxxxxxx/gems/mechanize-0.9.3/lib/www/mechanize/chain/handler.rb:6:in handle' xxxxxxxxxxx/gems/mechanize-0.9.3/lib/www/mechanize/chain/parameter_resolver.rb:18:in
handle'
xxxxxxxxxxx/gems/mechanize-0.9.3/lib/www/mechanize/chain.rb:30:in pass' xxxxxxxxxxx/gems/mechanize-0.9.3/lib/www/mechanize/chain/handler.rb:6:in
handle'
xxxxxxxxxxx/gems/mechanize-0.9.3/lib/www/mechanize/chain/uri_resolver.rb:72:in handle' xxxxxxxxxxx/gems/mechanize-0.9.3/lib/www/mechanize/chain.rb:25:in
handle'
xxxxxxxxxxx/gems/mechanize-0.9.3/lib/www/mechanize.rb:457:in fetch_page' xxxxxxxxxxx/gems/mechanize-0.9.3/lib/www/mechanize.rb:557:in
fetch_page'
xxxxxxxxxxx/gems/mechanize-0.9.3/lib/www/mechanize.rb:238:in `get'
Although I'm pretty sure it's not a valid header, I have seen this a couple of times on servers that I'm using Mechanize with. I think that it's caused by someone getting gzip encoding really badly wrong somehow.
Patch is very simple and would be nice to have in (if only for me!):
http://github.com/edmund/mechanize/commit/ece2a3cb80285523bde24ef5c33dd11bf6f004c0
Apache complains about uri mismatch when fetching page with some query params:
Digest: uri mismatch - </some/page> does not match request-uri </some/page?some_param=some_value>
AuthHeaders#gen_auth_header
should use full uri.request_uri
not uri.path
It's choking on the "Content-Transfer-Encoding" after the "Content-Disposition" line. It's expecting the "Content-Type" line to be directly after the "Content-Disposition" line.
Source for the java parser clearly shows the root issue:
http://www-inf.int-evry.fr/~meunier/ProjetsIG3/TtImagesRep/node78.html
Note how extractContentType
is directly after all the extractDispositionInfo
handling.
scanning rubyforge
/Library/Ruby/Gems/1.8/gems/rubyforge-1.0.4/lib/rubyforge/client.rb:24:in tmp_dh_callback=': can't modify frozen object (TypeError) from /Library/Ruby/Gems/1.8/gems/rubyforge-1.0.4/lib/rubyforge/client.rb:24:in
use_ssl='
from /Library/Ruby/Gems/1.8/gems/mechanize-0.9.3/lib/www/mechanize/chain/ssl_resolver.rb:25:in handle' from /Library/Ruby/Gems/1.8/gems/mechanize-0.9.3/lib/www/mechanize/chain.rb:30:in
pass'
from /Library/Ruby/Gems/1.8/gems/mechanize-0.9.3/lib/www/mechanize/chain/handler.rb:6:in handle' from /Library/Ruby/Gems/1.8/gems/mechanize-0.9.3/lib/www/mechanize/chain/connection_resolver.rb:73:in
handle'
from /Library/Ruby/Gems/1.8/gems/mechanize-0.9.3/lib/www/mechanize/chain.rb:30:in pass' from /Library/Ruby/Gems/1.8/gems/mechanize-0.9.3/lib/www/mechanize/chain/handler.rb:6:in
handle'
from /Library/Ruby/Gems/1.8/gems/mechanize-0.9.3/lib/www/mechanize/chain/request_resolver.rb:27:in handle' ... 18 levels... from /Library/Ruby/Gems/1.8/gems/omnifocus-1.2.0/lib/omnifocus.rb:53:in
run'
from /Library/Ruby/Gems/1.8/gems/omnifocus-1.2.0/bin/omnifocus:6
from /usr/bin/omnifocus:19:in `load'
from /usr/bin/omnifocus:19
There are "option_with" and "options_with" in Mechanize::Form::SelectList.
Mechanize::Form::MultiSelectList has no option-selection methods now.
While there are tags for releases up to 0.9.2, there is no tag for 0.9.3 and that makes it difficult to download a tarball of the already-released code.
There seems to be a bug when using pretty print on responses from request methods, but only classes inherited from WWW::Mechanize.
The following example causes an exception to be raised, and demonstrates that it only occurs
on inherited classes.
require 'mechanize'
class TestMech < WWW::Mechanize; end
p WWW::Mechanize.new.get('http://www.example.com') # This works just fine...
p TestMech.new.get('http://www.example.com') # but not this.
Here it is in IRB, along the first few lines of the back-trace.
irb(main):001:0> require 'mechanize'
=> true
irb(main):002:0> class TestMech < WWW::Mechanize;end
=> nil
irb(main):003:0> x = TestMech.new.get 'http://www.google.com'
NoMethodError: undefined method parse' for nil:NilClass from /Library/Ruby/Gems/1.8/gems/tenderlove-mechanize-0.9.3.20090623142847/lib/www/mechanize/page.rb:82:in
parser'
from /Library/Ruby/Gems/1.8/gems/tenderlove-mechanize-0.9.3.20090623142847/lib/www/mechanize/page.rb:142:in meta' from /Library/Ruby/Gems/1.8/gems/tenderlove-mechanize-0.9.3.20090623142847/lib/www/mechanize/inspect.rb:22:in
pretty_print'
from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/prettyprint.rb:201:in group' from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/prettyprint.rb:227:in
nest'
from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/prettyprint.rb:200:in `group'
"http://#{uri.host}#{url}"
I think "#{uri.to_s}#{url}" is correct.
boy do I have issues (2)
(ignore me... testing omnifocus-github)
If I want to use the latest changes made to master using Bundler, I can't because there is no .gemspec in the repository.
So, I'm requesting that the generated gemspec from the Rakefile be committed to the repository and maintained. A fork and commit will be produced momentarily.
It would be very useful to be able to get full access to the request and response objects after a navigation. For example, to get the response code (2xx, 3xx) after the navigation when there is no error would be very handy. Additionally, all header information, encoding, response length, etc.
In this case you clear the variable @encoding and rely on the parsed page to detect the encoding. So Page#parser sends Nokigiri nil and no proper conversion takes place.
For example, I get the page 'windows-1251' encoded and it is not converted to utf-8 as it should.
Hi,
I am stumped on how to proceed with this problem.
So I am building an application to scrape data off of the website, biblegateway.com, to get Bible passages, that I can then export the retrieved data to a file.
So I am just trying to get the behavior correct before I write the Ruby script.
So here is what I do:
I fire up an irb console.
irb
I'll declare the required Ruby libraries
require 'rubygems'
require 'mechanize'
I'll then create a new object of the Mechanize class.
agent = Mechanize.new # Callilng WWW::Mechanize.new throws a warning message
I'll then tell it what page to scrape.
agent.get("http://www.biblegateway.com/passage")
I'll then tell it to use the last form on the page, which is the one I am working with.
form = agent.page.forms.last
I'll then find the name of the fields, and set their values
form.search1 = "John 3:16"
form.version1 = "NKJV"
That is all the options needed to get the results, so then submit the form.
form.submit
Now technically speaking, the form does in fact submit. That's not the problem. The problem is, Mechanize is designed to render the results from a new page to a new Mechanize::Page object.
But how they have their website setup, the same page is rendered with the results then loaded on the page, and it uses a get method instead of a post method, and the URL ends up looking like:
http://www.biblegateway.com/passage/?search=John%203:16&version=NKJV
So, what I need to know is, what do I need to do to render the same page in a "get fashion" so to speak? The documentation is very difficult to pick apart, and I haven't had much luck with Google...
Thank you in advance for the help.
It would be nice if the code like
form.field_with(:id => 'foo')
would work as expected. It's possible to do page.at(..) but it's a)counter-intuitive b)Requires manual instantiation of Field class.
If I'm not missing something obvious here, then I guess that something like this would have to be added to the relevant class:
class Mechanize::Form::Field
def id
::Mechanize::Util.html_unescape(@node['id'])
end
end
Some web sites contain links with non-ASCII characters (for example german umlauts), especially in get parameters.
Mechanize seems to break such links, making it impossible to follow that links.
Below I put a small sinatra app demonstrating the problem. Accessing this with a normal browser and clicking on the link will result in a YES. Doing the same with mechanize results in a NO.
Is there a way to work around that?
require 'sinatra' get '/' do '<a href="?query=müller">test</a>'+ params.inspect+ ((params["query"]=="müller") ? "YES":"NO") end #test{"query"=>"m\303\274ller"}YES (Firefox) #test{"query"=>"m%..FE7%..FAB%..FAFller"}NO (Mechanize)
When I try to use CookieJar.load_cookiestxt to parse 3k+ records dump from Chrome cookies database, it cost almost 36mins.
Is cleanup() necessary when call add() everytime?
Results in various failures, 'Rescuing EOF error' or 'wrong status line'. I traced the problem to the ssl_resolver.rb line 23. This checks ! ssl.frozen? I remember this being an issue in earlier version of mechanize. I removed the check and things are dandy again. Hardly a solution.
I'm using Ruby 1.9.1p129 and the very latest Mechanize 0.9.3 gem build directly from source.
If follow_meta_refresh is turned on and body of html page has meta tag for meta-refresh,
mechanize try to follow link on it.
Mechanize-0.9.3 has this issue
A request to http://www.rmfpc.com/ returns an HTML document with a meta refresh tag that looks like this:
<meta http-equiv="Refresh" content="0;URL=home-ruskin.htm" />
Mechanize is parsing the redirect as: 'http://www.rmfpc.comhome-ruskin.htm'
It appears the code that is responsible for this is on line 38 of mechanize/page/meta.rb
url = case url
when nil, "" then uri.to_s
when /^http/i then url
else "http://#{uri.host}#{url}" # <<<<
end
Not sure what the best way to handle this. One option is to simply prepend the parsed url with a '/' if it doesn't already have one, but that may break other requests.
Here is a test script reproducing the issue (this will emit timeout error on ruby 1.8.6):
agent = Mechanize.new
agent.follow_meta_refresh = true
res = agent.get('http://www.rmfpc.com')
Amazon's affiliate page: https://affiliate-program.amazon.com/
Trying to signin to this page results in Amazon stating that you need to enable cookies and redirecting the client to an alternative form, but that form doesn't all results in Amazon saying you need to enable cookies.
I'm not sure if Amazon has some mechanism to prevent Mechanize from working, but if it doesn't this is an interesting case of cookie bug.
The fix appears to be at form.rb, line 150: change "if page" to "if page && page.encoding" in
def from_native_charset(str, enc=nil)
if page
enc ||= page.encoding
Util.from_native_charset(str,enc)
else
str
end
end
Currently mechanize raises an error (mechanize.rb line 610) if it sees an error code like 400 in the response. I'd like to use webrat + mechanize to automate live testing of my staging environment including error cases. So I would like to be able to configure Mechanize to just return the page as is rather than raising an error so that I can assert that a 400 status was returned as expected.
Something like:
raise ResponseCodeError.new(page), "Unhandled response", caller if raise_errors?
page
where Mechanize.raise_errors? would default to true.
So this is a bit perverse, I know, but . . .
The Varnish caching proxy allows for a pseudo HTTP method, "PURGE" -- Here's the description:
http://varnish-cache.org/wiki/FAQ#HowcanIforcearefreshonaobjectcachedbyvarnish
AND
http://varnish-cache.org/wiki/VCLExamplePurging
The big trick here is that the PURGE request has to be identical to the matching GET: The same headers, etc. So piggybacking on the existing functionality for GET would be convenient.
I've written some code using Mechanize to verify our expectations for cache hits/misses based on our rules, and the first thing I want to do is PURGE this way.
At present, I monkeypatch both Mechanize and Net::HTTP -- In Mechanize, I change the fetch_page in #get to . . .
page = fetch_page( :uri => url,
:referer => referer,
:headers => headers || {},
:verb => options[:verb] || :get,
:params => parameters
)
And then for Net::HTTP I add a new class for Purge.
The upshot of all this is that it would be awfully nice if Mechanize would allow the option to be set for the :verb on get. Then the only thing I'd have to monkeypatch is Net::HTTP.
Or . . . the get method could be streamlined to make it easier to monkeypatch for this little thing. E.g., the fetch_page could be isolated in its own method.
ver 0.9.2
>> agent = WWW::Mechanize.new >> agent.get("http://lenta.ru/").links.size => 4 >> require 'hpricot' => [] >> WWW::Mechanize.html_parser = Hpricot => Hpricot >> agent.get("http://lenta.ru/").links.size => 550
from aaron starr via mailing list:
Hi, all,
I found that I had to make the following patch to Mechanize for one of the sites I'm scraping:
alias fetch_page_original_version fetch_page
def fetch_page(params)
params[:uri] = params[:uri].gsub(/^https?:/i) {|m| m.downcase } if String == params[:uri].class
fetch_page_original_version(params)
end
(Also, here: http://pastie.org/1077542)
The problem was that the site was returning a 302 redirect with a Location header that looked like: httpS://www.blah-blah-blah... The weirdly capitalized protocol was causing EOF errors, so it needed to be adjusted.
My versions:
I have found a case where searching for "a" elements yields fewer results with Mechanize than with Nokogiri for the same page.
Code to reproduce:
http://gist.github.com/473014
On my machine, this code prints:
Number of links using Nokogiri: 275
Number of links using mechanize: 128
I'm using mechanize 1.0.0 and nokogiri 1.4.1. Also, after seeing a similar old issue (http://bit.ly/9F1xp7), I installed libxml2 2.2.7, but that didn't change this result.
Hope I'm not missing something obvious.
Mechanize sends "www.host"'s cookie to "wwwxhost", because /www.host/ matches to "wwwxhost".
lib/mechanize/cookie_jar.rb, L34
domains = @jar.find_all { |domain, _|
url.host =~ /#{CookieJar.strip_port(domain)}$/i
}
to
domains = @jar.find_all { |domain, _|
url.host =~ /#{Regexp.escape(CookieJar.strip_port(domain))}$/i
}
This sample code requires WebMock gem.
require 'rubygems'
require 'webmock'
require 'mechanize'
require 'logger'
WebMock.stub_request(:get, 'http://www.host/').to_return(:headers => {'Set-Cookie'=>'NAME=WWW.HOST'})
WebMock.stub_request(:get, 'http://other/')
WebMock.stub_request(:get, 'http://wwwxhost/')
# www.host -> other. Mechanize does not send cookie to other.
Mechanize.new{|a|
a.get('http://www.host/')
sio = StringIO.new
a.log = Logger.new(sio)
a.get('http://other/')
sio.rewind
p sio.read.scan(/request-header: cookie.+?$/)
}
# www.host -> wwwxhost. Mechanize sends cookie to wwwxhost.
Mechanize.new{|a|
a.get('http://www.host/')
sio = StringIO.new
a.log = Logger.new(sio)
a.get('http://wwwxhost/')
sio.rewind
p sio.read.scan(/request-header: cookie.+?$/)
}
result:
[]
["request-header: cookie => NAME=WWW.HOST"]
It would be so cool if mechanize could click on Javascript links. Do you have any thoughts on this and how hard it would be to do? Between Rhino and a bunch of other tools, it seems like all the necessary building blocks are there but they just need to be put together.
rubyforge example is using it ..
It would be nice to have a google group where people who are trying to do things with mechanize could talk to each other.
It seems that if you have multiple form fields with the same name, Mechanize does not Do The Right Thing when submitting the form.
According to the rails API docs, the HTML spec says that the first field encountered with a given name should "win" and subsequent fields with that name should be ignored.
The result is that given HTML like this, the form submitted should only contain the checkbox, not the hidden field.
I've written a failing test case using that HTML, you can see it here. Both are contained in the commit at bleything/mechanize@60bbf4c6ebde41a3326ca4ed7a54b60229c8fcb4, for which I've sent a pull request :)
I tried to subclass WWW::Mechanize, but got errors that parser
was not defined. It turned out, that @html_parser has no default when subclassing, probably because it is not in the initialize method. So I had to define the parser myself, but would have expected it to use the default one.
Hi, I'm having issues logging to LinkedIn using mechanize and it seems to me it has to do with cookie management. The following code returns me the home page but with my user not signed in :
mech = WWW::Mechanize.new
mech.follow_meta_refresh = true
home_page = mech.get('http://www.linkedin.com')
sign_in_link = home_page.links.find{|link| link.text == "Sign In"}
login_form = sign_in_link.click.form('login')
# with email and password variables properly set
login_form.set_fields(:session_key => email, :session_password => password)
return_page = mech.submit(login_form, login_form.buttons.first)
When I use curl to login to LinkedIn and load the cookies in the cookie jar (mech.cookie_jar), it works fine.
I get the error
undefined method instance_variable_defined?' for #<Net::HTTP controlside.ds.corp.kelkoo.net:80 open=false> (NoMethodError) with stacktrace /usr/lib/ruby/gems/1.8/gems/mechanize-0.9.3/lib/www/mechanize/chain/ssl_resolver.rb:20:in
handle'
/usr/lib/ruby/gems/1.8/gems/mechanize-0.9.3/lib/www/mechanize/chain.rb:30:in pass' /usr/lib/ruby/gems/1.8/gems/mechanize-0.9.3/lib/www/mechanize/chain/handler.rb:6:in
handle'
/usr/lib/ruby/gems/1.8/gems/mechanize-0.9.3/lib/www/mechanize/chain/connection_resolver.rb:73:in handle' /usr/lib/ruby/gems/1.8/gems/mechanize-0.9.3/lib/www/mechanize/chain.rb:30:in
pass'
/usr/lib/ruby/gems/1.8/gems/mechanize-0.9.3/lib/www/mechanize/chain/handler.rb:6:in handle' /usr/lib/ruby/gems/1.8/gems/mechanize-0.9.3/lib/www/mechanize/chain/request_resolver.rb:27:in
handle'
/usr/lib/ruby/gems/1.8/gems/mechanize-0.9.3/lib/www/mechanize/chain.rb:30:in pass' /usr/lib/ruby/gems/1.8/gems/mechanize-0.9.3/lib/www/mechanize/chain/handler.rb:6:in
handle'
/usr/lib/ruby/gems/1.8/gems/mechanize-0.9.3/lib/www/mechanize/chain/parameter_resolver.rb:18:in handle' /usr/lib/ruby/gems/1.8/gems/mechanize-0.9.3/lib/www/mechanize/chain.rb:30:in
pass'
/usr/lib/ruby/gems/1.8/gems/mechanize-0.9.3/lib/www/mechanize/chain/handler.rb:6:in handle' /usr/lib/ruby/gems/1.8/gems/mechanize-0.9.3/lib/www/mechanize/chain/uri_resolver.rb:72:in
handle'
/usr/lib/ruby/gems/1.8/gems/mechanize-0.9.3/lib/www/mechanize/chain.rb:25:in handle' /usr/lib/ruby/gems/1.8/gems/mechanize-0.9.3/lib/www/mechanize.rb:457:in
fetch_page'
/usr/lib/ruby/gems/1.8/gems/mechanize-0.9.3/lib/www/mechanize.rb:238:in get' /usr/lib/ruby/gems/1.8/gems/webrat-0.6.0/lib/webrat/adapters/mechanize.rb:18:in
get'
/usr/lib/ruby/gems/1.8/gems/webrat-0.6.0/lib/webrat/core/session.rb:281:in send' /usr/lib/ruby/gems/1.8/gems/webrat-0.6.0/lib/webrat/core/session.rb:281:in
process_request'
/usr/lib/ruby/gems/1.8/gems/webrat-0.6.0/lib/webrat/core/session.rb:122:in request_page' /usr/lib/ruby/gems/1.8/gems/webrat-0.6.0/lib/webrat/core/session.rb:220:in
visit'
(eval):2:in `visit'
I can guess that there is some inconsistency in the versions of mechanize, webrat, ruby
I have the standard ruby (1.8.5) for CentOS
I also tried compiling the latest version of ruby (1.9.1)
but both seems to produce an object Net::HTTP that doesn't have the instance_variable_defined? function :
irb(main):002:0> require 'net/http'
=> true
irb(main):003:0> n = Net::HTTP.new("http://google.fr")
=> #<Net::HTTP http://google.fr:80 open=false>
irb(main):004:0> n.instance_variable_defined?("@something")
NoMethodError: undefined method `instance_variable_defined?' for #<Net::HTTP http://google.fr:80 open=false>
from (irb):4
from /usr/lib/ruby/1.8/net/http.rb:2268
I'm not sure wether this is a bug of mechanize not. However I did solve the problem by changing
line 20 in ssl_resolver.rb from
"if http_obj.instance_variable_defined?(:@ssl_context)"
to "if http_obj.instance_variable_get("@ssl_context")"
Could I submit the patch? Or should I continue to try some other version of mechanize, webrat?
Regards,
Johan Martinsson
I made support for NTLM - see my blog here for the gem I made and you can work out what I did so you can add it in a nice way: http://www.mindflowsolutions.net/2009/5/21/ruby-ntlm-mechanize
[gluenow.com ~/gluenow/current]$ gem list | grep mechanize
mechanize (0.9.3)
[gluenow.com ~/gluenow/current]$ gem list | grep nokogiri
nokogiri (1.3.3)
[gluenow.com ~/gluenow/current]$ ruby script/console production
Loading production environment (Rails 2.2.2)
WARNING: Nokogiri was built against LibXML version 2.6.30, but has dynamically loaded 2.6.31
...
>> page = agent.submit(login_form)
TypeError: can't convert nil into String
from /opt/local/lib/ruby/gems/1.8/gems/mechanize-0.9.3/lib/www/mechanize/util.rb:40:in iconv' from /opt/local/lib/ruby/gems/1.8/gems/mechanize-0.9.3/lib/www/mechanize/util.rb:40:in
from_native_charset'
from /opt/local/lib/ruby/gems/1.8/gems/mechanize-0.9.3/lib/www/mechanize/form.rb:152:in from_native_charset' from /opt/local/lib/ruby/gems/1.8/gems/mechanize-0.9.3/lib/www/mechanize/form.rb:144:in
proc_query'
from /opt/local/lib/ruby/gems/1.8/gems/mechanize-0.9.3/lib/www/mechanize/form.rb:143:in map' from /opt/local/lib/ruby/gems/1.8/gems/mechanize-0.9.3/lib/www/mechanize/form.rb:143:in
proc_query'
from /opt/local/lib/ruby/gems/1.8/gems/mechanize-0.9.3/lib/www/mechanize/form.rb:166:in build_query' from /opt/local/lib/ruby/gems/1.8/gems/mechanize-0.9.3/lib/www/mechanize/form.rb:165:in
each'
from /opt/local/lib/ruby/gems/1.8/gems/mechanize-0.9.3/lib/www/mechanize/form.rb:165:in build_query' from /opt/local/lib/ruby/gems/1.8/gems/mechanize-0.9.3/lib/www/mechanize/form.rb:214:in
request_data'
from /opt/local/lib/ruby/gems/1.8/gems/mechanize-0.9.3/lib/www/mechanize.rb:401:in post_form' from /opt/local/lib/ruby/gems/1.8/gems/mechanize-0.9.3/lib/www/mechanize.rb:344:in
submit'
from (irb):9
patched, working after
in mechanize-0.9.3/lib/www/mechanize/util.rb
BEFORE
40 Iconv.iconv(code, "UTF-8", s).join("")
AFTER
40 Iconv.iconv(code.to_s, "UTF-8", s.to_s).join("")
require 'rubygems'
require 'mechanize'
agent=Mechanize.new
agent.user_agent='wget' # default user-agent string is banned apparantly
page=agent.get('http://en.wikipedia.org/wiki/Comparison_of_file_archivers')
page.search('table.wikitable:first-child') #does not work
page.search('.wikitable:first-child') #does not work
page.search('table:first-child') #works - but it's not a .wikitable table
page.search('table.wikitable') #works - returns 5 tables
The error is:
> page.search('table.wikitable:first-child')
RuntimeError: xmlXPathCompOpEval: function first-child not found
from /usr/lib/ruby/gems/1.8/gems/nokogiri-1.4.0/lib/nokogiri/xml/node.rb:142:in `evaluate'
from /usr/lib/ruby/gems/1.8/gems/nokogiri-1.4.0/lib/nokogiri/xml/node.rb:142:in `xpath'
from /usr/lib/ruby/gems/1.8/gems/nokogiri-1.4.0/lib/nokogiri/xml/node.rb:139:in `map'
from /usr/lib/ruby/gems/1.8/gems/nokogiri-1.4.0/lib/nokogiri/xml/node.rb:139:in `xpath'
from /usr/lib/ruby/gems/1.8/gems/nokogiri-1.4.0/lib/nokogiri/xml/node.rb:106:in `search'
from (irb):119
from /usr/local/lib/site_ruby/1.8/rubygems.rb:168
RuntimeError: xmlXPathCompOpEval: function first-child not found
from /usr/lib/ruby/gems/1.8/gems/nokogiri-1.4.0/lib/nokogiri/xml/node.rb:142:in `evaluate'
from /usr/lib/ruby/gems/1.8/gems/nokogiri-1.4.0/lib/nokogiri/xml/node.rb:142:in `xpath'
from /usr/lib/ruby/gems/1.8/gems/nokogiri-1.4.0/lib/nokogiri/xml/node.rb:139:in `map'
from /usr/lib/ruby/gems/1.8/gems/nokogiri-1.4.0/lib/nokogiri/xml/node.rb:139:in `xpath'
from /usr/lib/ruby/gems/1.8/gems/nokogiri-1.4.0/lib/nokogiri/xml/node.rb:106:in `search'
from (irb):119
from /usr/local/lib/site_ruby/1.8/rubygems.rb:168
Using Mechanize 1.0.0.
When HTML is:
Mechanize sends "imgbtn=&imgbtn.x=0&imgbtn.y=0" to the server.
InternetExplorer, Firefox, and w3m send "imgbtn.x=0&imgbtn.y=0".
On the June 24th commit, cookie_jar.rb file. http://github.com/tenderlove/mechanize/blob/0cb78c906ea339e16829636be47fe35b5dc8be6f/lib/www/mechanize/cookie_jar.rb
On the #add method
def add(uri, cookie)
return unless uri.host =~ /#{CookieJar.strip_port(cookie.domain)}$/i
normal_domain = cookie.domain.downcase
unless @jar.has_key?(normal_domain)
@jar[normal_domain] = Hash.new { |h,k| h[k] = {} }
end
@jar[normal_domain][cookie.path][cookie.name] = cookie
cleanup
cookie
end
I sometimes get an error on
@jar[normal_domain][cookie.path][cookie.name] = cookie
complaining about "[]= called on nil object".
I suspect that when a cookie_jar is loaded from YAML, the default value for @jar[normal_domain] is not being correctly set. I'm currently adding
@jar[normal_domain][cookie.path] ||= {}
@jar[normal_domain][cookie.path][cookie.name] = cookie
Why is thera a form.fields_with(:name => "blah etc")
and not a
form.fields.first.options_with(:value => "the right value") ?
Bit confusing that there's only an ".options" that is an ordinary Array.
Mechanize 1.0.0
Feature request to selectively turn off Content-Length checking? (???)
require 'rubygems'
require 'mechanize'
require 'logger'
a = Mechanize.new
a.user_agent = "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7 GTB7.0"
a.log = Logger.new(STDOUT)
p = a.get(:url => "http://www.retailmenot.com/view/abesofmaine.com")
If I comment-out the Content-Length check in Mechanize::Chain::ResponseReader, I do get the full page (and the length is the reported Content-Length).
Maybe there should be an option to suppress the Content-Length check? Not sure if it should be at the Mechanize-level, or at the level of a particular request.
In any case, I'm sure not getting why it would be EOF'ing anyways.
Log ------
I, [2010-02-16T14:34:21.230419 #4270] INFO -- : Net::HTTP::Get: /view/abesofmaine.com
D, [2010-02-16T14:34:21.230557 #4270] DEBUG -- : request-header: accept-language => en-us,en;q=0.5
D, [2010-02-16T14:34:21.230602 #4270] DEBUG -- : request-header: connection => keep-alive
D, [2010-02-16T14:34:21.230643 #4270] DEBUG -- : request-header: accept => /
D, [2010-02-16T14:34:21.230684 #4270] DEBUG -- : request-header: accept-encoding => gzip,identity
D, [2010-02-16T14:34:21.230724 #4270] DEBUG -- : request-header: user-agent => Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7 GTB7.0
D, [2010-02-16T14:34:21.230766 #4270] DEBUG -- : request-header: accept-charset => ISO-8859-1,utf-8;q=0.7,*;q=0.7
D, [2010-02-16T14:34:21.231365 #4270] DEBUG -- : request-header: host => www.retailmenot.com
D, [2010-02-16T14:34:21.231416 #4270] DEBUG -- : request-header: keep-alive => 300
D, [2010-02-16T14:34:21.353167 #4270] DEBUG -- : Read 550 bytes
D, [2010-02-16T14:34:21.353423 #4270] DEBUG -- : Read 914 bytes
D, [2010-02-16T14:34:21.353894 #4270] DEBUG -- : Read 1938 bytes
D, [2010-02-16T14:34:21.354088 #4270] DEBUG -- : Read 2302 bytes
D, [2010-02-16T14:34:21.354302 #4270] DEBUG -- : Read 3326 bytes
D, [2010-02-16T14:34:21.354510 #4270] DEBUG -- : Read 3690 bytes
D, [2010-02-16T14:34:21.460328 #4270] DEBUG -- : Read 4714 bytes
D, [2010-02-16T14:34:21.460593 #4270] DEBUG -- : Read 5078 bytes
D, [2010-02-16T14:34:21.461439 #4270] DEBUG -- : Read 6102 bytes
D, [2010-02-16T14:34:21.461647 #4270] DEBUG -- : Read 6466 bytes
D, [2010-02-16T14:34:21.462294 #4270] DEBUG -- : Read 7490 bytes
D, [2010-02-16T14:34:21.462503 #4270] DEBUG -- : Read 7854 bytes
D, [2010-02-16T14:34:21.500857 #4270] DEBUG -- : Read 8878 bytes
D, [2010-02-16T14:34:21.501224 #4270] DEBUG -- : Read 9437 bytes
E, [2010-02-16T14:34:21.501605 #4270] ERROR -- : Rescuing EOF error
D, [2010-02-16T14:34:21.729018 #4270] DEBUG -- : Read 550 bytes
D, [2010-02-16T14:34:21.729343 #4270] DEBUG -- : Read 1574 bytes
D, [2010-02-16T14:34:21.729561 #4270] DEBUG -- : Read 2598 bytes
D, [2010-02-16T14:34:21.729748 #4270] DEBUG -- : Read 3622 bytes
D, [2010-02-16T14:34:21.729930 #4270] DEBUG -- : Read 3690 bytes
D, [2010-02-16T14:34:21.831662 #4270] DEBUG -- : Read 4714 bytes
D, [2010-02-16T14:34:21.831962 #4270] DEBUG -- : Read 5738 bytes
D, [2010-02-16T14:34:21.832202 #4270] DEBUG -- : Read 6466 bytes
D, [2010-02-16T14:34:21.832404 #4270] DEBUG -- : Read 7490 bytes
D, [2010-02-16T14:34:21.832586 #4270] DEBUG -- : Read 7854 bytes
D, [2010-02-16T14:34:21.899798 #4270] DEBUG -- : Read 8878 bytes
D, [2010-02-16T14:34:21.900050 #4270] DEBUG -- : Read 9437 bytes
E, [2010-02-16T14:34:21.900409 #4270] ERROR -- : Rescuing EOF error
D, [2010-02-16T14:34:22.139545 #4270] DEBUG -- : Read 550 bytes
D, [2010-02-16T14:34:22.139962 #4270] DEBUG -- : Read 1574 bytes
D, [2010-02-16T14:34:22.140176 #4270] DEBUG -- : Read 2598 bytes
D, [2010-02-16T14:34:22.140357 #4270] DEBUG -- : Read 3622 bytes
D, [2010-02-16T14:34:22.140538 #4270] DEBUG -- : Read 3690 bytes
D, [2010-02-16T14:34:22.245101 #4270] DEBUG -- : Read 4714 bytes
D, [2010-02-16T14:34:22.245364 #4270] DEBUG -- : Read 5078 bytes
D, [2010-02-16T14:34:22.245758 #4270] DEBUG -- : Read 6102 bytes
D, [2010-02-16T14:34:22.245971 #4270] DEBUG -- : Read 6466 bytes
D, [2010-02-16T14:34:22.246691 #4270] DEBUG -- : Read 7490 bytes
D, [2010-02-16T14:34:22.246890 #4270] DEBUG -- : Read 7854 bytes
D, [2010-02-16T14:34:22.300396 #4270] DEBUG -- : Read 8878 bytes
D, [2010-02-16T14:34:22.300651 #4270] DEBUG -- : Read 9437 bytes
E, [2010-02-16T14:34:22.301006 #4270] ERROR -- : Rescuing EOF error
Mechanize does not re-parse when Page#encoding='s argument equals Page#encoding.
In non-utf8/16/32 multibyte HTML, it sometimes occurs that "Nokogiri fails parsing and the Nokogiri returns the correct encoding".
It is OK
multibyte characters
...
It sometimes fails parsing (agent.page.parser.errors returns "Input is not proper UTF-8, indicate encoding" errors)
multibyte characters
...
Nokogiri::HTML::Document#encoding just returns meta['content'] charset token (if you set nothing to Nokogiri.parse 3rd argument).
"Nokogiri::HTML::Document#encoding is same as a meta charset" does not always stand for the success of parsing HTML.
#!ruby -Ku
# This script should be saved in UTF-8 and run under Ruby1.8
require 'rubygems'
require 'mechanize'
require 'kconv'
# "テスト" is "test" in Japanese, "tosjis" changes encoding into Japanese "Shift_JIS"
bad_html = <<HTML.tosjis
<html>
<title>テスト</title>
<meta http-equiv="Content-Type" content="text/html; charset=Shift_JIS">
</html>
HTML
class Mechanize
def local(p)
page = Page.new(URI.parse(p[:url].to_s), {'content-type' => 'text/html'}, p[:html]||p[:body], '200', self)
add_to_history(page)
page
end
end
agent = Mechanize.new
agent.local(:url => 'http://example.com/', :body => bad_html)
puts "Nokogiri errors: #{agent.page.parser.errors}"
puts "title is same?: #{!!(agent.page.at('title').inner_text == 'テスト'.toutf8)}"
# lazy-re-parsing of Page#encoding= does not work because Page#encoding is already "Shift_JIS"
agent.page.encoding = 'Shift_JIS'
puts "title is same? (after Page#encoding=): #{!!(agent.page.at('title').inner_text == 'テスト'.toutf8)}"
# getting rid of the identity check
agent.page.instance_variable_set(:@parser, nil)
agent.page.encoding = 'Shift_JIS'
puts "title is same? (after @parser=nil): #{!!(agent.page.at('title').inner_text == 'テスト'.toutf8)}"
/Library/Ruby/Gems/1.8/gems/mechanize-0.9.3/lib/www/mechanize/chain/response_reader.rb:18: [BUG] Segmentation fault
ruby 1.8.7 (2008-08-11 patchlevel 72) [universal-darwin10.0]
will gather further details
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.