wrong encoding of non-ASCII characters about mechanize HOT 5 CLOSED

Allright, I think I found the bug in chain/uri_resolver.rb:

sprintf('%%%X', match.unpack($KCODE == 'UTF8' ? 'U' : 'c')[0])

The lowercase "c" unpacks a signed Integer, which will in this case always be negativ, causing sprintf to generate garbage. Just substituting the lowercase "c" by a uppercase "C" to get an unsigned value should do exactly the right thing.

from mechanize.

There seems to be another issue: (even after fixing the above unpack)

Mechanize.new.get("http://localhost:4567/?query=müller").content # YES
Mechanize.new.get("http://localhost:4567/?query=müller").links[0].click.content # NO
Mechanize.new.get("http://localhost:4567/?query=müller").links[0].href # "?query=m\347\253\257ller"

Seems there is something broken while parsing the link.
page.content looks fine, but the extracted link is not correctly encoded.

from mechanize.

If the Server didn't supply a proper encoding[1], the constructor of Mechanize::Page guesses the encoding:

@encoding ||= Util.detect_charset(body)

That encoding is given to Nokogiri, which tries to "decode" all strings based on that encoding.

Commenting out the above line solved the problem. This works for me, but it surely might break cases where encoding is actually important. - Please give the user an option to set an own default encoding and only try to guess it if there is no other way.

[1]: there seems to be a minor issue if the server sent something like "CharSet".

        next unless v =~ /charset/i # case insensitive
        encoding = v[/charset=([^; ]+)/, 1] #case sensitive

from mechanize.

I have fixed the unpack and made the charset be case insensitive but I can't fix (or provide a workaround for) the encoding auto-detect without a test case.

from mechanize.

encoding auto-detect seems to be covered by other bugs in the issues tracker.

from mechanize.