Weird encoding error in __str__ of Story about hackernewsapi HOT 2 CLOSED

I've hit a few encoding issues in the application and while I can't yet answer some questions I have, I'll go ahead and brain dump what I've got rolling around.

Originally I was running into trouble encoding some common unicode characters that are outside the range of ascii (\u201x, which seem to correspond to some annoyingly common characters). I attempted to 'fix' this with 975501c which seemed to solve the issue for me. It seemed that encoding to utf-8 translated the unknown ascii characters into their appropriate unicode characters. However, I was doing my testing from the terminal, which (for me) meant a standard encoding of utf-8. It was being represented elsewhere (HNify responses) as the unicode escape sequence (for example: \u2019 rather than ’ - which is different entirely from ' ).

I think I've got a mental disconnect in how strings are being passed around, because BS4 should by default use unicode if I understand this page. At line 107, you've got:

title = u'%s' % info_cells[2].find('a').string

which should be a unicode object at that point, then at line 195 you've got:

self.title = title.encode('cp850', errors='replace').decode('cp850')

which, if I understand things, is taking unicode --> bytes (encoding cp850 "a code page used under MS-DOS in Western Europe. It is the code page commonly used by the version of MS-DOS underlying Windows ME" ??? ) --> unicode

I haven't fully though through the implications of this beyond the possibility that python is producing some weirdness in encoding (to bytes) and then decoding (to unicode) characters that may be outside the range of cp850 that are missed when replacing errors.

If I get a chance I'll think on it some more, my hunch is to remove the decoding entirely and/or change the character encoding. HN's header indicates charset=utf-8, but it's possible that's unreliable I suppose.

from hackernewsapi.

The problem is encoding strings on both py2 and py3. Some characters like '\xa0' were a pain in the ass, because utf-8 encoding worked in py2 but not in py3. Thus the scattered encodings.

I'm sure there's a better way of doing things. I'll keep trying. If you come up with something, send a PR or leave a comment.

from hackernewsapi.