Comments (2)
I've hit a few encoding issues in the application and while I can't yet answer some questions I have, I'll go ahead and brain dump what I've got rolling around.
Originally I was running into trouble encoding some common unicode characters that are outside the range of ascii (\u201x, which seem to correspond to some annoyingly common characters). I attempted to 'fix' this with 975501c which seemed to solve the issue for me. It seemed that encoding to utf-8 translated the unknown ascii characters into their appropriate unicode characters. However, I was doing my testing from the terminal, which (for me) meant a standard encoding of utf-8. It was being represented elsewhere (HNify responses) as the unicode escape sequence (for example: \u2019 rather than ’ - which is different entirely from ' ).
I think I've got a mental disconnect in how strings are being passed around, because BS4 should by default use unicode if I understand this page. At line 107, you've got:
title = u'%s' % info_cells[2].find('a').string
which should be a unicode object at that point, then at line 195 you've got:
self.title = title.encode('cp850', errors='replace').decode('cp850')
which, if I understand things, is taking unicode --> bytes (encoding cp850 "a code page used under MS-DOS in Western Europe. It is the code page commonly used by the version of MS-DOS underlying Windows ME" ??? ) --> unicode
I haven't fully though through the implications of this beyond the possibility that python is producing some weirdness in encoding (to bytes) and then decoding (to unicode) characters that may be outside the range of cp850 that are missed when replacing errors.
If I get a chance I'll think on it some more, my hunch is to remove the decoding entirely and/or change the character encoding. HN's header indicates charset=utf-8, but it's possible that's unreliable I suppose.
from hackernewsapi.
The problem is encoding strings on both py2 and py3. Some characters like '\xa0' were a pain in the ass, because utf-8 encoding worked in py2 but not in py3. Thus the scattered encodings.
I'm sure there's a better way of doing things. I'll keep trying. If you come up with something, send a PR or leave a comment.
from hackernewsapi.
Related Issues (20)
- user data. HOT 16
- get_comments : IndexError: list index out of range HOT 3
- Get any stories comments.
- Why limit = 30? HOT 6
- IndexError: list index out of range (comments) HOT 5
- Comment.body_html is incomplete HOT 9
- newest and best options do not work in get_stories
- Make tests work HOT 2
- Respect robots.txt HOT 3
- UnicodeEncodeError in my_test_bot.py
- Get comments for a story is breaking when no comments are there on Hackernews page
- HackerNews HTML has changed, API has stopped working HOT 2
- Is there any way to find stories by title? HOT 4
- Add python 3 support and test it
- Cannot parse result HOT 1
- Fix simple typo: explititly -> explicitly
- Wf
- Beast
- Can't access my accounts
- packages/modules/adbd
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from hackernewsapi.