The pymongo driver is very strict and if it can't decode a mongo document it raises an

<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="30

To fix this, I wrote a white list of Post IDs: <div class="snippet-clipboard-conte

Wrote a called blaze.py - which goes through a

Unicode decode error about baleen HOT 3 OPEN

bbengfort commented on August 16, 2024

Unicode decode error

from baleen.

Comments (3)

bbengfort commented on August 16, 2024

yougov/mongo-connector#101

from baleen.

bbengfort commented on August 16, 2024

To fix this, I wrote a white list of Post IDs:

Wrote 566142 Posts IDs in 644.759 seconds

Post.objects.count() == 566142

from baleen.

bbengfort commented on August 16, 2024

Wrote a script called blaze.py - which goes through and attempts to find any bad decoding errors in posts:

100%|███████████████████████████████████████████████████████████████████████████████| 566142/566142 [02:09<00:00, 4365.45id/s]
Phase One: wrote 566142 Posts IDs in 2 minutes 9 seconds
100%|█████████████████████████████████████████████████████████████████████████████| 566142/566142 [37:09<00:00, 267.01posts/s]
Phase Two: wrote 2 Post errors in 37 minutes 9 seconds

It only came up with 2 errors:

"571a333ac1808103a0d6067c",'utf-8' codec can't decode byte 0xed in position 48824: invalid continuation byte
"57726c2ac1808103a5ed63d6",'utf-8' codec can't decode byte 0xed in position 21004: invalid continuation byte

from baleen.

Recommend Projects