indexing docid with non-ascii characters causes 503 error about indextank-service HOT 3 OPEN

We should support unicode docids. Actually, __validate_docid on api/restapi.py.

Check the code at https://github.com/linkedin/indextank-service/blob/master/api/restapi.py#L45

So it seems the code sending the update to the LogStorage is not supporting non-ascii docids ..

from indextank-service.

I'm not a python expert, but I dug around, I noticed that thrift uses StringIO, and I found this in the python docs:

The StringIO object can accept either Unicode or 8-bit strings, but mixing the two may take some care. If both are used, 8-bit strings that cannot be interpreted as 7-bit ASCII (that use the 8th bit) will cause a UnicodeError to be raised when getvalue() is called.

from http://docs.python.org/library/stringio.html

Maybe this is what's happening (and why it's happening in the middle of a thrift call)?

from indextank-service.

It also seems that when batch indexing, a single document causing this issue in the batch can cause the whole batch to fail and return a 503.

from indextank-service.