getEPG() encoding error... about tvheadend-ng.bundle HOT 5 CLOSED

Was following your post within the Plex forums and decided to do some digging within the Plex Framework.

The JSON.ObjectFromURL() default setup is as follows within the Plex Framework:

JSON.ObjectFromURL(url, values=None, headers={}, cacheTime=None, encoding=None, errors=None, timeout=GLOBAL_DEFAULT_TIMEOUT, sleep=0, follow_redirects=True, method=None, max_size=None)

After doing some digging I found that when the response is read it checks if the encoding value is filled. If so, it then checks the errors value. Its setup as follows:

def __str__(self):
    self.load()
    if self._encoding:
      if self._errors:
        result = str(unicode(self._data, self._encoding, self._errors))
      else:
        result = str(unicode(self._data, self._encoding))
    else:
      result = self._data
    return result

This shows how the string encoding is done for the JSON response (as-well-as all URL responses within Plex Framework). You might try adding errors="ignore" to line 124. For more info on valid errors values, refer here.

Note: I do not use this channel nor have the means to do so.

from tvheadend-ng.bundle.

After removing the raised exception that the author was looking for, this is the real error:

2016-08-06 01:27:02,755 (-b722470) : INFO (__init__:127) - JSON-Request failed: <type 'exceptions.UnicodeDecodeError'>

As far as I can see from the PCAP output, the gzipped response isn't unicode. Hmm.

from tvheadend-ng.bundle.

After doing some digging I found that when the response is read it checks if the encoding value is filled. If so, it then checks the errors value. Its setup as follows:

Thanks for the input. I'm not familiar enough with Python to know what are built-in functions and what belong to imports. Your response has helped me drill down further.

As for your recommendation, if you look in the code at line 127 you'll see that he attempts to make another call with ObjectFromUrl without the encoding parameter.

from tvheadend-ng.bundle.

Couple more notes about the usage with JSON.ObjectFromURL(), when it returns the json object, it will also try and use the encoding flag.

class JSON(SubComponent):
  def _init(self):
    self._lock = self._core.runtime.lock()

  def from_string(self, jsonstring, encoding=None):
    try:
      self._lock.acquire()
      return simplejson.loads(jsonstring, encoding)
    except simplejson.decoder.JSONDecodeError, e:
      self._core.log.warn("Error decoding with simplejson, using demjson instead (this will cause a performance hit) - %s" % str(e.args[0]))
      return demjson.decode(jsonstring, encoding)
    finally:
      self._lock.release()

All of that said, I downloaded your response.gz and did some test outside of Plex's Framework (and recreated how Plex behaves to track down the error). I found from your data that it gives an error when including the encoding flag. It gives the following error for me... (note my position will be different from others as I did some formatting to read the file easier, but didn't change the encoding)

'ascii' codec can't encode character u'\xed' in position 1386: ordinal not in range(128)

This happens when I try and recreate the following portion,

str(unicode(self._data, self._encoding, self._errors))

but if I skip the encoding it works fine.

As for your recommendation, if you look in the code at line 127 you'll see that he attempts to make another call with ObjectFromUrl without the encoding parameter.

Correct, but line 31 has the global cache time set to 1 second. I'm assuming when the first JSON.ObjectFromURL() fails it is still caching the response and reusing the bad cache for the second requests before the 1 second cache time passes.

A workaround, could be to set the JSON.ObjectFromURL() cache time to zero, i.e.

json_data = dict()
  try:
    json_data = JSON.ObjectFromURL(encoding='utf-8', url=url, headers=headers, cacheTime=0)
  except:
    try:
      json_data = JSON.ObjectFromURL(url=url, headers=headers, cacheTime=0)
    except:
      raise Exception("JSON encoding error")

Notes:

• I removed the values=None as it does not need to be included since the channel code never calls for values
• Please do not copy/paste the above code blindly into the __init__.py file, as the spacing will be incorrect and Python will complain and not run. Make sure to use tabs instead of spaces because the author of the __init__.py file used tabs.
• You could also change each except: to except Exception, e: and then add a Log.Error(str(e)) after each except to log each exception and track down the errors

from tvheadend-ng.bundle.

'ascii' codec can't encode character u'\xed' in position 1386: ordinal not in range(128)

Ah ha! I searched for that byte and found it. It is part of the description for a Spanish television show.

Correct, but line 31 has the global cache time set to 1 second. I'm assuming when the first JSON.ObjectFromURL() fails it is still caching the response and reusing the bad cache for the second requests before the 1 second cache time passes.

This is true, in fact you can tell this is the case because the log shows the word "fetching" when it uses the cache.

However, while working on this, I switched his code around -- starting without the encoding:

        json_data = ""
        try:
            json_data = JSON.ObjectFromURL(url=url, headers=headers, values=None, encoding=None)
        except:
            e = sys.exc_info()[0]
            if debug == True: Log("JSON-Request without encoding set failed: %s" % e)
            try:
                json_data = JSON.ObjectFromURL(encoding='utf-8', url=url, headers=headers, values=None, errors=None)
            except:
                e = sys.exc_info()[0]
                raise Exception("JSON-Request with encoding set failed: %s" % e)

    except Exception, e:
        if debug == True: Log(str(e))
        return False
    if debug == True: Log("JSON-Request successfull!")
    return json_data

Yet, the problem remains.

2016-08-07 13:22:33,320 (-b737470) :  INFO (__init__:105) - JSON-Request: getEpgGrid
2016-08-07 13:22:33,352 (-b737470) :  DEBUG (networking:166) - Requesting 'http://127.0.0.1:9981/api/epg/events/grid?start=0&limit=2000'
2016-08-07 13:22:33,426 (-b737470) :  INFO (__init__:128) - JSON-Request without encoding set failed: <type 'exceptions.UnicodeDecodeError'>
2016-08-07 13:22:46,272 (-b737470) :  DEBUG (networking:166) - Requesting 'http://127.0.0.1:9981/api/epg/events/grid?start=0&limit=2000'
2016-08-07 13:22:46,345 (-b737470) :  INFO (__init__:136) - JSON-Request with encoding set failed: <type 'exceptions.UnicodeDecodeError'>
2016-08-07 13:22:46,352 (-b737470) :  INFO (__init__:149) - Failed to fetch EPG!

I've continued this discussion in the Plex forums, since it looks like the problem isn't with tvheadend-ng:
https://forums.plex.tv/discussion/227780/json-objectfromurl-unicode-issue#latest

from tvheadend-ng.bundle.