tvgrabbers / tvgrabpyapi Goto Github PK

Hi Hika, first of all my compliments for taking tvgrabpy to this level. It's truly impressive!

A useful feature would be to make the directory paths customizable, in particular opt_dict['var_dir'] and opt_dict['xmltv_dir'] (and thus implicitly also source_dir).

My ideal would be to be able to set this in the configuration file, but on the command line is fine too.

Hello

I'm trying to use your api 1 with version 3 for belgium on mythtv.

1. setup in xmltv
2. run in xmltv with config file in ./mythtv
3. setup mythtv -> video source pointing to config file in ./Mythtv
4. running mythfilldatabase seems not to find the programs

Hi there, thanks for all efforts with this grabber! It is quite awesome, but I have one thing that does not work as well as I expected it to work. The categories / subcategories in tvheadend are empty for like 90% of the cases.

I wondered if this was because of data going into the SQLite database was lacking or because of translation towards the xml coming from the database... It seems to be that for a LOT of programs this data does not arrive at the database... Is this a known issue?

I used to get EPG from EIT, which worked flawless for many channels, but was lacking logos, did not supply two weeks etc. But if this grabber is not able to deal properly with categories I don't know what's worse ;).

Every now and then the script procudes invalid XML. The file is invalid because there are unescaped quotes (") inside a XMP attribute. I have a hard time reproducing the bug, probably because it there is some interaction with the cache.

winfried@miepie:/var/lib/mythtv$ xmllint pg.xml pg.xml:67062: parser error : attributes construct error <actor role="Burkhard Schulz "Butsch"">Götz Schubert</actor> ^ pg.xml:67062: parser error : Couldn't find end of Start Tag actor line 67062 <actor role="Burkhard Schulz "Butsch"">Götz Schubert</actor>

the script runs normally, without errors, but mythfilldatabase doesn't want to eat the XML.

Yesterday I used this version for my daily updates, and these are the summary statistics:
2017-02-23 07:13:51 : Verzamel statistieken van 1721 programma's voor 16 zenders:
2017-02-23 07:13:51 : Start tijd: 2017-02-23 06:52
2017-02-23 07:13:51 : Eind tijd: 2017-02-23 07:13
2017-02-23 07:13:51 : Duur: 0:21:26.437174
2017-02-23 07:13:51 : 114 pagina('s) opgehaald, waarvan 5 faalden
2017-02-23 07:13:51 : 49 cache vonst(en)
2017-02-23 07:13:51 : 180 succesvolle ttvdb.com verwijzingen
2017-02-23 07:13:51 : 75 misluktte ttvdb.com verwijzingen
2017-02-23 07:13:51 : Tijd/pagina: 11.284536614 seconds
2017-02-23 07:13:51 : 84 pagina('s) opgehaald van theTVDB.com
Today I ran the 2.2 version again and these are the summary stattistics:
2017-02-24 06:46:55 : Fetch statistics for 1606 programms on 16 channels:
2017-02-24 06:46:55 : Start time: 2017-02-24 06:45
2017-02-24 06:46:55 : End time: 2017-02-24 06:46
2017-02-24 06:46:55 : Duration: 0:01:53.098212
2017-02-24 06:46:55 : 107 page(s) fetched, of which 0 failed
2017-02-24 06:46:55 : 564 cache hits
2017-02-24 06:46:55 : 10 succesful ttvdb.com lookups
2017-02-24 06:46:55 : 8 failed ttvdb.com lookups
2017-02-24 06:46:55 : Time/fetch: 1.05699263551 seconds

So looks like this version needs 20 times more elapsed for roughly the same number of pages. Will stay on 2.2 until this version is indeed faster / comparable to 2.2

After the daily run, again no xml output for canvas (like reported earlier 16/11). Only thing I can see in the logs (line 4308) is that just after it says that it has 84 programs, it removes one with same stop and start time (line 4326) and I suspect this is the first of the total list of programs, seen the time of the program, start of day this morning (could that be the bug ?). Log and xml output attached.
Documents_1.zip

In recente runs van tvgrabpyAPI zie ik dat horizon.tv dummy programma-informatie teruggeeft, bijvoorbeeld:

Added from horizon.tv:#23 Aug 06:00 - 23 Aug 07:00: Zender verstrekt geen informatie:--- Genre: Unknown.

Aangezien tvgrabpyAPI dit als valide programma-informatie ziet, wordt de informatie op deze dagen/tijdstippen niet meer verder aangevuld met informatie van sources met lagere prioriteit, zoals tvgids.tv.

Is het mogelijk om dit soort programma-informatie te negeren (bijvoorbeeld negeren van alle programma's met als titel 'Zender verstrekt geen informatie' of vergelijkbare configureerbare titels), zodat tvgrabpyAPI alsnog de gegevens van een andere source probeert op te halen?

Their original API located at
https://www.humo.be/api/epg/humosite/schedule/main/2020-02-01
seems disabled since last night.
tvgrabpyAPI --configure fails with error De url: http://www.humo.be/api/epg/humosite/channels gaf een JSON fout: No JSON object could be decoded.
The title for most of the programs is "Download onze nieuwe app!"

It seems they now have a newer one locate at:
https://www.humo.be/tv-gids/api/v2/broadcasts/2020-02-01

The last few weeks, the daily run is missing canvas output. Since I had the issue before I turned on logging and had the problem today. Issue appears to be the same, after extracting info, the last entry for canvas in the log is that a program (by the looks of it, the first in the chain) is removed because of same time, and all programs from canvas disappear. Attached log and xml output.
xmltv.zip

Recently I reinstalled the grabber on my Synology NAS. With a bit of help from Hika, I managed to get it running again, with the new 1.0.9 version.

In my setup, I get an e-mail every time a tv_grab run is completed. In the past it took upto two hours for getting all the data. Now, with the new install the first mail came after 5 days!!! What is with that?

In the mail, a lot of errors were shown, such as:
0 resterend in de npo.nl queue om te verwerken
223 resterend in de tvgids.nl queue om te verwerken
0 resterend in de primo.eu queue om te verwerken
59 resterend in de tvgids.tv queue om te verwerken
Er is een fout: "Unable to parse the HTML data. Invalid dataset!"
opgetreden bij het inlezen van een detail DataTree voor tvgids.nl
De pagina 68395091 leverde geen data op
Kan de pagina niet lezen. https://www.tvgids.nl/tv/null/68139181/: code=404: Not Found
De pagina 68139181 leverde geen data op
Er is een fout: "Unable to parse the HTML data. Invalid dataset!"
opgetreden bij het inlezen van een detail DataTree voor tvgids.nl
De pagina 68395141 leverde geen data op

Nu wordt SBS 6(xmltvid=0-36) van tvgids.tv opgehaald
(zender 10 van 16) voor dag 13 van 14.
Er is een fout: "Unable to parse the HTML data. Invalid dataset!"
opgetreden bij het inlezen van een detail DataTree voor tvgids.nl
De pagina 68395471 leverde geen data op
Er is een fout: "Unable to parse the HTML data. Invalid dataset!"
opgetreden bij het inlezen van een detail DataTree voor tvgids.nl
De pagina 68138701 leverde geen data op
Er is een fout: "Unable to parse the HTML data. Invalid dataset!"
opgetreden bij het inlezen van een detail DataTree voor tvgids.nl
De pagina 68395421 leverde geen data op

What is happening? I got the latest DataTree version (1.0.4).

My logs are full with messages like the following:

[tvgids.tv fetch] RTL 8:( 63%)  11 Jun 00:10 - 11 Jun 01:05 CEST: McLeod's Daughters
An error: "Unable to parse the HTML data. Invalid dataset!"
   occured while extracting a detail DataTree for tvgids.nl
Page 65816021 returned no data
Error creating message text! (fetch, report: 31)
An error: "Unable to parse the HTML data. Invalid dataset!"
   occured while extracting a detail DataTree for tvgids.nl
Page 66364471 returned no data
Error creating message text! (fetch, report: 31)
An error: "Unable to parse the HTML data. Invalid dataset!"
   occured while extracting a detail DataTree for tvgids.nl
Page 65814901 returned no data
Error creating message text! (fetch, report: 31)

Any idea why this is happening? I guess tvgids.nl may have changed the format of their detail pages?

I guess for now I could simply pass --disable-detail-source 3 on the command line, both to avoid these messages and speed up the fetch? Or would this have any unwanted side effects?

Tried this version as I understood elsewhere this would solve the probe but does not. Still get error message.

========
iMac:~ tonio$ python /Applications/_EyeTV/xmlgen3/tv_grab_nl3.py --config-file /Applications/_EyeTV/xmlgen3/xmlgen3.config --output /Applications/_EyeTV/XMLgen3/XMLgen3.xml --configure
Loaded the English texts file
Using config file: /Applications/_EyeTV/xmlgen3/xmlgen3.config
File: "/Applications/_EyeTV/xmlgen3/xmlgen3.config" not found or could not be accessed.
Downloading tv_grab_API.json...
Downloading tv_grab_nl.json...
The channel/source matching data is newer!
Nickelodeon and Spike changes, use the following:
0-89: Nickelodeon 24/7.
0-467: Spike 24/7.
1-nickelodeon: Combined Nickelodeon/Spike.
PRIME is now Play More.
HBO and Fox Life have closed.
Omroep Brabant is now 24/7 on Canal Digitaal, use 0-114.
Wild FM changed name to Wild Hitradio.
Creating config file: /Applications/_EyeTV/xmlgen3/xmlgen3.config
Verifying the database
Cannot open url https://web-api-pepper.horizon.tv/oesp/api/NL/nld/web/channels/ <-------
Unable to get channel info from horizon.tv
Not all channel info could be retrieved.
Try again in 15 minutes or so; or disable the failing source.
Waiting for all remaining threads to finish

Sinds kort wordt de output niet meer weergegeven in het log file. Als ik met putty onder mijn eigen naam inlog en (als test) even het helpfile weergeef (tv_grab_nl3.py --help --language nl), krijg ik bij elke output "Error creating message text! (config, help ##)" waarbij ## een oplopend nummer van 1 tot 46 is.
Als ik het als root do (sudo tv_grab_nl3.py --help --language nl) krijg ik wel correcte Nederlandse output.

Ik heb inmiddels alle files en directories volledig toegankelijk gemaakt voor alles en iedereen (777), maar dat maakt niet uit. Ik heb de verschillende directories toegevoegd aan het path, maar ook dat heeft geen resultaat.

Wie oh wie weet hoe ik de output kan herstellen?

Ik heb recent geupgrade van tvgrabnlpy naar tvgrabpyAPI. Op zich werkt dit goed, behalve voor programmagegevens die van tvgids.tv worden opgehaald. Bijvoorbeeld de film 'La Vache' op NPO2 wordt volgens de papieren TV-gids uitgezonden op 24-8 om 23:40, maar volgens de door tvgrabpyAPI gegenereerde tvguide.xml wordt deze film een dag later uitgezonden op 25-8.

De site tvgids.tv werkt met dagnummers ten opzichte van de huidige datum, dus ik vermoed dat daarmee iets misgaat. Als ik zelf op tvgids.tv kijk dan klopt de informatie wel (vandaag 15/8 wordt de film correct getoond op https://www.tvgids.tv/zenders/nederland-2/9, wat overeenkomt met 15+9=24 augustus).

Ik weet niet exact hoe laat tvgids.tv verspringt naar het volgende dagnummer dus ik heb geprobeerd de tijd aan te passen waarop tvgrabpyAPI wordt uitgevoerd (voorheen 5u 's ochtends, nu 8:30 's ochtends) maar dit maakt geen verschil.

Enig idee hoe dit opgelost kan worden?

It appears there are problems with tvgids,tv, saw many 404 not found (in terminal window, noit in the log !!) on fetches from that source and run finished with:
_An unexpected error has occured in the tvgids.tv thread:
While fetching the base pages
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/tvgrabpyAPI/tv_grab_fetch.py", line 2007, in run
self.config.infofiles.check_new_channels(self, self.config.source_channels)
File "/usr/local/lib/python2.7/dist-packages/tvgrabpyAPI/tv_grab_IO.py", line 2609, in check_new_channels
source.get_channels()
File "/usr/local/lib/python2.7/dist-packages/tvgrabpyAPI/tv_grab_fetch.py", line 2471, in get_channels
if isinstance(channel_list, list):
UnboundLocalError: local variable 'channel_list' referenced before assignment

If you want assistence, please attach your configuration and log files!
/home/pi/.xmltv/tv_grab_nl3_py.conf
/home/pi/.xmltv/tv_grab_nl3_py.log
Waiting for all remaining threads to finish_

Output appears to be only partial.

Hi,
I nhope you can help me.
Where is the xml file being saved
I have tvheadend on a raspberry pi dir: "/home/osmc/.hts/tvheadend"
and your program is on "/home/osmc/.xmltv"
if i run " tv_grab_nl3.py" program it will get the data
but how do i sow it in tveheadend.

Fetch statistics for 827 programms on 1 channels:
Start time: 2017-05-22 10:17
End time: 2017-05-22 10:36
Duration: 0:18:53.408862
255 page(s) fetched, of which 179 failed
0 cache hits
424 succesful ttvdb.com lookups
100 failed ttvdb.com lookups
Time/page: 4.44474063529 seconds

49 page(s) fetched from theTVDB.com
 2 failure(s) on theTVDB.com

 6   base page(s) fetched from tvgids.tv
 0 detail page(s) fetched from tvgids.tv
 4 failure(s) on tvgids.tv

 4   base page(s) fetched from tvgids.nl

228 detail page(s) fetched from tvgids.nl
174 failure(s) on tvgids.nl

 0   base page(s) fetched from primo.eu
 0 detail page(s) fetched from primo.eu
 0 failure(s) on primo.eu

 0   base page(s) fetched from rtl.nl
 0 failure(s) on rtl.nl

 0   base page(s) fetched from npo.nl
 0 failure(s) on npo.nl

11   base page(s) fetched from horizon.tv
 1 failure(s) on horizon.tv

 0   base page(s) fetched from humo.be
 0 failure(s) on humo.be

 4   base page(s) fetched from vpro.nl
 0 failure(s) on vpro.nl

 0   base page(s) fetched from nieuwsblad.be
 0 failure(s) on nieuwsblad.be

 0   base page(s) fetched from vrt.be
 0 failure(s) on vrt.be

 0   base page(s) fetched from oorboekje.nl
 0 failure(s) on oorboekje.nl

Waiting for all remaining threads to finish

Hi all,

When running tv_grab_nl3 at the end it hangs on:

Detail statistics for Nickelodeon / Spike (virtual) (channel 64 of 65) 0 cache hit(s) 0 excluded by genre 228 without details in cache

What can i be doing wrong?

Using latest code from git and I've just enabled the flemish tv channels and ran a /usr/bin/tv_grab_nl3.py. Next, when I run /usr/bin/tv_grab_nl3.py --use-only-cache, I get the following errors:

Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/tvgrabpyAPI/tv_grab_fetch.py", line 1985, in __init__
    self.language = self.source_data['language']
TypeError: 'NoneType' object has no attribute '__getitem__'
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/tvgrabpyAPI/tv_grab_fetch.py", line 1985, in __init__
    self.language = self.source_data['language']
TypeError: 'NoneType' object has no attribute '__getitem__'
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/tvgrabpyAPI/tv_grab_fetch.py", line 1985, in __init__
    self.language = self.source_data['language']
TypeError: 'NoneType' object has no attribute '__getitem__'
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/tvgrabpyAPI/tv_grab_fetch.py", line 1985, in __init__
    self.language = self.source_data['language']
TypeError: 'NoneType' object has no attribute '__getitem__'
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/tvgrabpyAPI/tv_grab_fetch.py", line 1985, in __init__
    self.language = self.source_data['language']
TypeError: 'NoneType' object has no attribute '__getitem__'
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/tvgrabpyAPI/tv_grab_fetch.py", line 1985, in __init__
    self.language = self.source_data['language']
TypeError: 'NoneType' object has no attribute '__getitem__'
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/tvgrabpyAPI/tv_grab_fetch.py", line 1985, in __init__
    self.language = self.source_data['language']
TypeError: 'NoneType' object has no attribute '__getitem__'
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/tvgrabpyAPI/tv_grab_fetch.py", line 1985, in __init__
    self.language = self.source_data['language']
TypeError: 'NoneType' object has no attribute '__getitem__'
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/tvgrabpyAPI/tv_grab_fetch.py", line 1985, in __init__
    self.language = self.source_data['language']
TypeError: 'NoneType' object has no attribute '__getitem__'
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/tvgrabpyAPI/tv_grab_fetch.py", line 1985, in __init__
    self.language = self.source_data['language']
TypeError: 'NoneType' object has no attribute '__getitem__'
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/tvgrabpyAPI/tv_grab_fetch.py", line 1985, in __init__
    self.language = self.source_data['language']
TypeError: 'NoneType' object has no attribute '__getitem__'
No valid source description for 1 found. Disableing it!
No valid source description for 3 found. Disableing it!
No valid source description for 4 found. Disableing it!
No valid source description for 5 found. Disableing it!
No valid source description for 6 found. Disableing it!
No valid source description for 7 found. Disableing it!
No valid source description for 8 found. Disableing it!
No valid source description for 9 found. Disableing it!
No valid source description for 10 found. Disableing it!
No valid source description for 11 found. Disableing it!
No valid source description for 12 found. Disableing it!

An unexpected error has occured:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/tvgrabpyAPI/tv_grab_config.py", line 142, in grabber_main
    x = config.validate_commandline()
  File "/usr/lib/python2.7/site-packages/tvgrabpyAPI/tv_grab_config.py", line 757, in validate_commandline
    self.write_opts_to_log()
  File "/usr/lib/python2.7/site-packages/tvgrabpyAPI/tv_grab_config.py", line 2707, in write_opts_to_log
    log_array.append(u'  prime_source = %s (%s)\n' % (src_id, self.channelsource[src_id].source))
KeyError: -1

If you want assistence, please attach your configuration and log files!
     /config/.xmltv/tv_grab_nl3_py.conf
     /config/.xmltv/tv_grab_nl3_py.log
Waiting for all remaining threads to finish
Waiting for all remaining threads to finish
Verifying the database

LOG:

2018-08-27 16:20:24 UTC: Loaded the English texts file
2018-08-27 16:20:24 UTC: Using config file: /config/.xmltv/tv_grab_nl3_py.conf
2018-08-27 16:20:24 CEST: No valid source description for 1 found. Disableing it!
2018-08-27 16:20:24 CEST: No valid source description for 3 found. Disableing it!
2018-08-27 16:20:24 CEST: No valid source description for 4 found. Disableing it!
2018-08-27 16:20:24 CEST: No valid source description for 5 found. Disableing it!
2018-08-27 16:20:24 CEST: No valid source description for 6 found. Disableing it!
2018-08-27 16:20:24 CEST: No valid source description for 7 found. Disableing it!
2018-08-27 16:20:24 CEST: No valid source description for 8 found. Disableing it!
2018-08-27 16:20:24 CEST: No valid source description for 9 found. Disableing it!
2018-08-27 16:20:24 CEST: No valid source description for 10 found. Disableing it!
2018-08-27 16:20:24 CEST: No valid source description for 11 found. Disableing it!
2018-08-27 16:20:24 CEST: No valid source description for 12 found. Disableing it!
2018-08-27 16:20:24 CEST: An unexpected error has occured:
2018-08-27 16:20:24 CEST: Traceback (most recent call last):
2018-08-27 16:20:24 CEST:   File "/usr/lib/python2.7/site-packages/tvgrabpyAPI/tv_grab_config.py", line 142, in grabber_main
2018-08-27 16:20:24 CEST:     x = config.validate_commandline()
2018-08-27 16:20:24 CEST:   File "/usr/lib/python2.7/site-packages/tvgrabpyAPI/tv_grab_config.py", line 757, in validate_commandline
2018-08-27 16:20:24 CEST:     self.write_opts_to_log()
2018-08-27 16:20:24 CEST:   File "/usr/lib/python2.7/site-packages/tvgrabpyAPI/tv_grab_config.py", line 2707, in write_opts_to_log
2018-08-27 16:20:24 CEST:     log_array.append(u'  prime_source = %s (%s)\n' % (src_id, self.channelsource[src_id].source))
2018-08-27 16:20:24 CEST: KeyError: -1
2018-08-27 16:20:24 CEST: If you want assistence, please attach your configuration and log files!
2018-08-27 16:20:24 CEST:      /config/.xmltv/tv_grab_nl3_py.conf
2018-08-27 16:20:24 CEST:      /config/.xmltv/tv_grab_nl3_py.log
2018-08-27 16:20:24 CEST: Waiting for all remaining threads to finish
2018-08-27 16:20:25 CEST: Waiting for all remaining threads to finish
2018-08-27 16:20:25 CEST: Verifying the database

Beide config files zijn identiek, idem de CLI.

1. Toch is er een (ontzettend groot) verschil tussen de hoeveelheid informatie. Waar v2.2 voor de volle 14 dagen alle informatie laat zien is dit bij v3 voor vrijwel alle zenders veel beperkter.

2. Ook opvallend is dat v2.2 aan 1 stuk door doorloppt terwijl v3 (altijd na Veronica/Disney XD vele minuten stilstaat.

Volledigheidshalve heb ik beide logfiles alsmede .config (beide zijn identiek) bijgesloten.

log v2.2.txt
log v3.txt
Config.txt

I understand there are some issues with the TVgids.nl and therefor the duration of a complete EPG-data-grabbing run.

Since I had some other issues on my Synology NAS (DS415Play), I was monitoring the CPI and memory usage by the NAS. Seems that tv_grab_nl3.py is using a growing amount of memory, during its run. This is of course not a problem on a PC, but on a NAS, with limited CPU and memory resources it slows down all not only other processes, but itself as well.

Of course I do not know exactly how tv_grab works, but it seems it builds up a data file with the EPG data. Once it is finished with all the data gathering, it writes the result to the designated directory / file. If this is the case, there are 2 issues to be considered:

1. If a (fatal) error occurs during the grabbing process, the gathered data and used time is lost.
2. The memory usage keeps growing.

Would it be possible to write the results after each step (after each source, retrieved day and/or channel to the database / XML file and releasing the memory after successful doing so?

Hi

Have a question after setting up the config and get the TV guide info how do i show it op on tvheadend?

I noticed that on Mac OS X (or any posix system), the default locations are in setup.py:

source_dir = u'/var/lib/tvgrabpyAPI'
opt_dict['etc_dir'] = u'/etc/tvgrabpyAPI'

while the default location in tv_grab_config.py are:

opt_dict['etc_dir'] = u'/etc/tvgrabpyAPI'
opt_dict['xmltv_dir'] = opt_dict['home_dir'] + u'/.xmltv/sources'

Three (related) comments:

1. The source dir is not the same in setup.py and tv_grab_config.py.

2. The correct default on Mac OS X should be respectively opt_dict['home_dir'] + u'/Library/Preferences/com.github.tvgrabbers/sources' and opt_dict['home_dir'] + u'/Library/Caches/com.github.tvgrabbers/sources', or possibly all in opt_dict['home_dir'] + u'/Library/Application Support/xmltv/'

3. source_dir in setup.py should possibly be dependant on the package manager. For example, the install location for tvgrabpyAPI for me is
   /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/tvgrabpyAPI/. The most logical install location is /opt/local/var/lib/tvgrabpyAPI.

It would be nice if the tool would also grab images for programmes. A lot of sources include images for programmes. For example horizon.tv, tvgids.nl, tvgids.tv and npo.nl.

According to the XMLTV format, this image should be saved in an icon tag (https://github.com/XMLTV/xmltv/blob/master/xmltv.dtd#L216) and should contain a src attribute (https://github.com/XMLTV/xmltv/blob/master/xmltv.dtd#L336).

I'm not sure how many PVR / DVR / EPG software supports displaying images. I do now for sure that Plex DVR, TVHeadend and Kodi support it.

My daily tv program update this morning was looping, which happens from time to time, but even after repeated manual starts it kept failing, I suspect the culprit is this:
`Loaded the English texts file
Using config file: /home/geert/.xmltv/tv_grab_nl3_py.conf
Downloading tv_grab_API.json...
Downloading tv_grab_nl.json...
Exception in thread source-primo.eu:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
self.run()
File "/usr/local/lib/python2.7/dist-packages/tvgrabpyAPI/tv_grab_fetch.py", line 2037, in run
detail_idx = self.config.detail_sources.index(self.proc_id)
ValueError: 9 is not in list

Verifying the database
`
Suspect that problem originates from the downloaded files outside my control. Please advise which documentation you need.

It appears than since a few days (suspect 3) I don't get output for canvas, I see plenty of entries in the logs for programs from canvas, but no output in the xml output. Cant find any other error messages either, they appear to sink in a black hole. See logs and xml output attached.
tvgrab.zip

Hi there,

I am getting an error when running the configure command:
Cannot open url https://web-api-pepper.horizon.tv/oesp/api/NL/nld/web/channels/

kind regards
tony

full log:
root@tvh:/tmp/tvgrabpyAPI # python tv_grab_nl3.py --configure
Loaded the English texts file
Using config file: /etc/tvgrabpyAPI/tv_grab_nl3_py.conf
File: "/etc/tvgrabpyAPI/tv_grab_nl3_py.conf" not found or could not be accessed.
Downloading tv_grab_API.json...
Downloading tv_grab_nl.json...
The channel/source matching data is newer!
Nickelodeon and Spike changes, use the following:
0-89: Nickelodeon 24/7.
0-467: Spike 24/7.
1-nickelodeon: Combined Nickelodeon/Spike.
PRIME is now Play More.
HBO and Fox Life have closed.
Omroep Brabant is now 24/7 on Canal Digitaal, use 0-114.
While we are working on a renewed source-file for the new npo.nl site
this source is disabled! Further news follows in the comming days.
Wild FM changed name to Wild Hitradio.
The npo.nl source has been reactivated, but as the channelids
have all changed you need to run --configure to use it.
It now gives basic listings for all the NPO channels for up to 2 weeks.
Unfortunatly only start time and titel and no longer for the regional channels.
Creating config file: /etc/tvgrabpyAPI/tv_grab_nl3_py.conf
Verifying the database
Cannot open url https://web-api-pepper.horizon.tv/oesp/api/NL/nld/web/channels/
Unable to get channel info from horizon.tv
Not all channel info could be retrieved.
Try again in 15 minutes or so; or disable the failing source.
Waiting for all remaining threads to finish

Hi,

Ik heb nooit eerder gebruik gemaakt van tv_grab_nl dus dacht bij het in gebruik nemen ervan maar meteen versie 3 te pakken.
Ik heb me wat ingelezen en wat test runs gedaan in een VM en dat leek allemaal naar wens te verlopen en nu wilde ik het naar mijn Synology brengen wat ook geen probleem hoeft te zijn.
Waar ik alleen tegen aanloop is hoe de data nu in TVHeadend ingelezen te krijgen?

Ik had al gelezen om voor TVH de use-only-cache optie te gebruiken en de normale run zonder.
En daar zit hem nu net het probleem. Voor zover ik kan zien heb ik in TVH geen invloed in hoe een extern script gerunt wordt, ik kan alleen maar een sock file uitlezen en dat is meteen het tweede probleem, hoe krijg ik de data naar de TVH sock ?

On Windows: installed a fresh copy of Python 2.7.13; pytz, requests via pip and DataTreeGrab and tvgrabpyAPI using setup.py install. Ran tv_grab_nl3.py --configure. Output suggests that config file has been created in /Users/[me]/.xmltv/tv_grab_nl3_py.conf but it is not there...

Tried it on a Raspberry Pi with Python 2.7.9, same thing happens. No config file is created (not in /home, /etc or ~/.xmltv). Tried it using both root and normal account to no avail.

I don't understand what's going wrong. Can somebody maybe paste a .conf file so I can create it manually?

I'm running tv_grab_nl3.py in a cron job. stdout is piped via a socket to tvheadend, but every time the job runs I get an email from cron with text like

Loaded the English texts file
Using config file: /home/janpascal/.xmltv/tv_grab_nl3_py.conf
Downloading tv_grab_API.json...
Downloading tv_grab_nl.json...

It looks like this is the stderr output from tv_grab_nl3.py, which it outputs even though the --quiet flag is used. I think this output should be suppressed with the --quiet flag.

I just got this error with Python 2.7.13:

Traceback (most recent call last):
  File "/Users/freek/Repository/tvgrabpyAPI/tv_grab_nl3.py", line 123, in <module>
    locale.setlocale(locale.LC_ALL, '')
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/locale.py", line 581, in setlocale
    return _setlocale(category, locale)
locale.Error: unsupported locale setting

My recommendation is to change:

locale.setlocale(locale.LC_ALL, '')

into:

try:
    locale.setlocale(locale.LC_ALL, 'C.UTF-8')
except locale.Error:
    locale.setlocale(locale.LC_ALL, 'C')

Since a couple of weeks the episode numbers for NPO channels don't work anymore (they are missing). I'm not sure what changed and I can't figure out what the problem is.
All other channels (that I've checked) have no problems.

Hi,

I have since several day the following, recurring, crash:

`Now merging 82 programs from horizon.tv into 220 programs from Euronews
(channel 17 of 61)

An unexpected error has occured in the Euronews (0-422) thread:
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/tvgrabpyAPI/tv_grab_channel.py", line 216, in run
self.channel_node.merge_source(programs, index)
File "/usr/local/lib/python2.7/dist-packages/tvgrabpyAPI/tv_grab_channel.py", line 1140, in merge_source
check_gaps(group_slots[index], True)
File "/usr/local/lib/python2.7/dist-packages/tvgrabpyAPI/tv_grab_channel.py", line 979, in check_gaps
add_to_list(add_to_end, pp, is_groupslot)
File "/usr/local/lib/python2.7/dist-packages/tvgrabpyAPI/tv_grab_channel.py", line 957, in add_to_list
pn.scandate = programs[index]['scandate']
IndexError: list index out of range

If you want assistence, please attach your configuration and log files!
/var/lib/mythtv/.xmltv/tv_grab_nl3_py.conf
/var/lib/mythtv/.xmltv/tv_grab_nl3_py.log
Waiting for all remaining threads to finish
`

The config for euronews is:
Euronews;14;0-422;;euronews;;422;;24443943027;;;euronews;euronews;;;;4;euronews_1.png

What I tried to fix it:

• update to the latest API
• clear cache
  But still the crash occurs. If you need some more info or something else, please let me know.

Just upgraded to the latest release and now have this error.
I tried looking for a class with a similar name in the source (assuming a typo) but could't find it.

Traceback (most recent call last):
File "/volume2/Software/xmltv/tvgrabpy/tv_grab_nl3.py", line 108, in
import sys, locale, tvgrabpyAPI
File "/volume2/Software/xmltv/tvgrabpy/tvgrabpyAPI/init.py", line 8, in
from tv_grab_config import *
File "/volume2/Software/xmltv/tvgrabpy/tvgrabpyAPI/tv_grab_config.py", line 111, in
import tv_grab_IO, tv_grab_fetch, tv_grab_channel, pytz
File "/volume2/Software/xmltv/tvgrabpy/tvgrabpyAPI/tv_grab_IO.py", line 2865, in
class DD_Convert(DataDef_Convert):
NameError: name 'DataDef_Convert' is not defined

After daily run (twice) no output at all for any station in xml output after midnight tonight, despite that log indicates data was retrieved. This issue looks similar to earlier reported problem for canvas, but now ALL stations are affected. Enclosed log and xml output.

The file is not there

pi@raspberrypi:/home/tvgrabpyAPI-master $ sudo python ./tv_grab_nl3.py --configure
Loaded the English texts file
Using config file: /etc/tvgrabpyAPI/tv_grab_nl3_py.conf
File: "/etc/tvgrabpyAPI/tv_grab_nl3_py.conf" not found or could not be accessed.
Downloading tv_grab_API.json...
Downloading tv_grab_nl.json...
The channel/source matching data is newer!
Nickelodeon and Spike changes, use the following:
0-89: Nickelodeon 24/7.
0-467: Spike 24/7.
1-nickelodeon: Combined Nickelodeon/Spike.
PRIME is now Play More.
HBO and Fox Life have closed.
Omroep Brabant is now 24/7 on Canal Digitaal, use 0-114.
Wild FM changed name to Wild Hitradio.
Creating config file: /etc/tvgrabpyAPI/tv_grab_nl3_py.conf
Verifying the database
Error reading the base-channels-page: http://www.primo.eu/Tv%20programma's%20in%20volledig%20scherm%20bekijken
Error reading the base-channels-page: http://www.primo.eu/Tv%20programma's%20in%20volledig%20scherm%20bekijken
Unable to get channel info from primo.eu
Not all channel info could be retrieved.
Try again in 15 minutes or so; or disable the failing source.
Waiting for all remaining threads to finish

Sometimes never stops running when run from either mythfilldatabase or update_epg.sh.
And if it ever finishes it only fills current day for Veronica/Disney-XD.
For a full-run with mythfilldatabase and sometimes update_epg.sh I have had it running for several days without stopping.

Noticed that lately there are a lot of programs where series and episode is missing for example following program:

<title lang="nl">Floortje terug naar het einde van de wereld</title>
In 2014 bezocht Floortje de Engelse dierarts Joe Hollins op het prachtige en extreem geïsoleerde eiland Sint-Helena. Het eiland is inmiddels een stuk minder geïsoleerd.
Overige

had episode and series in the past and when I look at tvgids.nl, there is series and episodes to be found. This has been happening the last few weeks, but wasnt able to pinpoint it till now. Let me know if you need debug info.

Hi,

Horizon.tv provides some weird episode numbers for some programmes.

When running tv_grab_nl3.py without --disable-source 5 (5 = horizon.tv), the episode number of, for example, the NOS Journaal looks like this: <episode-num system="xmltv_ns"> . 821185866 . </episode-num>.
When running tv_grab_nl3.py with --disable-source 5 (5 = horizon.tv), the episode number of, for example, the NOS Journaal looks like this: <episode-num system="xmltv_ns"> . 82 . </episode-num>.

I can see in the horizon JSON response that this episode number is stated there. So the data is retrieved correctly. So I don't think there is any way to fix that.
Is it possible in any way to disable the episode numbers from horizon for specific programmes? Or to prefer certain sources for episode numbering?

Hello,
I'm wonder if it ready for user? What sites are already supported?

If I call tv_grab_nl3.py --add-ttvdb-title "goede tijden, slechte tijden" it takes for ever and when I brake the process it is still busy matching nodes.
Also during normal grabbing it is hanging on the ttvdb module.
Possibly a memory limit?
I for now will block that ttvdbID

Allereerst hartelijk dank voor deze tool, ik gebruik het al jaren en ben recent overgestapt op de nieuwe versie. Na enig configuratiewerk heb ik het inmiddels tot volle tevredenheid draaien. Top!

Vandaag viel me echter iets vreemds op.
Volgende week begint een nieuw seizoen van Prison Break op FOX die ik alvast in wilde stellen om op te nemen en tot mijn verbazing kon ik die niet vinden in de gids.

Wat blijkt... Op de 1 of andere manier wordt de serie nu onder de naam "Sequel" geplaatst.
De serie heet inderdaad Prison Break: Sequel maar ik denk dat de grabber nu onterecht de originele titel er ergens afsplitst.

Ik heb even gecheckt in de xmltv file en daar staat nu inderdaad de volgende vermelding:

<programme  start="20170405210000 +0200" stop="20170405220000 +0200" channel="0-440">
    <title  lang="nl">Sequel</title>
    <sub-title  lang="nl">Ogygia</sub-title>
    <desc  lang="nl">Dramaserie. Prison Break is terug en klaar voor zijn grootste ontsnapping ooit. De doodgewaande Michael blijkt nog in leven te zijn en zijn broer is klaar om hem te bevrijden. | Kijkwijzer: Let op met kinderen tot 12 jaar, programma bevat geweld, angst en grof taalgebruik.</desc>
 

Wat zou ik aan moeten passen om dit op te lossen ?
Mogelijk dat dit probleem ook bij andere series optreed, het valt me alleen nu bij deze ineens op.

De serie wordt a.s woensdag 5 april om 21:00 uitgezonden trouwens.

Alvast bedankt voor je hulp.

Ik heb het geheel werkend en in een .command file gezet. Als ik die file start of door een double click of door Terminal te openen en hem daarin te starten dan werkt alles prima.

Echter als ik de .command file in Launchitems zet dan komt er bij de start de melding "ImportError: No module named DataTreeGrab" m.a.w. hij vind DTG niet.

Enig idee waar ik moet zoeken?

My NAS updated its OS and (as usual) this gave problems with the grabber. Usually, I just installed it again and everything worked fine. But now I get errors...
I installed the latest version of the grabber (1.0.9), but as soon as I run "tv_grab_nl3.py --configure --language nl", I get errors: Error creating message text, error loading language file, error creating message text and that several times in a row.

What is going wrong?

Hi,

I've noticed that the tvgids.nl source includes HTML tags in the description output for programmes.

When running tv_grab_nl3.py without --disable-source 3 (3 = tvgids.nl), the output of for example the NOS Journaal looks like this:

  <programme  start="20190425000500 +0200" stop="20190425002500 +0200" channel="0-1">
    <title  lang="nl">NOS Journaal</title>
    <desc  lang="nl">&lt;html&gt;&lt;p&gt;Met het laatste nieuws, gebeurtenissen van nationaal en internationaal belang en de weersverwachting voor vandaag.&lt;/p&gt;&lt;p&gt;&lt;ul&gt;&lt;li&gt; 3,5 miljard voor nieuwe stations &lt;/li&gt;&lt;li&gt; Dodental Sri Lanka naar gestegen &lt;/li&gt;&lt;li&gt; Vier doden bij ongeluk op de A12 &lt;/li&gt;&lt;li&gt; Bodycam politie remt agressie &lt;/li&gt;&lt;li&gt; Angst voor meer schade Notre-Dame &lt;/li&gt;&lt;li&gt; Mildere straffen protesten Hongkong &lt;/li&gt;&lt;li&gt; Explosie in hotel Zaandijk &lt;/li&gt;&lt;li&gt; Hoofdkantoor Gasterra Groningen...</desc>
    <category>News</category>
  </programme>

Please notice the HTML tags included in the description.

When running tv_grab_nl3.py with --disable-source 3 (3 = tvgids.nl), the output of for example the NOS Journaal looks like this:

  <programme  start="20190425015500 +0200" stop="20190425021000 +0200" channel="0-1">
    <title  lang="nl">NOS Journaal</title>
    <desc  lang="nl">Met het laatste nieuws, gebeurtenissen van nationaal en internationaal belang en de weersverwachting voor vandaag.</desc>
    <date>2017</date>
    <category>News</category>
  </programme>

Please notice that the description looks different (without the HTML tags) because it's coming from a different source.

I suspect that the source file for tvgids.nl needs to be modified. I'm new to DataTreeGrab, so I don't quite understand what should be changed.

The tvgids.nl page for this programme is: https://www.tvgids.nl/tv/journaal/63510031.

I've tested this with the last stable release, as well as with beta-1.0.9

Ik heb alles van oude naar nieuwe iMac overgezet. Ogenschijnlijk werkt alles maar de .XML file staat op lengte 0.
Enige foutmelding die ik ontdek is "Cannot open url https://web-api-pepper.horizon.tv/oesp/api/NL/nld/web/listings/" en ook wordt er geen mail meer verzonden na afloop.

Is er iets wat ik handmatig moet doen omdat het nu op een nieuw systeem draait?

Since the last 2 days, all (by the looks of it) reference to tvgids.tv return following error:

Error reading the base-page: http://www.tvgids.tv/zenders/nederland-3/0/
Error reading the base-page: http://www.tvgids.tv/zenders/nederland-3/1/
Error reading the base-page: http://www.tvgids.tv/zenders/nederland-3/2/
Error reading the base-page: http://www.tvgids.tv/zenders/nederland-2/0/
Error reading the base-page: http://www.tvgids.tv/zenders/nederland-2/1/
Error reading the base-page: http://www.tvgids.tv/zenders/nederland-2/2/
Error reading the base-page: http://www.tvgids.tv/zenders/nederland-1/0/
Error reading the base-page: http://www.tvgids.tv/zenders/nederland-1/1/
Error reading the base-page: http://www.tvgids.tv/zenders/nederland-1/2/

Have they changed their website ??

The last couple of days I've noticed that tv_grab_nl3.py keeps running using 100% cpu. After running it manually I noticed the following error being thrown:

An unexpected error has occured in the horizon.tv thread:
While fetching the base pages
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/tvgrabpyAPI/tv_grab_fetch.py", line 1996, in run
self.init_channel_source_ids()
File "/usr/lib/python2.7/site-packages/tvgrabpyAPI/tv_grab_fetch.py", line 2143, in init_channel_source_ids
self.source_data[ptype]['url'] = self.source_data[ptype]['alt-url']
KeyError: u'channel'

I've attached the request conf an log files
tv_grab_nl3.tar.gz

Ik merk dat (m.n. NPO 1) er soms verkeerde programmagegevens (tijden, titel) worden opgehaald. De v2.2 haalt in die gevallen wel de goede op. Blijkbaar gebruiken beide verschillende bronnen.

De oude/v2.2 gebruikt m.i. de juiste/beste bron.

Here's an idea:

When no season number is available for a programme (either from sources or TheTVDB), the season is assumed to be 1. (No season number is specified, so many PVR's interpret this as season 1; as per XMLTV specification; https://github.com/XMLTV/xmltv/blob/master/xmltv.dtd#L377).

    <episode-num system="xmltv_ns"> . 83 . </episode-num>

When recording this provides a problem where episodes from different years (mainly daily shows) are interpreted as the same episode. Automatic recording systems might therefore not record these new episodes and episodes from different years get put in the same directory (depending on the recording configuration of course).

For most of these daily shows the production year is know. A configurable option could be added that uses the production year as season number. For example for 2019, this would look like this:

    <episode-num system="xmltv_ns">2018 . 83 . </episode-num>

(Because of the 0-indexed numbers)

Maybe we could even use the genre to determine if this behavior should be enabled (for example News).

If anyone has additional ideas that I missed, please mention them!