freearhey / epg-grabber Goto Github PK

Hello,

It will be really use full to add the optional LCN property in channel tag record:
LCN: Logical Channel Number
https://en.wikipedia.org/wiki/Virtual_channel
It will help to sort the channels by this number in applications.
Most of users prefer to see the channels list by channel number.
(I know that for some countries this number can be different by state, but in Europe it's fix by country)
Thanks

Hi

When I'm grabbing telenet.tv the grabbing suddenly stops without telling why.... I do first get a cookie error, and 'usually' it doesn't take long to crash afterwards, but I can't say that cookie error is the problem, as sometimes it also continues after this error (see below log)
I started running the program with a log file and a log-level=debug but this didn't add anything extra (I also tried level=trace but then the program didn't want to start, so I don't thing trace exists...)

[2023-01-15T16:27:48.441Z] INFO: Starting...
[2023-01-15T16:27:48.442Z] INFO: Loading 'sites/telenet.tv/telenet.tv.config.js'...
[2023-01-15T16:27:48.444Z] INFO: Loading 'sites/telenet.tv/telenet.tv.channels.xml'...
[2023-01-15T16:27:56.563Z] INFO: [1/728] telenet.tv - 2MInternational.ma - Jan 15, 2023 (42 programs)
[2023-01-15T16:28:04.045Z] INFO: [2/728] telenet.tv - 2MInternational.ma - Jan 16, 2023 (49 programs)
[2023-01-15T16:28:12.418Z] INFO: [3/728] telenet.tv - 2MInternational.ma - Jan 17, 2023 (49 programs)
[2023-01-15T16:28:20.882Z] INFO: [4/728] telenet.tv - 2MInternational.ma - Jan 18, 2023 (50 programs)
[2023-01-15T16:28:27.958Z] INFO: [5/728] telenet.tv - 2MInternational.ma - Jan 19, 2023 (49 programs)
[2023-01-15T16:28:35.410Z] INFO: [6/728] telenet.tv - 2MInternational.ma - Jan 20, 2023 (48 programs)
[2023-01-15T16:28:42.760Z] INFO: [7/728] telenet.tv - 2MInternational.ma - Jan 21, 2023 (43 programs)
[2023-01-15T16:28:49.568Z] INFO: [8/728] telenet.tv - AB3.be - Jan 15, 2023 (28 programs)
[2023-01-15T16:28:52.755Z] INFO: [9/728] telenet.tv - AB3.be - Jan 16, 2023 (0 programs)
[2023-01-15T16:28:52.755Z] ERROR: Cookie not in this host's domain. Cookie:horizon.tv Request:static.spark.telenet.tv
[2023-01-15T16:28:58.985Z] INFO: [10/728] telenet.tv - AB3.be - Jan 17, 2023 (30 programs)
[2023-01-15T16:29:05.311Z] INFO: [11/728] telenet.tv - AB3.be - Jan 18, 2023 (31 programs)
[2023-01-15T16:29:12.223Z] INFO: [12/728] telenet.tv - AB3.be - Jan 19, 2023 (25 programs)
[2023-01-15T16:29:17.878Z] INFO: [13/728] telenet.tv - AB3.be - Jan 20, 2023 (32 programs)
[2023-01-15T16:29:23.996Z] INFO: [14/728] telenet.tv - AB3.be - Jan 21, 2023 (33 programs)
[2023-01-15T16:29:30.119Z] INFO: [15/728] telenet.tv - AlJazeeraEnglish.qa - Jan 15, 2023 (40 programs)
[2023-01-15T16:29:36.185Z] INFO: [16/728] telenet.tv - AlJazeeraEnglish.qa - Jan 16, 2023 (40 programs)
[2023-01-15T16:29:43.375Z] INFO: [17/728] telenet.tv - AlJazeeraEnglish.qa - Jan 17, 2023 (39 programs)
[2023-01-15T16:29:50.423Z] INFO: [18/728] telenet.tv - AlJazeeraEnglish.qa - Jan 18, 2023 (39 programs)
[2023-01-15T16:29:56.486Z] INFO: [19/728] telenet.tv - AlJazeeraEnglish.qa - Jan 19, 2023 (39 programs)
[2023-01-15T16:30:02.673Z] INFO: [20/728] telenet.tv - AlJazeeraEnglish.qa - Jan 20, 2023 (40 programs)
[2023-01-15T16:30:08.818Z] INFO: [21/728] telenet.tv - AlJazeeraEnglish.qa - Jan 21, 2023 (40 programs)
[2023-01-15T16:30:14.678Z] INFO: [22/728] telenet.tv - AnimalPlanetNetherlands.nl - Jan 15, 2023 (33 programs)
[2023-01-15T16:30:20.894Z] INFO: [23/728] telenet.tv - AnimalPlanetNetherlands.nl - Jan 16, 2023 (30 programs)
[2023-01-15T16:30:26.922Z] INFO: [24/728] telenet.tv - AnimalPlanetNetherlands.nl - Jan 17, 2023 (31 programs)
[2023-01-15T16:30:32.596Z] INFO: [25/728] telenet.tv - AnimalPlanetNetherlands.nl - Jan 18, 2023 (30 programs)
[2023-01-15T16:30:38.746Z] INFO: [26/728] telenet.tv - AnimalPlanetNetherlands.nl - Jan 19, 2023 (30 programs)
[2023-01-15T16:30:45.520Z] INFO: [27/728] telenet.tv - AnimalPlanetNetherlands.nl - Jan 20, 2023 (30 programs)
[2023-01-15T16:30:51.570Z] INFO: [28/728] telenet.tv - AnimalPlanetNetherlands.nl - Jan 21, 2023 (31 programs)
[2023-01-15T16:31:21.549Z] INFO: [29/728] telenet.tv - BabyTV.uk - Jan 15, 2023 (354 programs)
[2023-01-15T16:31:51.689Z] INFO: [30/728] telenet.tv - BabyTV.uk - Jan 16, 2023 (356 programs)
[2023-01-15T16:32:19.698Z] INFO: [31/728] telenet.tv - BabyTV.uk - Jan 17, 2023 (357 programs)
[2023-01-15T16:32:49.801Z] INFO: [32/728] telenet.tv - BabyTV.uk - Jan 18, 2023 (355 programs)
[2023-01-15T16:33:23.059Z] INFO: [33/728] telenet.tv - BabyTV.uk - Jan 19, 2023 (354 programs)
[2023-01-15T16:33:54.300Z] INFO: [34/728] telenet.tv - BabyTV.uk - Jan 20, 2023 (355 programs)
[2023-01-15T16:34:25.709Z] INFO: [35/728] telenet.tv - BabyTV.uk - Jan 21, 2023 (350 programs)
[2023-01-15T16:34:34.186Z] INFO: [36/728] telenet.tv - BBCEntertainment.uk - Jan 15, 2023 (39 programs)
[2023-01-15T16:34:39.861Z] INFO: [37/728] telenet.tv - BBCEntertainment.uk - Jan 16, 2023 (37 programs)
[2023-01-15T16:34:46.278Z] INFO: [38/728] telenet.tv - BBCEntertainment.uk - Jan 17, 2023 (38 programs)
[2023-01-15T16:34:52.699Z] INFO: [39/728] telenet.tv - BBCEntertainment.uk - Jan 18, 2023 (38 programs)
[2023-01-15T16:34:58.746Z] INFO: [40/728] telenet.tv - BBCEntertainment.uk - Jan 19, 2023 (36 programs)
[2023-01-15T16:35:06.498Z] INFO: [41/728] telenet.tv - BBCEntertainment.uk - Jan 20, 2023 (39 programs)
[2023-01-15T16:35:12.812Z] INFO: [42/728] telenet.tv - BBCEntertainment.uk - Jan 21, 2023 (40 programs)
[2023-01-15T16:35:18.002Z] INFO: [43/728] telenet.tv - BBCFirstBenelux.uk - Jan 15, 2023 (30 programs)
[2023-01-15T16:35:23.281Z] INFO: [44/728] telenet.tv - BBCFirstBenelux.uk - Jan 16, 2023 (27 programs)
[2023-01-15T16:35:28.866Z] INFO: [45/728] telenet.tv - BBCFirstBenelux.uk - Jan 17, 2023 (29 programs)
[2023-01-15T16:35:33.882Z] INFO: [46/728] telenet.tv - BBCFirstBenelux.uk - Jan 18, 2023 (31 programs)
[2023-01-15T16:35:54.132Z] INFO: [47/728] telenet.tv - BBCFirstBenelux.uk - Jan 19, 2023 (31 programs)
[2023-01-15T16:35:59.964Z] INFO: [48/728] telenet.tv - BBCFirstBenelux.uk - Jan 20, 2023 (29 programs)
[2023-01-15T16:36:04.952Z] INFO: [49/728] telenet.tv - BBCFirstBenelux.uk - Jan 21, 2023 (30 programs)
[2023-01-15T16:36:11.026Z] INFO: [50/728] telenet.tv - BBCOneEngland.uk - Jan 15, 2023 (37 programs)
[2023-01-15T16:36:17.307Z] INFO: [51/728] telenet.tv - BBCOneEngland.uk - Jan 16, 2023 (38 programs)
[2023-01-15T16:36:24.435Z] INFO: [52/728] telenet.tv - BBCOneEngland.uk - Jan 17, 2023 (36 programs)
[2023-01-15T16:36:30.818Z] INFO: [53/728] telenet.tv - BBCOneEngland.uk - Jan 18, 2023 (37 programs)
[2023-01-15T16:36:34.076Z] INFO: [54/728] telenet.tv - BBCOneEngland.uk - Jan 19, 2023 (0 programs)
[2023-01-15T16:36:34.076Z] ERROR: Cookie not in this host's domain. Cookie:horizon.tv Request:static.spark.telenet.tv
[2023-01-15T16:36:40.772Z] INFO: [55/728] telenet.tv - BBCOneEngland.uk - Jan 20, 2023 (40 programs)
[2023-01-15T16:36:46.742Z] INFO: [56/728] telenet.tv - BBCOneEngland.uk - Jan 21, 2023 (36 programs)
[2023-01-15T16:36:51.456Z] INFO: [57/728] telenet.tv - BBCTwoEngland.uk - Jan 15, 2023 (19 programs)
[2023-01-15T16:36:56.235Z] INFO: [58/728] telenet.tv - BBCTwoEngland.uk - Jan 16, 2023 (27 programs)
[2023-01-15T16:37:01.424Z] INFO: [59/728] telenet.tv - BBCTwoEngland.uk - Jan 17, 2023 (23 programs)
[2023-01-15T16:37:06.097Z] INFO: [60/728] telenet.tv - BBCTwoEngland.uk - Jan 18, 2023 (25 programs)

In this library the stop attribute is required (Or at least the README claims it is) but the XMLTV format doesn't require it.

Hey man, your work is incredible, but I have a doubt, is it possible or will it be possible in the future to add the program icons? I'm using it on Android TV and to be perfect, the program icons are just missing.

Hi, I don't know if this should go here, but I'm trying to create a config.js file for a new EPG site and it's grabbing the content fine, but the data is JSON and for some dates is a big file. I'm having this issue:

  [2/6] example.com (es) - channelexample.us - Oct 20, 2023 (0 programs)
    ERR: maxContentLength size of 5242880 exceeded

Is there a way to increase that "maxContentLength" to avoid the error? Thank you.

Is it possible to fill in the empty tv guide with the channel name as default

Hello, I love the program and what it is capable of doing.
I was looking for a way to combine multiple outputs to create one EPG around a playlist.

Is this something that's possible with the latest version?
If not available at the moment, Is that something that could be added with later releases?

I use a script from here iptv-org which only takes data from page one, so how do I get the data from the second page and so on?

Hi Guys

Great software! Small remark:

Usage: epg-grabber [options]
Node.js CLI tool for grabbing EPG from different sites
Options:
-v, --version output the version number
-c, --config Path to [site].config.js file
-o, --output Path to output file
--channels Path to channels.xml file
--lang Set default language for all programs
--days Number of days for which to grab the program
--delay Delay between requests (in mileseconds)
--timeout Set a timeout for each request (in mileseconds)
--cache-ttl Maximum time for storing each request (in milliseconds)
--gzip Compress the output (default: false)
--debug Enable debug mode (default: false)
--curl Display request as CURL (default: false)
--log Path to log file
--log-level Set log level (default: "info")
-h, --help display help for command

Hey freearhey

Hope you are well....

I was investigating why directv.com is no longer working... and found something strange
After cutting the requests down and asking to see them in curl
I tried to manually execute a single request in curl and got the follow response:

which was actually quite strange because in different browsers from different locations I was getting the data fine
so I added the -A flag to curl to specify my agent, and it was suddenly working perfect:

So I was wondering where we can specify an agent in the config.js file? After reading thru it I'm assuming we can't...
So this would have to be added to epg-grabber itself, as a flag, or an option in the config.js file

Many Thanks!

Ben

Hello, first of all, thank you for your effort in the requested file. I am very thankful.

However, I am trying to use it with these codes:

`const dayjs = require('dayjs')
const timezone = require('dayjs/plugin/timezone')
const customParseFormat = require('dayjs/plugin/customParseFormat')

dayjs.extend(timezone)
dayjs.extend(customParseFormat)

module.exports = {
site: 'turksatkablo.com.tr',
output: 'ide.xml', // path to output file (default: 'guide.xml')
channels: 'turksatkablo.com.tr_tr.channels.xml', // path to channels.xml file (required)
lang: 'tr', // default language for all programs (default: 'en')
days: 3, // number of days for which to grab the program (default: 1)
delay: 3000, // delay between requests (default: 3000)

url: function ({ date }) {
return https://www.turksatkablo.com.tr/userUpload/EPG/y.json?_=${date.valueOf()}
},
parser: function ({ content, channel, date }) {
let PM = false
let programs = []
const items = parseItems(content, channel)
items.forEach(item => {
let start = parseStart(item, date)
if (start.hour() > 11) PM = true
if (start.hour() < 12 && PM) start = start.add(1, 'd')
let stop = parseStop(item, date)
if (stop.hour() > 11) PM = true
if (stop.hour() < 12 && PM) stop = stop.add(1, 'd')

  programs.push({
    title: item.b,
    start: start.toString(),
    stop: stop.toString()
  })
})

return programs

}
}

function parseStart(item, date) {
const time = ${date.format('MM/DD/YYYY')} ${item.c}

return dayjs.tz(time, 'MM/DD/YYYY HH:mm', 'Europe/Istanbul')
}

function parseStop(item, date) {
const time = ${date.format('MM/DD/YYYY')} ${item.d}

return dayjs.tz(time, 'MM/DD/YYYY HH:mm', 'Europe/Istanbul')
}

function parseItems(content, channel) {
const parsed = JSON.parse(content)
const channels = parsed.k
if (!channels) return []
const data = channels.find(c => c.x == channel.site_id)

return data ? data.p : []
}
`

When I use epg-grabber --config=turksatkablo.com.tr2.config.js I get that error:
`pi@rpi:~/EPG $ epg-grabber --config=turksatkablo.com.tr2.config.js

Starting...
Loading 'turksatkablo.com.tr2.config.js'...
Loading 'turksatkablo.com.tr_tr.channels.xml'...
turksatkablo.com.tr - TR:24Kitchen - Oct 27, 2021 (0 programs)
Error: o.utc is not a function`

How can I fix it? (I have installed "dayjs" with sudo npm install dayjs)

Thanks in advance.

Is it possible to do asynchronous functions? for example, make an each, search for a url, wait to make a request on that url and scratch data and return with the function, is it possible to do this?

getting the below error on several channels grabbing osn.com epg from the iptv-org osn.com
anything specific I need to do?

......
[11/196] osn.com - AlMamlakaTV.jo - Nov 21, 2021 (0 programs)
Error: Unexpected token < in JSON at position 0
.......

Hello

I played white some epg grabbers for different devices.
There is noticed some of the grabbers allow to get data from multiple channel at same time approximate between 4 to 10 channels per request depending on the site. This is often done by separate the side_id by a comma.

But also means the parser is able to handle the channels.

Quest is can you grabber provide this option or can be implemented. This will speed up the process in future.

I recently updated epg-grabber and it seems something broke?

getting the below error when running it.

grabbing the guide...

Starting...
Loading './sites.test/elcinema.com.test.config.js'...
Parsing:
Loading 'sites.test/elcinema.com.test.channels.xml'...
/data/data/com.termux/files/usr/lib/node_modules/epg-grabber/src/utils.js:54
return channels.elements
^

TypeError: Cannot read properties of undefined (reading 'elements')
at Object.utils.parseChannels (/data/data/com.termux/files/usr/lib/node_modules/epg-grabber/src/utils.js:54:19)
at main (/data/data/com.termux/files/usr/lib/node_modules/epg-grabber/bin/epg-grabber.js:40:26)
at Object. (/data/data/com.termux/files/usr/lib/node_modules/epg-grabber/bin/epg-grabber.js:66:1)
at Module._compile (node:internal/modules/cjs/loader:1101:14)
at Object.Module._extensions..js (node:internal/modules/cjs/loader:1153:10)
at Module.load (node:internal/modules/cjs/loader:981:32)
at Function.Module._load (node:internal/modules/cjs/loader:822:12)
at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:79:12)
at node:internal/main/run_main_module:17:47

attached the two files gzipped

elcinema.com.test.channels.xml.gz
elcinema.com.test.config.js.gz