hoshsadiq / m3ufilter Goto Github PK

I am trying to add a small enhancement - output CUID as a hash of the url, but am having difficulty building. It fails with the error:

fatal: repository 'https://github.com/hoshsadiq/goval/' not found
fatal: repository 'https://github.com/hoshsadiq/m3u8/' not found

I don't see those, so they are probably private?

could you make them available, or add this small enhancement to the code to create a hash from the url:

index fb67b7a..6e6e6ef 100644
--- a/writer/m3u.go
+++ b/writer/m3u.go
@@ -5,6 +5,8 @@ import (
 	"io"
 	"strconv"
 	"strings"
+	"crypto/md5"
+	"encoding/hex"
 )

 func writeM3U(w io.Writer, streams []*m3u.Stream) {
@@ -22,6 +24,12 @@ func writeM3U(w io.Writer, streams []*m3u.Stream) {
 	}
 }

+func GetMD5Hash(text string) string {
+    hasher := md5.New()
+    hasher.Write([]byte(text))
+    return hex.EncodeToString(hasher.Sum(nil))
+}
+
 func getStreamExtinf(stream *m3u.Stream) []byte {
 	b := &strings.Builder{}
 	b.WriteString("\n")
@@ -32,6 +40,7 @@ func getStreamExtinf(stream *m3u.Stream) []byte {
 		writeKV(b, "tvg-chno", stream.ChNo)
 	}

+	writeKV(b, "CUID", GetMD5Hash(stream.Uri))
 	writeKV(b, "tvg-id", stream.Id)
 	writeKV(b, "tvg-name", stream.Name)
 	writeKV(b, "group-title", stream.Group)```

Hello,

I had a panic with the master branch (go 1.13.8):

go run cmd/m3u-filter/main.go -config=config.yaml

panic: runtime error: index out of range [1] with length 1

goroutine 1 [running]:
github.com/hoshsadiq/m3ufilter/logger.Get.func1(0xc0000984b0, 0x0, 0x83f49f, 0x3, 0xb7a0c0)
	/home/user/git/m3ufilter/logger/logger.go:21 +0x314
github.com/sirupsen/logrus.(*TextFormatter).Format(0xc000098320, 0xc0000e8070, 0x30, 0x7f9220, 0xc0000e6401, 0xc0000815c0, 0xc00009f9f0)
	/home/user/go/pkg/mod/github.com/sirupsen/[email protected]/text_formatter.go:133 +0x53b
github.com/sirupsen/logrus.(*Entry).write(0xc0000e8070)
	/home/user/go/pkg/mod/github.com/sirupsen/[email protected]/entry.go:255 +0xa1
github.com/sirupsen/logrus.Entry.log(0xc000097180, 0xc000081590, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/home/user/go/pkg/mod/github.com/sirupsen/[email protected]/entry.go:231 +0x19d
github.com/sirupsen/logrus.(*Entry).Log(0xc0000e8000, 0x4, 0xc00009fc10, 0x1, 0x1)
	/home/user/go/pkg/mod/github.com/sirupsen/[email protected]/entry.go:268 +0xeb
github.com/sirupsen/logrus.(*Entry).Logf(0xc0000e8000, 0xc000000004, 0x848ea5, 0x1d, 0xc00009fdb8, 0x2, 0x2)
	/home/user/go/pkg/mod/github.com/sirupsen/[email protected]/entry.go:314 +0xe2
github.com/sirupsen/logrus.(*Logger).Logf(0xc000097180, 0x4, 0x848ea5, 0x1d, 0xc00009fdb8, 0x2, 0x2)
	/home/user/go/pkg/mod/github.com/sirupsen/[email protected]/logger.go:145 +0x94
github.com/sirupsen/logrus.(*Logger).Infof(...)
	/home/user/go/pkg/mod/github.com/sirupsen/[email protected]/logger.go:159
github.com/hoshsadiq/m3ufilter/m3u.GetPlaylist(0xc000081350, 0xb790c8, 0x0, 0x0, 0x4e8200)
	/home/user/git/m3ufilter/m3u/process.go:35 +0x22e
main.run(0x7ffded7c860d, 0xb, 0xc000010018, 0xc000010020)
	/home/user/git/m3ufilter/cmd/m3u-filter/main.go:42 +0x11e
main.main()
	/home/user/git/m3ufilter/cmd/m3u-filter/main.go:31 +0x26b
exit status 2

Println for some variables:

go run cmd/m3u-filter/main.go -config=config.yaml

f.File: /home/user/git/m3ufilter/m3u/process.go
basePkg: github.com/hoshsadiq/m3ufilter
strings.Split: [/home/user/git/m3ufilter/m3u/process.go]

I'm using Windows 10 computer.

I've made a config and in the logg I see it finds streams according to my file but the output file is empty.

I run this command: m3ufilter -config /path/to/config.yaml >myfile.m3u

But the myfile.m3u is empty

My config looks like this

core:
update_schedule: "*/24 * * * *"
output: m3u
group_order:
providers:

• uri: file:///path/to/m3u/playlist.m3u
  filters:

  • match(Group, "Swedish")

  setters:
  filters:

epg_providers:

• uri: file:///path/to/epg.xml
  channel_id_renames:

I hope you can help me

Is your feature request related to a problem? Please describe.
This request is made based on this dialog:
s79dev:

I want use VLC or other player. I wanted use m3ufilter for filter playlist my provider, skip all channels not allow for me. I realized that I could not do this when I saw that the program used a structure, not a map.

hoshsadiq:

Feel free to raise a feature request and/or a PR. If you raise a feature request, I'll look at it if I have some time or something, but can't promise anything. Adding additional fields and things like this essentially requires a full implementation of the m3u RFC, which adds a lot of overheads to the application, to the point where it might not be possible to run it on low resource devices such as the pi.

Describe the solution you'd like
I think the solution problem full implementation of the m3u RFC with lot of overheads may be the following:
Change static structure "Stream" with predefined tag, on dynamic structure map: map[string]string where tag name is key and tag value as value. I thing what in golang for map not many overheads. And additionally user can set tags need for output

Hello,

I would like to remove almost all groups from the original m3u but in the output, channels are mixed (French and USA channels).

Here my config:

providers:
  - uri: http://randomurl.com
    filters:
        - match(Group, "EU FRANCE FULL HD") || match(Group, "^USA$")

Is my configuration wrong?

It would be nice if you could also filter the entries from an XMLTV file to reduce it.
Basically this would mean removing from the XMLTV file the entries linked to channels that are filtered out by the M3U filters.

The resulting XMLTV file should contain only information in line with the channels present in the .m3u file after filtering.
The filtered XMLTV file could also be served by the web integrated web server through another url (/epg.xml).

Thanks for the consideration.

Some of the documentation is outdated. This should be updated.

From a PM:

Another thing that should be documented is that the CSV output doesn't show anything but: tvg-id group-title tvg-name duration tvg-logo uri

And that the variables are case sensitive. I think they may have been shown in some documentation with a capital letter when they are all lower case. Needs confirming, but it does make sense

Hello I used your docker file on my server and it seems that i can't get it to output a m3u to my output folder.

"-config /config/config.yaml -playlist /config"

My logs aren't showing any errors.

time="2019-08-10T20:49:32+02:00" level=info msg="Scheduling cronjob to periodically update playlist."
time="2019-08-10T20:49:32+02:00" level=info msg="Parsing for the first time..."
time="2019-08-10T20:49:32+02:00" level=info msg="starting server"
time="2019-08-10T20:49:32+02:00" level=info msg="updating playlists"
time="2019-08-10T20:49:32+02:00" level=info msg="reading from provider http://xxxxxxx.com:1234/get.php?username=xxx&password=xxxx&type=m3u_plus&output=ts"
time="2019-08-10T20:49:32+02:00" level=info msg="Parsing 1000 streams"
time="2019-08-10T20:49:33+02:00" level=info msg="Parsing 2000 streams"
time="2019-08-10T20:49:33+02:00" level=info msg="Parsing 3000 streams"
time="2019-08-10T20:49:33+02:00" level=info msg="Parsing 4000 streams"
time="2019-08-10T20:49:33+02:00" level=info msg="Parsing 5000 streams"
time="2019-08-10T20:49:33+02:00" level=info msg="Parsing 6000 streams"
time="2019-08-10T20:49:33+02:00" level=info msg="Found 763 valid streams"
time="2019-08-10T20:49:33+02:00" level=info msg=done

Revision:

commit bea7bb59a13698cb06652db676b9ed0ff4de614d (HEAD -> master, origin/master, origin/HEAD)
Author: Hosh <[email protected]>
Date:   Wed Apr 29 15:33:45 2020 +0100

Build result:
go version
go version go1.14.3 linux/amd64
go build main.go

go: downloading github.com/sirupsen/logrus v1.4.2
go: downloading github.com/mitchellh/go-homedir v1.1.0
go: downloading gopkg.in/yaml.v2 v2.2.2
go: downloading github.com/mileusna/crontab v0.0.0-20180909013013-f182897de5d8
go: downloading github.com/hoshsadiq/goval v1.0.1-0.20190525223338-f1ea9f026acd
go: downloading github.com/maja42/no-comment v0.0.0-20180113082502-512948848672
go: downloading github.com/PuerkitoBio/rehttp v1.0.0
go: downloading golang.org/x/sys v0.0.0-20190422165155-953cdadca894

File "main" rename to "m3ufilter" and move
Config:

core:
  output: m3u
providers:
  - uri: http:\vladlink.tv\playlist

Comand line
./m3ufilter -config=./config.yaml -log ./err -playlist ./test
Result:
panic: runtime error: index out of range [1] with length 1

goroutine 1 [running]:
github.com/hoshsadiq/m3ufilter/logger.Get.func1(0xc00007c500, 0x0, 0x827432, 0x3, 0xb7abc0)
	/home/s79/mk/m3ufilter/logger/logger.go:30 +0x393
github.com/sirupsen/logrus.(*TextFormatter).Format(0xc00007c370, 0xc00016a0e0, 0xc000125a28, 0x59f28d, 0x7e7d60, 0xc000109620, 0xc000125a70)
	/home/s79/bin/go/pkg/mod/github.com/sirupsen/[email protected]/text_formatter.go:133 +0x536
github.com/sirupsen/logrus.(*Entry).write(0xc00016a0e0)
	/home/s79/bin/go/pkg/mod/github.com/sirupsen/[email protected]/entry.go:255 +0x7c
github.com/sirupsen/logrus.Entry.log(0xc00011f180, 0xc0001095f0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/home/s79/bin/go/pkg/mod/github.com/sirupsen/[email protected]/entry.go:231 +0x19e
github.com/sirupsen/logrus.(*Entry).Log(0xc00016a070, 0x4, 0xc000125c38, 0x1, 0x1)
	/home/s79/bin/go/pkg/mod/github.com/sirupsen/[email protected]/entry.go:268 +0xeb
github.com/sirupsen/logrus.(*Entry).Logf(0xc00016a070, 0xc000000004, 0x831002, 0x1d, 0xc000125de0, 0x2, 0x2)
	/home/s79/bin/go/pkg/mod/github.com/sirupsen/[email protected]/entry.go:314 +0xe2
github.com/sirupsen/logrus.(*Logger).Logf(0xc00011f180, 0x4, 0x831002, 0x1d, 0xc000125de0, 0x2, 0x2)
	/home/s79/bin/go/pkg/mod/github.com/sirupsen/[email protected]/logger.go:145 +0x94
github.com/sirupsen/logrus.(*Logger).Infof(...)
	/home/s79/bin/go/pkg/mod/github.com/sirupsen/[email protected]/logger.go:159
github.com/hoshsadiq/m3ufilter/m3u.GetPlaylist(0xc000109350, 0xb79b10, 0x0, 0x0, 0x200)
	/home/s79/mk/m3ufilter/m3u/process.go:35 +0x22e
main.run(0x7ffd22d61278, 0xd, 0xc000010050, 0xc000010058)
	/home/s79/mk/m3ufilter/cmd/m3u-filter/main.go:49 +0x111
main.main()
	/home/s79/mk/m3ufilter/cmd/m3u-filter/main.go:38 +0x294

Create 2 files with zero size:
err
test

I am also finding that in the version I am using, tvg-id is not being set.

Is your feature request related to a problem? Please describe.

IPTV repo (https://github.com/iptv-org/iptv) provides m3u lists for each country and each entry has a language attribute (tvg-language) and would like to filter based on it.

In some countries where there is more than one spoken language, for example India, there are a lot of IPTV entries for various languages and would like to filter to include the ones I want.

Describe the solution you'd like

m3ufilter is able to parse tvg-language field from m3u (example: https://github.com/iptv-org/iptv/blob/master/channels/in.m3u) and filter based on language.

For example, the filtering config would look something like below

providers:
  - uri: https://raw.githubusercontent.com/iptv-org/iptv/master/channels/in.m3u
    filters:
     - match(language, "English")
     - match(language, "Tamil")

Describe alternatives you've considered

One alternative I have considered is to take the per language list provided by the project (https://github.com/iptv-org/iptv/blob/master/README.md#playlists-by-language) and merge them before passing it on to m3ufilter.

My provider sends a barebone M3U file.
Ex:

#EXTM3U
#EXTINF:-1,ABC East
http://some.stream/23

I'd still like to filter the file via the provider filter but there's no attribute to filter on. If the provider filter isn't used, it brings the name in as the tvg-name which is good but that doesn't mean that I can filter based on the attribute tvg-name.
Is there a way to do this besides having m3ufilter import from the website and then import it's own created file again where tvg-name would be available for filtering?

I would like to see the possibility to filter channels by quality.
My IPTV provider makes most channels (but not all) available in multiple qualities. Here is an example:

BBC One
BBC One SD
BBC One HD
BBC One FHD
BBC One UHD

So far, in m3u-filter, I can use a regex filter to select only FHD channels for example.
However, not all channels are available in every quality. For example:

BBC Two SD
BBC Two HD

So if I filter for example on FHD, I will not get BBC Two.
If I filter on HD, I get both channels, but not with the quality I would like (FHD).

Would it be possible to add a filter to choose a certain quality level and avoid duplicate?
An idea would be to define a score by quality, for example:

SD or no quality indication = 1
HD = 2
FHD = 3
UHD = 4

Then define the target quality, for example FHD (score 3)
Then, for any channel with duplicates, take the channel with the highest score which is less or equal to the target quality.

Using my previous example, this would select:

BBC One FHD (score 3)
BBC Two HD (score 2)

This would help filter out a lot of duplicate channels.
Thanks for your consideration on this request.

First off, Great script ! It's very useful,started to implement it into my system.

However some of my other scripts that are run after yours are throwing a wobbly about the inclusion of the CUID's. Whilst i understand why it is useful, it's not always needed and in this case (.strm converter script h0me5k1n/iptv-strmmaker) it causes error.

Also i'm not too sure if tvheadend is wiping the mux when the url changes as it thinks it's a new channel rather than an update.

I'm a bit of a n00b with linux & scripts, apologies!

PS I am using your docker image from the docker hub

If an item contains a tvg-name attribute with a value containing quotes, the parsing would break:

#EXTINF:-1 tvg-id="" tvg-name="This is "a test"" tvg-logo="http://localhost/logo.jpg" group-title="TEST",This is "a test"

results in the error

time="2019-09-09T14:36:30+02:00" level=error msg="could not retrieve playlist from provider file:////playlist_1.m3u, err = Unexpected character '\"' found, expected '=' for key test on position 46 in line: #EXTINF:-1 tvg-id=\"\" tvg-name=\"This is \"a test\"\" tvg-logo=\"http://localhost/logo.jpg\" group-title=\"TEST\",This is \"a test\"

I was looking for something like this. Unfortunatety I'm not a go programmer and a missed some steps to install and run this.
I belive that it could be possible to install binaries someway in linux/windows, or build and run from this repository code

go build can't load package: package github.com/hoshsadiq/m3ufilter: unknown import path "github.com/hoshsadiq/m3ufilter": cannot find module providing package github.com/hoshsadiq/m3ufilter

I would like o reopen this issue as I have the same problem.

Platform: Ubuntu 18.04.4 LTS - AMD64
Version: v0.1-beta.3 - using binary: m3u-filter_linux_arm64 - Downloaded executable, did NOT build from source myself

Command line: ./m3u-filter -config ./data/m3u.conf -playlist ./data/playlist.m3u

Error Message:
panic: open ./data/playlist.m3u: no such file or directory

goroutine 1 [running]: main.fd(0x7ffeacbae7c5, 0x13, 0x7ffeacbae700, 0xf) /home/travis/gopath/src/github.com/hoshsadiq/m3ufilter/cmd/m3u-filter/main.go:59 +0xf6 main.main() /home/travis/gopath/src/github.com/hoshsadiq/m3ufilter/cmd/m3u-filter/main.go:30 +0x228

The error is the same if I specify a logfile, with or without the -playlist argument.

I tried creating an empty playlist.m3u file but then I get:
FATA[2020-04-01T20:19:31+02:00]writer/m3u.go:15 writer.writeM3U() unable to write extm3u, err = !!write ./data/playlist.m3u: bad file descriptor

Content of m3u.conf file:
core:
output: m3u
group_order:
- France
- Switzerland
- Belgium
- VOD France
providers:
- uri: file:///home/Services/m3u-filter/data/playlist.ts
filters:
- match(Group, "France")
- match(Group, "Switzerland")
- match(Group, "Belgium")
- match(Group, "VOD France")

If I don't specify the -log or -playlist arguments the filtered m3u list is displayed on the console properly.

Hopefully this will help narrowing down the issue.

Originally posted by @Olivier6767 in #4 (comment)

I have a m3u with channel aaa, the epg grabber generates channel ids aa.provider.com
to make most part to match I change the id of the m3u channel to be tvg_id(Name)+".provider.com"

but I wondering if possible to leave the channel id as is and have a function to map like

epg_providers.channel_id_matcher

• id : replace(Id, ".provider.com", "")

In m3u with non English characters all non English characters is broken.
This is because the stream reads single-byte characters, not runes.
Possible Solution:

--- a/m3u/m3u.go
+++ b/m3u/m3u.go
@@ -146,8 +146,8 @@ func parseExtinfLine(attrline string, urlLine string) (*Stream, error) {
	value := ""
	quote := "\""
	escapeNext := false
-	for i := 8; i < len(attrline); i++ {
-		c := attrline[i]
+	for i,c := range attrline {
+		if i<8  {continue}

		if escapeNext {
			if state == "duration" {
-- 
2.17.1

I would like to suggest a way to specify an input csv file to batch filter/set individual streams:

core:
  server_listen: localhost:8080
  update_schedule: "*/24 * * * *"
  output: m3u
  group_order:
    - Entertainment
    - Family/Kids
    - News
    - Drama
providers:
  - uri: file:///path/to/m3u/playlist.m3u
    csv_filter: file:///path/to/m3u/filter.csv
    csv_setter:
        - Id: tvg_id_csv_col
        csv_filter_field: CUID_csv_col

what do you think @hoshsadiq ?

Hi!

I need some help, I have configured config.yaml file but when I run

m3ufilter -config /pathTo/m3ufilter-master/config.yaml

I get terminal error

zsh: command not found: m3ufilter

What do I need to run before I run m3ufilter script?

EDIT:
When I run docker build I get an error message:
COPY failed: stat /var/lib/docker/tmp/docker-builder662332369/build/m3u-filter_linux_amd64: no such file or directory

Thank you!

Looks interesting project.

Do you have a demo?

This is not really a bug nor a feature request, as the only thing I need is some help to set up the config file as I'm having serious problems to understand how to work with YAML. What I'm trying is to remove from my list channels that end with mkv (as they are just movies and not real live channels.

I tried following your config file but it says the syntax is not correct when using: - !match(Uri, "mkv").

This is my code:

core:
  server_listen: localhost:8090
  update_schedule: "*/24 * * * *"
  output: m3u
  group_order:
    - Entertainment
    - Family/Kids
    - News
    - Drama
providers:
  - uri: myserverurl
    filters:
      - !match(Uri, "mkv")
    setters:
      - name: replace(Name, "[\\s\\:\\|]+", " ")
      - name: replace(Name, "^VIP ", "")
      - name: replace(Name, "USA", "")
        attributes:
          tvg-id: tvg_id(Name) + ".us"
        filters:
          - Name == "USA CNN"
          - Name == "CNN"
          - Name == "CNN HD"
epg_providers:
  - uri: myserverurl
    channel_id_renames:
      replacement: find # key = what to replace it with, value = what to find
      bbc.uk: "BBC One"