wader / ydls Goto Github PK

View Code? Open in Web Editor NEW

182.0 6.0 27.0 796 KB

youtube-dl HTTP download and transcode service

Home Page: https://hub.docker.com/r/mwader/ydls/

License: MIT License

Shell 0.89% Go 96.50% HTML 0.31% Dockerfile 2.29%

youtube-dl ffmpeg docker-image id3v2 go http-server transcoder podcast rss golang

ydls's Introduction

youtube-dl HTTP service

HTTP service for yt-dlp that downloads media from requested URL and streams, transmux and transcode to requested format if needed.

It can be used to convert media, create podcasts and audio only versions of media from various site like youtube, vimeo etc.

Docker image uses ffmpeg from wader/static-ffmpeg which can handle most format and codecs. Default configuration can transcode to these format, container and codecs:

Format	Container	Audio	Video	Subtitle
alac	mp4	alac
flac	flac	flac
gif	gif		gif
m4a	mp4	aac
mp3	mp3	mp3
ogg	ogg	vorbis, opus
wav	wav	pcm_s16le
mkv	matroska	aac, alac, ac3, mp3, vorbis, opus, flac	h264, hevc, vp8, vp9, theora, av1	subrip, ass
mp4	mp4	aac, alac, mp3, vorbis, flac	h264, vp9, av1, hevc	mov_text
mxf	mxf	pcm_s16le	mpeg2video
ts	mpegts	aac, mp3, ac3	h264, hevc
webm	webm	vorbis, opus	vp8, av1, vp9	webvtt
rss	mp3	mp3

The rss format transforms a playlist into a RSS audio podcast.

See ydls.json for more details.

Usage

Run with docker

Pull mwader/ydls or build image using the Dockerfile. Run a container and publish TCP port 8080 somehow.

docker run -p 8080:8080 mwader/ydls

Build and install yourself

Run go get github.com/wader/ydls/cmd/ydls to install ydls. Make sure you have ffmpeg, yt-dlp, rtmpdump and mplayer installed and in PATH.

Copy and edit ydls.json to match your ffmpeg builds supported formats and codecs.

Start with ydls -server -config /path/to/ydls.json and it default will listen on port 8080.

Endpoints

Download and make sure media is in specified format:
GET /<format>[+option+option...]/<URL-not-encoded>
GET /?format=<format>&url=<URL>[&codec=...&codec=...&retranscode=...]

Download in best format:
GET /<URL-not-encoded>
GET /?url=<URL-encoded>

Parameters

format - Format name. See table above and ydls.json
URL - Any URL that yt-dlp can handle
URL-not-encoded - Non-URL-encoded URL. The idea is to be able to simply prepend the download URL with the ydls URL by hand without doing any encoding (for example in the browser location bar)
codec - Codec to use instead of default for format (can be specified one or two times for audio and video codec)
retranscode - Retranscode even if input codec is same as output
time - Only download specificed time range. Ex: 30s, 20m30s, 1h20m30s will limit duration. 10s-30s will seek 10 seconds and stop at 30 seconds (20 second output duration)
items - If playlist only include this many items

option - Codec name, time range, retranscode or <N>items

Examples

Download and make sure media is in mp3 format:
http://ydls/mp3/https://www.youtube.com/watch?v=cF1zJYkBW4A

Download using query parameters and make sure media is in mp3 format:
http://ydls/?format=mp3&url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DcF1zJYkBW4A

Download and make sure media is in webm format:
http://ydls/webm/https://www.youtube.com/watch?v=cF1zJYkBW4A

Download and make sure media is in mkv format using mp3 and h264 codecs:
http://ydls/mkv+mp3+h264/https://www.youtube.com/watch?v=cF1zJYkBW4A

Download and retranscode to mp3 even if input is already mp3:
http://ydls/mp3+retranscode/https://www.youtube.com/watch?v=cF1zJYkBW4A

Download specified time range in mp3:
http://ydls/mp3+10s-30s/https://www.youtube.com/watch?v=cF1zJYkBW4A

Download in best format:
http://ydls/https://www.youtube.com/watch?v=cF1zJYkBW4A

Playlist as audio podcast with 3 latest items:
http://ydls/rss+3items/https://www.youtube.com/watch?list=PLtLJO5JKE5YCYgIdpJPxNzWxpMuUWgbVi

Tricks and known issues

For some formats the transcoded file might have zero length or duration as transcoding is done while streaming. This is usually not a problem for most players.

Download with curl and save to filename provided by response header:

curl -OJ http://ydls-host/mp3/https://www.youtube.com/watch?v=cF1zJYkBW4A

Docker image can download from command line. This will download in mp3 format to current directory:

docker run --rm -v "$PWD:$PWD" -w "$PWD" mwader/ydls https://www.youtube.com/watch?v=cF1zJYkBW4A mp3

yt-dlp URL can point to a plain media file.

If you run the service using some cloud services you might run into geo-restriction issues with some sites like youtube.

Development

When fiddling with ffmpeg and yt-dlp related code I usually do this:

docker build --target ydls-dev -t ydls-dev . && docker run --rm -ti -v "$PWD:/$PWD" -w "$PWD" ydls-dev

Then inside dev container:

# run cli version
go run cmd/ydls/main.go -config ./ydls.json -debug
# run all tests
CONFIG="$PWD/ydls.json" TEST_EXTERNAL=1 go test -v -cover -race ./...

TODO

Optional stream for format? example mp4 at least video or audio?
Download playlist as zip archive.
JSON output?
Refactor messy downloadFormat function. Maybe a media interface with audio, video and subtitle implementations that select best format and return reader and mappings? should share a common format picker so formats can be shared and not re-downloaded.
Make download part into an interface to support more than yt-dlp.
Bitrate factor per codec when sorting formats (prefer aac over mp3 at same bitrate etc)
X-Remote IP header?

License

ydls is licensed under the MIT license. See LICENSE for the full license text.

ydls's People

Contributors

Stargazers

Watchers

ydls's Issues

RSS fails if any playlist videos are private

ydls is already ignoring errors which allows youtube-dl to successfully run in the case of private videos in a playlist.

Unfortunately, ydls still fails to load an rss feed if there is a private video. Perhaps because the youtube-dl info has nulls?

Example playlist (borrowed from ytdl-org/youtube-dl#1594)
https://www.youtube.com/playlist?list=PLX0g748fkegS54oiDN4AXKl7BR7mLIydP

Non-ASCII problem with files written

Try to download this: https://twitter.com/czaplinskiii/status/925709755649282048
youtube-dl downloads the video just fine, while ydls fails:

DEBUG: 14:58:45 ydl-new stdout> [twitter] 925709755649282048: Downloading webpage DEBUG: 14:58:46 ydl-new stdout> [twitter:card] 925709755649282048: Downloading webpage DEBUG: 14:58:47 ydl-new stdout> [twitter:card] 925709755649282048: Downloading m3u8 information DEBUG: 14:58:47 ydl-new stdout> [twitter:card] 925709755649282048: Downloading mobile webpage DEBUG: 14:58:48 ydl-new stdout> [twitter:card] 925709755649282048: Downloading main script DEBUG: 14:58:49 ydl-new stdout> [twitter:card] 925709755649282048: Downloading mobile API data DEBUG: 14:58:50 ydl-new stdout> [info] Writing video description metadata as JSON to: Marcin Czapliski - Uwaga Warszawa! Czy zapalicie znicz na grobach naszych dzielnych Uan�w, kt�rzy polegli w obronie 1939 roku Licz na Was i daj wskaz�wki.-925709755649282048.info.json DEBUG: 14:58:50 Failed to download: Cannot write metadata to JSON file Marcin Czapliski - Uwaga Warszawa! Czy zapalicie znicz na grobach naszych dzielnych Uan�w, kt�rzy polegli w obronie 1939 roku Licz na Was i daj wskaz�wki.-925709755649282048.info.json INFO: 14:58:50 [::1]:56376 Download failed GET /https://twitter.com/czaplinskiii/status/925709755649282048 (Cannot write metadata to JSON file Marcin Czapliski - Uwaga Warszawa! Czy zapalicie znicz na grobach naszych dzielnych Uan�w, kt�rzy polegli w obronie 1939 roku Licz na Was i daj wskaz�wki.-925709755649282048.info.json)

Add subtitle support

Is it possible to execute additional commands like --write-subs for youtube-dl and --embed-subs for ffmpeg?

Could you make it support range requests?

When you run ydls as http service, you may let user download by multi threads or watch the real time transcoded video.

http redirect

Hi wader, great tool, just what I needed...

Is it possible to make it redirect to video link instead of starting download?

run without docker

SO then you can either run locally or inject into docker.

ffpmeg.mk is a simple downloader for each OS. amd64 only for now.

# ffmpeg

# https://github.com/ffbinaries/ffbinaries-prebuilt/releases

# https://github.com/BtbN/FFmpeg-Builds/releases/tag/latest
# latest windows and linxu

# export for building against ffmpeg.
export PATH:=$(FFMPEG_BIN):$(PATH)

FFMPEG_GO_BIN_NAME=go

# Computed variables
FFMPEG_GO_PATH?= $(shell $(FFMPEG_GO_BIN_NAME) env GOPATH)
FFMPEG_GO_OS=$(shell $(FFMPEG_GO_BIN_NAME) env GOOS)
FFMPEG_GO_ARCH=$(shell $(FFMPEG_GO_BIN_NAME) env GOARCH)

# override for testing only.
#FFMPEG_GO_OS=windows
# https://github.com/ffbinaries/ffbinaries-prebuilt/releases/tag/v$(FFMPEG_SRC_VERSION)
FFMPEG_SRC_VERSION=4.4.1
# default to .bin, as then most users wil have a .gitingore for it.
FFMPEG_SRC_OUT_FSPATH=$(PWD)/.bin

# computed
FFMPEG_BIN=$(FFMPEG_SRC_OUT_FSPATH)/$(FFMPEG_GO_OS)__$(FFMPEG_GO_ARCH)
FFMPEG_BIN_WINDOWS=$(FFMPEG_SRC_OUT_FSPATH)/windows__$(FFMPEG_GO_ARCH)
FFMPEG_BIN_DARWIN=$(FFMPEG_SRC_OUT_FSPATH)/darwin__$(FFMPEG_GO_ARCH)
FFMPEG_BIN_LINUX=$(FFMPEG_SRC_OUT_FSPATH)/linux__$(FFMPEG_GO_ARCH)

# const
FFMPEG_DOWNLOAD_URL_PREFIX=https://github.com/ffbinaries/ffbinaries-prebuilt/releases/download

ffmpeg-print:
	@echo ""
	@echo "FFMPEG_GO_PATH:           $(FFMPEG_GO_PATH)"
	@echo ""
	@echo "--Computed variables:"
	@echo "FFMPEG_GO_OS:             $(FFMPEG_GO_OS)"
	@echo "FFMPEG_GO_ARCH:           $(FFMPEG_GO_ARCH)"
	@echo "FFMPEG_BIN:               $(FFMPEG_BIN)"
	@echo "FFMPEG_BIN_WINDOWS:       $(FFMPEG_BIN_WINDOWS)"
	@echo "FFMPEG_BIN_DARWIN:        $(FFMPEG_BIN_DARWIN)"
	@echo "FFMPEG_BIN_LINUX:         $(FFMPEG_BIN_LINUX)"
	@echo ""
	@echo "--Override variables:"
	@echo "FFMPEG_SRC_VERSION:       $(FFMPEG_SRC_VERSION)"
	@echo "FFMPEG_SRC_OUT_FSPATH:    $(FFMPEG_SRC_OUT_FSPATH)"
	
	
ffmpeg-dep-pre:
	go install github.com/hashicorp/go-getter/cmd/go-getter@latest


ffmpeg-dep: ffmpeg-dep-pre
ifeq ($(FFMPEG_GO_OS),windows)
	$(MAKE) ffmpeg-dep-windows
endif

ifeq ($(FFMPEG_GO_OS),darwin)
	$(MAKE) ffmpeg-dep-darwin
endif

ifeq ($(FFMPEG_GO_OS),linux)
	ffmpeg-dep-linux
endif 

ffmpeg-dep-all: ffmpeg-dep-pre ffmpeg-dep-windows ffmpeg-dep-darwin ffmpeg-dep-linux

ffmpeg-dep-windows:
	go-getter $(FFMPEG_DOWNLOAD_URL_PREFIX)/v$(FFMPEG_SRC_VERSION)/ffmpeg-$(FFMPEG_SRC_VERSION)-win-64.zip $(FFMPEG_BIN_WINDOWS)
	go-getter $(FFMPEG_DOWNLOAD_URL_PREFIX)/v$(FFMPEG_SRC_VERSION)/ffprobe-$(FFMPEG_SRC_VERSION)-win-64.zip $(FFMPEG_BIN_WINDOWS)
	go-getter $(FFMPEG_DOWNLOAD_URL_PREFIX)/v$(FFMPEG_SRC_VERSION)/ffplay-$(FFMPEG_SRC_VERSION)-win-64.zip $(FFMPEG_BIN_WINDOWS)
	
	chmod +x $(FFMPEG_BIN_WINDOWS)/ffmpeg.exe
	chmod +x $(FFMPEG_BIN_WINDOWS)/ffprobe.exe
	chmod +x $(FFMPEG_BIN_WINDOWS)/ffplay.exe
ffmpeg-dep-darwin:
	go-getter $(FFMPEG_DOWNLOAD_URL_PREFIX)/v$(FFMPEG_SRC_VERSION)/ffmpeg-$(FFMPEG_SRC_VERSION)-osx-64.zip $(FFMPEG_BIN_DARWIN)
	go-getter $(FFMPEG_DOWNLOAD_URL_PREFIX)/v$(FFMPEG_SRC_VERSION)/ffprobe-$(FFMPEG_SRC_VERSION)-osx-64.zip $(FFMPEG_BIN_DARWIN)
	go-getter $(FFMPEG_DOWNLOAD_URL_PREFIX)/v$(FFMPEG_SRC_VERSION)/ffplay-$(FFMPEG_SRC_VERSION)-osx-64.zip $(FFMPEG_BIN_DARWIN)
	
	chmod +x $(FFMPEG_BIN_DARWIN)/ffmpeg
	chmod +x $(FFMPEG_BIN_DARWIN)/ffprobe
	chmod +x $(FFMPEG_BIN_DARWIN)/ffplay
ffmpeg-dep-linux:
	go-getter $(FFMPEG_DOWNLOAD_URL_PREFIX)/v$(FFMPEG_SRC_VERSION)/ffmpeg-$(FFMPEG_SRC_VERSION)-linux-64.zip $(FFMPEG_BIN_LINUX)
	go-getter $(FFMPEG_DOWNLOAD_URL_PREFIX)/v$(FFMPEG_SRC_VERSION)/ffprobe-$(FFMPEG_SRC_VERSION)-linux-64.zip $(FFMPEG_BIN_LINUX)
	
	# ffplay does not exist for linux
	chmod +x $(FFMPEG_BIN_LINUX)/ffmpeg
	chmod +x $(FFMPEG_BIN_LINUX)/ffprobe

### Delete

ffmpeg-dep-del:
	
ifeq ($(FFMPEG_GO_OS),windows)
	$(MAKE) ffmpeg-dep-del-windows
endif

ifeq ($(FFMPEG_GO_OS),darwin)
	$(MAKE) ffmpeg-dep-del-darwin
endif

ifeq ($(FFMPEG_GO_OS),linux)
	$(MAKE) ffmpeg-dep-del-linux
endif 

ffmpeg-dep-del-all: ffmpeg-dep-del-windows ffmpeg-dep-del-darwin ffmpeg-dep-del-linux
	

ffmpeg-dep-del-windows:
	rm -f $(FFMPEG_BIN_WINDOWS)/ffmpeg.exe
	rm -f $(FFMPEG_BIN_WINDOWS)/ffplay.exe
	rm -f $(FFMPEG_BIN_WINDOWS)/ffprobe.exe

ffmpeg-dep-del-darwin:
	rm -f $(FFMPEG_BIN_DARWIN)/ffmpeg
	rm -f $(FFMPEG_BIN_DARWIN)/ffplay
	rm -f $(FFMPEG_BIN_DARWIN)/ffprobe

ffmpeg-dep-del-linux:
	rm -f $(FFMPEG_BIN_LINUX)/ffmpeg
	rm -f $(FFMPEG_BIN_LINUX)/ffplay
	rm -f $(FFMPEG_BIN_LINUX)/ffprobe

Stream youtube-dl -> ffmpeg

Can you pass the youtube-dl stdout directly to ffmpeg and transmit it to the user?? This results in fast delivery of content.

feature: Specify bitrate

Specific use-case: When downloading voice audio (such as podcasts or lectures) from Youtube over limited data plans, it would be great to be able to specify a very low bitrate in combination with the opus codec.

Feature request: A bitrate option in the parameters that is passed ffmpeg.

Workaround: Would adding parameters to either OutputFlags or FormatFlags in ydls.json work? For a 25kbps example, Youtube-dl would be --audio-quality 25K and for ffmpeg, -b 25k --mode mono.

gif support

i forked the project and i tried to add gif support to JSON config and when i try to build it it give an error

ERROR: failed to solve: process "/bin/sh -c CONFIG=/src/ydls.json   TEST_EXTERNAL=1   go test -timeout=30m -buildvcs=false -v -cover $([ \"$TARGETARCH\" = \"amd64\" ] && echo -race) ./..." did not complete successfully: exit code: 1

Arm Support

Is there any way that you could release arm binaries?

Issues with reverse proxy?

Hi, thanks for making this great tool!

I've got it to work using the docker build with my internal IP like so http://192.168.x.xxx:8051/mp3/....

However, after putting it behind a reverse proxy with nginx I am getting an error using the same URL: https://ytcut.mydomain.me/mp3/...

INFO: 13:13:35 172.17.0.1:43952 Downloading (mp3) https:/www.youtube.com/watch?v=3wg5UdoyswQ
INFO: 13:13:35 172.17.0.1:43952 Downloading (mp3) https:/www.youtube.com/watch?v=3wg5UdoyswQ
INFO: 13:13:41 172.17.0.1:43952 Download failed GET /mp3/https:/www.youtube.com/watch ([generic] Unable to download webpage: <urlopen error no host given> (caused by URLError('no host given')))

This is what my reverse proxy block in nginx looks like:

server {
	listen 443 ssl http2;
	server_name ytcut.myDomain.me;
	
	location / {
        
        proxy_pass         http://192.168.x.xxx:8051/;
        auth_basic "Password Protected";
        auth_basic_user_file /config/nginx/.htpasswd;
  }
}

Any idea why I would be getting this error?

feature: just download locally

Please add this case: run ydls as service on remote host and storing downloaded content on same host

Bug in "Playlist as audio podcast with 3 latest items" mode without playlist-reverse

In debug mode generates this: yt-dlp --ignore-errors --no-call-home --no-cache-dir --skip-download --restrict-filenames --batch-file - -J --yes-playlist --playlist-end 3

but this gives oldest 3 files from playlist, not newest. Possible You forget --playlist-reverse param

External Downloader

Hello,

Any possibility to add option for external downloader or add some arguments?
eg. "--external-downloader aria2c" and "--http-chunk-size 10M"

On some files I get very slow downloads (same on ydls and youtube-dl directly) but when using external downloader or adding chunk size it goes well with youtube-dl

Reason for strict CSP of index?

I want to show some demos of the service on the index page but <audio /> and friends won't work because of CSP. Is there any reason why you have it this way? Given that the index template is provided by the server admin...?