Suppress/verbose flags for yt-dlp CLI logs

--debug
  Print all ytdl-sub debug and info messages, and yt-dlp messages

--verbose
  Print all ytdl-sub info messages and yt-dlp messages
  
--quiet
  Hide all ytdl-sub and yt-dlp messages
  
<default>
  Only show ytdl-sub info messages, not yt-dlp

Have it show all output files without downloading

ytdl-sub dl --preset "yt_music_video" --youtube.video_id "nZXRV4MezEw" --overrides.artist "Cher"

produced:

  File "/usr/bin/ytdl-sub", line 8, in <module>
    sys.exit(main())
  File "/usr/lib/python3.10/site-packages/ytdl_sub/main.py", line 78, in main
    _download_subscription_from_cli(config=config, extra_args=extra_args)
  File "/usr/lib/python3.10/site-packages/ytdl_sub/main.py", line 58, in _download_subscription_from_cli
    subscription.download()
  File "/usr/lib/python3.10/site-packages/ytdl_sub/subscriptions/subscription.py", line 247, in download
    plugin.post_process_entry(entry)
  File "/usr/lib/python3.10/site-packages/ytdl_sub/plugins/nfo_tags.py", line 118, in post_process_entry
    self.archive_entry_file_name(entry=entry, relative_file_path=nfo_file_name)
  File "/usr/lib/python3.10/site-packages/ytdl_sub/plugins/plugin.py", line 57, in archive_entry_file_name
    self.__enhanced_download_archive.mapping.add_entry(
  File "/usr/lib/python3.10/site-packages/ytdl_sub/ytdl_additions/enhanced_download_archive.py", line 391, in mapping
    raise ValueError("Tried to use download mapping before it was loaded")
ValueError: Tried to use download mapping before it was loaded

(plz dont laugh, my wife likes this song)

Here's where all the thumbnail/pillow logic exists:
https://github.com/jmbannon/ytdl-sub/blob/master/src/ytdl_sub/utils/thumbnail.py

There's a bug with yt-dlp that reports it downloading a jpg but actually downloads webp. Not sure if it's fixed yet.
For a first pass though I think we can just replace the PIL functions with an FFMPEG call using this helper class:
https://github.com/jmbannon/ytdl-sub/blob/master/src/ytdl_sub/utils/ffmpeg.py

All of the e2e test md5 hashes for jpgs will probably need to get updated when changing this:
https://github.com/jmbannon/ytdl-sub/blob/master/tests/e2e/youtube/test_channel_as_kodi_tv_show.py#L79

Make sure the image still looks good

Hi there,

thanks for the great software.

I want to set sponsorblock settings that yt-dlp has and was wondering if being able to set raw yt-dlp options to use during download might be a good feature for ytdl-sub, since it might also be used for other preferences people might have.

Might also be used for setting video/audio codecs to download for example.

Thanks again.

Makes for better sorting of variable names:

• title
• title_sanitized
• track_title
• track_title_sanitized

https://kodi.wiki/view/NFO_files/TV_shows

Should add support for writing to tv_show.nfo to append a namedseason field.

Media players universally accept jpg/jpeg images. We should just cast all image types (looking at you, webp) to jpg. We already do for youtube profile/banner pics

Be able to specify regex to include or exclude media by matching title name to regex

Should look like

my_preset:
  regex_filter:
    title:
      include:  # If one of these matches, include it
      - ".*Official Video.*"
      exclude:  # If one of these matches, exclude it
      - "sadfasdfdf"

my_preset:
  youtube:
    download_strategy: "split_video"
    video_id: <id of video>
    split_file: "path/to/split_file.txt"

For a video like this https://www.youtube.com/watch?v=Tf1DEI2lEe0 , it would be great to download the audio, and split it up into separate songs based on the description's timestamps:

0:00 DOOM Eternal
4:48 Hell on Earth
9:29 Deag Nilox - First Priest Death
10:14 Barging In
12:32 Demonic Corruption
16:23 Prayer of the Diminished
17:51 King Novik
...

Probably cannot always rely on videos having chapters, or descriptions having these timestamps (often found in comments). Copy-pasting it to a file one-time with an expected format like above would suffice as an MVP

The 0.1.0 release did not create a readthedocs version for that specific release. Needs to be fixed.

Ideally, we want this app to be hosted by linuxserver at some point. For now though, we should have our own Dockerfile.

The base image should be linuxserver's base image found here:
https://github.com/linuxserver/docker-baseimage-alpine

This image includes S6 overlay which is very convenient for cron jobs, as well as PUID/PGID/TZ env variables.

The Dockerfile will probably look somewhat similar to the linuxserver beets docker image, found here:
https://github.com/linuxserver/docker-beets/blob/master/Dockerfile

Source variables are information from the media downloaded from ytdlp that can be used to set file names or metadata. They are documented here:
https://ytdl-sub.readthedocs.io/en/latest/config.html#source-variables

If you notice, a few variables' formatting looks weird. Like this one:
https://ytdl-sub.readthedocs.io/en/latest/config.html#ytdl_sub.entries.variables.youtube_variables.YoutubeVideoVariables.artist

There are random asterisks, and it seems like the NOTE changes formatting in some way.

The documentation for these should get cleaned up. They are generated from the docstrings in here:
https://github.com/jmbannon/ytdl-sub/tree/master/src/ytdl_sub/entries/variables

The docs are created using reStructuredText (https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html) and the autodocs extension (https://www.sphinx-doc.org/en/master/usage/extensions/autodoc.html)

To build the docs locally, run make docs

If a config or subscription YAML's syntax is incorrect, and cannot be loaded, we should gracefully error with an informative message to the user.

Traceback (most recent call last):
  File "/usr/bin/ytdl-sub", line 8, in <module>
    sys.exit(main())
  File "/usr/lib/python3.10/site-packages/ytdl_sub/main.py", line 73, in main
    _download_subscriptions_from_yaml_files(config=config, args=args)
  File "/usr/lib/python3.10/site-packages/ytdl_sub/main.py", line 27, in _download_subscriptions_from_yaml_files
    presets += Preset.from_file_path(config=config, subscription_path=preset_path)
  File "/usr/lib/python3.10/site-packages/ytdl_sub/config/preset.py", line 284, in from_file_path
    with open(subscription_path, "r", encoding="utf-8") as file:
FileNotFoundError: [Errno 2] No such file or directory: 's'

Should use 'subscriptions.yaml' by default

Shows the user what has changed. Make another arg to save it to a file

There is no way to add metadata in the downloader.download for each individual entries. Would be nice for merge_playlist or split_videos to show chapter information

For adding personal video media to plex:

Some video filetypes (e.g. MP4, M4V, MOV) allow you to embed metadata within the file. If your files have embedded metadata, that information can be used. To do so, ensure that the Local Media Assets source is enabled and ordered correctly.

(source: https://support.plex.tv/articles/200265246-personal-media-movies/)

I do not use Plex and am not familiar with this process whatsoever. If anyone has done this before and could provide some insight, that would be great.

yt-dlp needs it to do fancy stuff

For massive downloads of videos from channels or playlists, we should support batch downloading out of the box.

For playlists, batch by index. For channels, batch by date interval (day, week, month).
For each batch, process all the files end-to-end. This can be done by invoking download multiple times using different yt-dlp args for rejecting videos that are not in the range.

The DownloadMappings class should ensure all files exist for each ID. If one or more files do not exist for an ID, remove it from its internal representation.

https://github.com/jmbannon/ytdl-sub/blob/master/src/ytdl_sub/ytdl_additions/enhanced_download_archive.py#L88

By removing it, the id is not written to the YTDL download-archive file. Therefore, YTDL will redownload it.

Add total tracks that appear in a soundcloud album. For singles, always set this to 1

Both in the README and readthedocs

example_preset:
  youtube:
    download_strategy: "multi_channels"
    multi:
      "First channel": https://youtube.com/...
      "Second channel": https://.....
  ...

should be equivalent to

example_preset:
  youtube:
    download_strategy: "multi_channels_from_playlist"
    playlist: https://youtube.com/...
  ...

where each unique channel in the playlist is an additional channel entry

To determine permissions for every file created in ytdl-sub

The wheel does not use the error-handling in main.py

Currently all logging is done with prints and with # TODO comments above them. We should instead use a proper logger, but also make sure it prints pretty. It should look similar to YTDL's logger:

[post-process] Moving file to ...
[plugin:music_tags] Tag 'track_number' not supported. Available tags: ....

Ideally should be automated

To use "22" from a "2022" upload year

From my quick manual inspection it did not look right

It would be nice for the config to offer something like

configuration:
  dl_aliases:
    mv: "--preset yt_music_video"
    v: "--youtube.video_url"
    p: "--youtube.playlist_url"
    artist: "--overrides.artist"

to convert this long command

ytdl-sub dl --preset yt_music_video --youtube.video_id https://youtube.com/... --overrides.artist "Tom Petty"

to this

ytdl-sub dl --mv --v https://youtube.com/... --artist 'Tom Petty'

Would need to update all path logic for windows-compatible file paths and make ytdl-sub an executable. Hoping a windows scripting god could drive and own this.

The current workaround is use WSL.

Getting your YT cookie file is confusing and will surely be asked when users see they cannot download age-restricted videos. Have a section for this in a FAQ

I've let ytdl-sub run for some channels but it errors out when a specific channel does not have a video within the requested timerange. In config.yaml the default 2 weeks were configured:

configuration:
  working_directory: '/media/youtube/.cache'

presets:
  yt_channel_as_tv:
    # YouTube channels are our source/download strategy
    # Use the channel avatar and banner images for Kodi
    youtube:
      download_strategy: "channel"
      channel_avatar_path: "poster.jpg"
      channel_banner_path: "fanart.jpg"
      after: "today-2weeks"

Normally the archive file is created when processing a channel.

Log output:

[ytdl-sub:yt-dlp] [download] 2022-04-06 upload date is not in range 2022-06-16 - 9999-12-31
[ytdl-sub:downloader] RejectedVideoReached, stopping additional downloads
[ytdl-sub] An uncaught error occurred:
Traceback (most recent call last):
  File "/ytdl-sub/src/ytdl_sub/main.py", line 95, in main
    _main()
  File "/ytdl-sub/src/ytdl_sub/main.py", line 78, in _main
    _download_subscriptions_from_yaml_files(config=config, args=args)
  File "/ytdl-sub/src/ytdl_sub/main.py", line 33, in _download_subscriptions_from_yaml_files
    subscription.download()
  File "/ytdl-sub/src/ytdl_sub/subscriptions/subscription.py", line 262, in download
    with self._prepare_working_directory(), self._maintain_archive_file():
  File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
    next(self.gen)
  File "/ytdl-sub/src/ytdl_sub/subscriptions/subscription.py", line 234, in _maintain_archive_file
    self._enhanced_download_archive.save_download_mappings()
  File "/ytdl-sub/src/ytdl_sub/ytdl_additions/enhanced_download_archive.py", line 489, in save_download_mappings
    self._download_archive = DownloadArchive.from_file(self._archive_working_file_path)
  File "/ytdl-sub/src/ytdl_sub/ytdl_additions/enhanced_download_archive.py", line 100, in from_file
    with open(file_path, "r", encoding="utf8") as file:
FileNotFoundError: [Errno 2] No such file or directory: '/media/youtube/.cache/yt_tokyolensexplore/.ytdl-subscribe-yt_tokyolensexplore-download-archive.txt'

Anything shown as an example should be tested. Hit two birds with one stone by having ytdl-sub e2e tests use each one as its config.yaml, perform a download, then check file md5 hashes to see if they match the expected hash.

Would be ideal for sphinx to automatically scrape all Plugin classes and display their docstrings

The dream will be to create mobile apps or browser extensions to invoke downloads from the couch. Once the server and API is written, the short-term plan could be to self-host a swaggerhub io instance and invoke the rest api from the browser.

Create a flask server that is able to handle requests in a format like this:

curl -X POST -H "Content-type: application/json"                \
    -d '{                                                       \
        "soundcloud.username": "alis_on",                       \
        "overrides.artist": "A.L.I.S.O.N.",                     \
        "overrides.genre": "Synthwave;Electronic;Instrumental"  \
    }'                                                          \
    'localhost:8080/soundcloud_discography'

The endpoint being the preset, and the payload including any overrides as a single-level dict

This one is on me to do. Add hooks for

1. Building the package
2. Running the linter
3. Running unit tests
4. Running e2e tests
5. (stretch) Building the docker image

Set this field as the index in which the video appears in the playlist. For channels and single videos, set this to 1

For a concert like this https://www.youtube.com/playlist?list=PLyUVnCw_kLV0J2OjvADXZL-NrSghlEgDg
that is split up into multiple videos, it'd be nice to merge it into a single video to not break ambiance when watching it.

Should add a new download strategy that looks like:

my_preset:
  youtube:
    download_strategy: "merge_playlist"
    playlist_id: <id of playlist>

Youtube channel + playlist should include

ytdl_options:
  breakonexisting: True 
  breakonreject: True
output_options:
  maintain_download_archive: True  # global default False

Soundcloud discography should include

output_options:
  maintain_download_archive: True 

by default, and have this documented in the download strategy section.

This should help reduce YAML redundancy for fields that will most likely always be enabled for particular download strategies.

Can you build the container for arm64 as well? Then it can directly run on raspberry pi using docker.
I've tried building the container locally and needed some more packages (zlib-dev jpeg-dev python3-dev libwebp-dev) in order to install the Pillow requirements, but otherwise it seems to be running without issues.

You should be able to use docker buildx to create both amd64 and arm64 builds in a single step and push the manifest to the repo.

All download strategy source fields (youtube.channel_id, youtube.video_id, etc) should support partial and full URL links,
i.e.

# only youtube
youtube.com/channel/tHe_ID

# https youtube
https://youtube.com/channel/tHe_ID

# www youtube
www.youtube.com/channel/tHe_ID

# https + www youtube
https://www.youtube.com/channel/tHe_ID

# only the ID
tHe_ID

A unit test should be written to support each case

Some of the e2e tests flake. If they do, we should rerun those tests without rerunning the whole suite and see if they pass on the second (possibly third) time.

my_preset:
  regex_capture:
    title:
      fail_if_not_captured: True  # or False, if all captures fail, fallback to original value of title
      capture:  # List of regex captures to try
      - "(.+) - (.+) [\[\(].*"
      - "(.+) - (.+)"

This would regex capture "Artist" and "Cool Song" from "Artist - Cool Song (Official Video)" into new source variables title_capture_1 and title_capture_2