Incorrect order and, sometimes, wrong season (year) about youtube-dl-agent.bundle HOT 11 OPEN

Hi all,

After some time trying I gave up but now I'm back to make this work because I think it's pointless to retrieve the data with another YouTube agent when I want an archive of files...

I dug into the logs and found a couple of errors that I fixed myself in regards of accessing JSON keys that may not be there, thus throwing an error. I will create a PR in case I manage to get everything working properly. That fix did not fix the ordering issue though.

I found some interesting behaviour tho: when I download the videos to the root folder ie /mnt/YouTube/vicesat [UC76hs2bdrebFYvdR-I8bwyw] Plex creates the Season 2020 and so on seasons with the episodes inside of them in the wrong order as showed above by another user and myself. But when I place the episodes in a subdirectory ie /mnt/YouTube/vicesat [UC76hs2bdrebFYvdR-I8bwyw]/test then Plex does not create year based seasons but normal ones like Season 01 and, of course, only Season 01 with all the vides in there BUT the order is the correct one!

So:

/mnt/YouTube/vicesat [UC76hs2bdrebFYvdR-I8bwyw] -> Season 2021 and so on -> Wrong order and wrong number of episodes 340 vs 336 real ones.
/mnt/YouTube/vicesat [UC76hs2bdrebFYvdR-I8bwyw]/test -> Season 01 only -> Correct order and correct number of episodes 336.

Does that give you guys any clue? To be honest the format that Plex uses for their agents is confusing to me and I cannot find any documentation to make it clearer to me to properly work on this.

I hope we can fix this guys. As a workaround having everything in one season and correctly ordered does not sound that bad, but I like it better with the year based seasons.

from youtube-dl-agent.bundle.

Hi, This plugin should set the "originally_available_at" as found in the ".info.json" file, this should make Plex sort them correctly (at least date wise).
But it does look to be doing something wrong in your case. What do the files say when they were created (or last modified)?

from youtube-dl-agent.bundle.

Hi, This plugin should set the "originally_available_at" as found in the ".info.json" file, this should make Plex sort them correctly (at least date wise).
But it does look to be doing something wrong in your case. What do the files say when they were created (or last modified)?

Thanks for your reply!

I've been greping all the .info.json files within different directories but the string originally_available_at is not present in any of the files... This is the grep I've used: grep "originally_available_at" *.info.json -o

I've tried that in the following channels: UCVpankR4HtoAVtYnFDUieYA, UCtHaxi4GTYDpJgMSGy7AeSw and UCqYPhGiB9tkShZorfgcL2lA. None of those channels have a single .info.json with the string originally_available_at. Has maybe YouTube updated their JSON structure? Is maybe youtube-dl not retrieving the data correctly? (see youtube-dl config above). I'm using version 2021.04.26 of youtube-dl.

To make sure my grep command was correct I ran grep "title" *.info.json -o and it works properly:

E001 - Laser Robot In Depth _ The Truth [HgMQTvDcn_s].info.json:title
E001 - Laser Robot In Depth _ The Truth [HgMQTvDcn_s].info.json:title
E001 - Laser Robot In Depth _ The Truth [HgMQTvDcn_s].info.json:title
E001 - Laser Robot In Depth _ The Truth [HgMQTvDcn_s].info.json:title
E002 - C# Beginner Tutorial _ Variables and Data Types [FID9sDXmCoQ].info.json:title
E002 - C# Beginner Tutorial _ Variables and Data Types [FID9sDXmCoQ].info.json:title
E002 - C# Beginner Tutorial _ Variables and Data Types [FID9sDXmCoQ].info.json:title
E002 - C# Beginner Tutorial _ Variables and Data Types [FID9sDXmCoQ].info.json:title
E003 - C# Beginner Tutorial _ if Statements and User Input [19C-78OKoCU].info.json:title
E003 - C# Beginner Tutorial _ if Statements and User Input [19C-78OKoCU].info.json:title
E003 - C# Beginner Tutorial _ if Statements and User Input [19C-78OKoCU].info.json:title
E003 - C# Beginner Tutorial _ if Statements and User Input [19C-78OKoCU].info.json:title
E004 - AMA _ Ft. The Annoying Machine _ Full Length [EpEXk3Gt8p0].info.json:title
E004 - AMA _ Ft. The Annoying Machine _ Full Length [EpEXk3Gt8p0].info.json:title
E004 - AMA _ Ft. The Annoying Machine _ Full Length [EpEXk3Gt8p0].info.json:title
E004 - AMA _ Ft. The Annoying Machine _ Full Length [EpEXk3Gt8p0].info.json:title
E005 - AMA Stream Highlights [OslFWEX2wTQ].info.json:title
E005 - AMA Stream Highlights [OslFWEX2wTQ].info.json:title
E005 - AMA Stream Highlights [OslFWEX2wTQ].info.json:title
E005 - AMA Stream Highlights [OslFWEX2wTQ].info.json:title
E008 - What Just Happened, 4.5k Subs in 24 Hours [wvYZ4DoYxMw].info.json:title
E008 - What Just Happened, 4.5k Subs in 24 Hours [wvYZ4DoYxMw].info.json:title
E008 - What Just Happened, 4.5k Subs in 24 Hours [wvYZ4DoYxMw].info.json:title
E008 - What Just Happened, 4.5k Subs in 24 Hours [wvYZ4DoYxMw].info.json:title
E009 - DIY Laser Turret _ Part 2 The Software [_P24em7Auq0].info.json:title
E009 - DIY Laser Turret _ Part 2 The Software [_P24em7Auq0].info.json:title
E009 - DIY Laser Turret _ Part 2 The Software [_P24em7Auq0].info.json:title
E009 - DIY Laser Turret _ Part 2 The Software [_P24em7Auq0].info.json:title
E010 - C# Tutorial _ Setting Up Visual Studio 2017 [l1yTfNIZSYU].info.json:title
E010 - C# Tutorial _ Setting Up Visual Studio 2017 [l1yTfNIZSYU].info.json:title
E010 - C# Tutorial _ Setting Up Visual Studio 2017 [l1yTfNIZSYU].info.json:title
E010 - C# Tutorial _ Setting Up Visual Studio 2017 [l1yTfNIZSYU].info.json:title
E011 - Avoiding Psychosis From Boredom (Vlog sort of) [1DkoicM0zcI].info.json:title
E011 - Avoiding Psychosis From Boredom (Vlog sort of) [1DkoicM0zcI].info.json:title
E011 - Avoiding Psychosis From Boredom (Vlog sort of) [1DkoicM0zcI].info.json:title
E011 - Avoiding Psychosis From Boredom (Vlog sort of) [1DkoicM0zcI].info.json:title
E012 - DIY Laser Turret _ Part 1 The Hardware [ZpdvnHVIeyI].info.json:title
E012 - DIY Laser Turret _ Part 1 The Hardware [ZpdvnHVIeyI].info.json:title
E012 - DIY Laser Turret _ Part 1 The Hardware [ZpdvnHVIeyI].info.json:title
E012 - DIY Laser Turret _ Part 1 The Hardware [ZpdvnHVIeyI].info.json:title
^C

What could it be?

Edit: I do see an upload_date field in the JSON. That's the only key that contains some sort of date. I guess I could edit the plugin to grab that variable, but first I'd like to know if there maybe is something wrong going on that's causing youtube-dl not to retrieve the originally_available_at key.

Edit 2: Sorry, I misunderstood your message. I realise now that originally_available_at is Plex's parameter, not youtube-dl. The date is being set by using upload_date indeed. Then I'm not sure why it's going crazy in my case... :/

from youtube-dl-agent.bundle.

Hi, yeah I might have said it a bit confusing, but the originally_available_at is indeed a Plex attribute. Youtube-DL should set the date the video file was created as the date it was uploaded to youtube. Maybe it isn't doing that in your case and that is confusing plex (?). Could you check what the dates are the video files were created (or last modified)?

Also if you don't mind, could you post an .info.json of one of the files that is getting stored incorrectly?

from youtube-dl-agent.bundle.

Hi,

Yes sure, this is the .info.json of two clearly wrongly sorted episodes from channel UCtHaxi4GTYDpJgMSGy7AeSw. As far as I know, youtube-dl is adding the correct upload_date.

Archive.zip

This is an example of how Plex has sorted those two episodes:

To verify the info you can check the videos on YouTube:

• Teaching a Robot Dog to Pee Beer -> https://www.youtube.com/watch?v=tqsy9Wtr1qE
• I Let a Twitter Bot Decorate My Room -> https://www.youtube.com/watch?v=hBP-NzOadL0

As you can see, the very latest two episodes of Season 2021 are E43 - I Let a Twitter Bot Decorate My Room and E42 - Teaching a Robot Dog to Pee Beer. That is wrong though.

If you look at the attached file E033 - Teaching a Robot Dog to Pee Beer [tqsy9Wtr1qE].info.json you'll see that this video's upload_date is 20210409 while the one from E043 - I Let a Twitter Bot Decorate My Room [hBP-NzOadL0].info.json is 20191023.

Things I see that go wrong:

1. youtube-dl is adding the wrong playlist_index to the video. Teaching a Robot Dog to Pee Beer should be E043 and not E033 as it's the very latest one.
2. Despite that, Plex is still putting E33 and E43 one after the other one, which makes no sense in terms of their playlist_index nor upload_date (see picture above).
3. Plex is adding video E33 within Season 2021 when that video's upload_date is 20191023.

And I'd say that's all, hopefully I'm not missing anything. It's quite a mess and I'm not sure what's going on...

If you need more info or testing let me know!

from youtube-dl-agent.bundle.

I don't really see what is going wrong, the only thing other thing we can check is what Plex thinks the "release" date is. Should be listed on the video page. If not you can click 'Edit' on a video.

As for 1. I do think I know why youtube-dl is giving the incorrect index. Pretty sure it is sorting by popularity and not by upload date. If you check the positions they are at they match exactly. I have had a similair issue myself. If you change the url to https://www.youtube.com/channel/UCtHaxi4GTYDpJgMSGy7AeSw/videos that should resolve the playlist index being incorrect.

As for 3. I think that is correct E033 is uploaded in 2021

from youtube-dl-agent.bundle.

Yeah me neither...

Here you can see screens from the entire Season 2021 on Plex:

As you can see, it makes no sense at all... The dates are properly set according to the upload_date and they match what I can see on YouTube, so that's good. It's Plex somehow ignoring that...

About the popularity order, I don't see the match you mention:

And as per your last comment, sorry, I misstyped. I wanted to say:
3. Plex is adding video E43 within Season 2021 when that video's upload_date is 20191023.

And to add to that, the third video of that season was uploaded in 2017! :(

This really is a strange thing... Could you please try downloading that channel with same config as me but lowest quality possible and without getting subs nor any other unnecessary files? The file channels.txt contains the line https://www.youtube.com/channel/UCtHaxi4GTYDpJgMSGy7AeSw.

The config would look like something like this:

-i
-a channels.txt
-o "/mnt/YouTube/%(uploader)s [%(channel_id)s]/E%(playlist_index)03d - %(title)s [%(id)s].%(ext)s"
--playlist-reverse
-f worstvideo/worseaudio

# Archive Settings
--download-archive archive.txt

# Get metadata
--add-metadata
--write-description
--write-info-json
--write-thumbnail

# Debug
-v

I'd really appreciate a tiny bit more debugging on your end to see if it's my Plex going nuts or somehow the agent is missing something with some sort of configuration...

Thank you very much for all the support so far!

from youtube-dl-agent.bundle.

Downloading from https://www.youtube.com/channel/UCtHaxi4GTYDpJgMSGy7AeSw/videos fixed youtube-dl playlist_index, at least there's some progress there. Thanks!

Also, Plex order changed, but it's still wrong. Now this is how Season 2021 looks like:

I've been playing with different values but I can't seem to find the root of the issue...

from youtube-dl-agent.bundle.

I believe that I'm seeing a similar issue when downloading from https://www.youtube.com/c/ToddintheShadows/videos. Each episode has the correct date, but the assigned episode numbers are out of sequence.

My download configuration is yt-dlp --dateafter "today-4months" 'https://www.youtube.com/c/ToddintheShadows/videos' --add-metadata --write-info-json --write-thumbnail -f 'bestvideo[ext=mp4]+bestaudio[ext=m4a]/mp4' -P "temp:./yt_scratch/" -o "./YouTube/%(uploader)s [%(uploader_id)s]/%(title)s[%(id)s].%(ext)s"

from youtube-dl-agent.bundle.

@KrLx1994roller, thanks for all the detail. I'm new to Plex and have been having this exact issue.

With this path /video/YouTube/Cosmic Kids Yoga [UC5uIZ2KOZZeQDQo_Gsi_qbQ]

I got random seasons:

With /video/YouTube/Cosmic Kids Yoga [UC5uIZ2KOZZeQDQo_Gsi_qbQ]/21-50 Minute Classes it drops the random seasons

I'll use -o "%(channel)s [%(channel_id)s]/%(playlist_title)s/%(title)s[%(id)s].%(ext)s" format for playlists and -o "%(channel)s [%(channel_id)s]/channel/%(title)s[%(id)s].%(ext)s" for channels to get the behavior I want.

from youtube-dl-agent.bundle.

@Vye glad my research helped!

Unfortunately for my of usage that's not a valid workaround. I'd still like to separate in year seasons but properly ordered unlike how it does it now.

from youtube-dl-agent.bundle.