Comments (13)
@refeed here's the symptoms that trigger upload failures for some youtube channels:
-
A copyrighted video or deleted video will kill the channel rip on download. We should be able to gracefully handle this better despite that we're using
--ignore-errors
and it not working. Even if you complete a channel rip AROUND the errored video, it will still kill future updates when when using--use-download-archive
. It will see the bad ID and kill the rip. -
Sometimes randomly the hand off between a completed rip and upload fails. This particular problem started when you switched us to OOP. So this is your bug that would be nice to see fixed before your vanish for another year.
In the interim I'm going up add a warning about ripping entire channels.
Edit: any future low quality / no output / hidden command issues about errors related to channel rips will be closed automatically like S3 timeout issues in the interest of sanity.
from tubeup.
@rudolphos Could you help us by giving an URL to reproduce this issue?
from tubeup.
It seems like a JSON error:
tubeup --use-download-archive --metadata=collection:weedtubers https://www.periscope.tv/CoureyPie/1vAGRRqZVDyGl
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2018.09.26
[debug] Python version 3.4.9 (CPython) - Linux-4.15.18-1-pve-x86_64-with-centos-7.5.1804-Core
[debug] exe versions: ffmpeg 2.8.15, ffprobe 2.8.15
[debug] Proxy map: {}
An exception just occured, if you found this exception isn't related with any of your connection problem, please report this issue to https://github.com/bibanon/tubeup/issues
Traceback (most recent call last):
File "/home/vxbinaca/.local/lib/python3.4/site-packages/tubeup/__main__.py", line 91, in main
use_download_archive):
File "/home/vxbinaca/.local/lib/python3.4/site-packages/tubeup/TubeUp.py", line 359, in archive_urls
identifier, meta = self.upload_ia(basename, custom_meta)
File "/home/vxbinaca/.local/lib/python3.4/site-packages/tubeup/TubeUp.py", line 273, in upload_ia
with open(json_metadata_filepath) as f:
FileNotFoundError: [Errno 2] No such file or directory: '/home/vxbinaca/.tubeup/downloads/Courey_-_Smoke_throwback_jams_hotboxtheinternet_420_stonergirls-1vAGRRqZVDyGl.info.json'
@rudolphos I'm going to make issue submission guidelines because of you. They'll require submission of the command you used with URL so we can test, and you'll have to use Github markup like I did in this post. Failure to either one of these will result in closure of the issue. We don't care that you back up right-wing content. Don't try to hide the URL from us we need it to fix the script.
from tubeup.
@vxbinaca I don't remember which channel it was, but it's possible it happens when some channels have videos put in limited state or geoblocked, I don't really know. That particular channel might have had a few videos in limited state+geoblocked.
Or maybe it's caused because youtubedl also outputs ids of videos that are already deleted/blocked due to copyright. So a video like this in a playlist / channel playlist could stop the program.
I just tried to archive this and got a similar error
sudo tubeup https://www.youtube.com/watch?v=4ekfBzeJBL8 --use-download-archive
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2018.09.18
[debug] Python version 3.6.6 (CPython) - Linux-4.4.0-17134-Microsoft-x86_64-with-Ubuntu-18.04-bionic
[debug] exe versions: ffmpeg 3.4.4-0ubuntu0.18.04.1, ffprobe 3.4.4-0ubuntu0.18.04.1
[debug] Proxy map: {}
Unable to extract video title
ERROR: "Tim Maia -..." The YouTube account associated with this video has been terminated due to multiple third-party notifications of copyright infringement.
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/youtube_dl/YoutubeDL.py", line 792, in extract_info
ie_result = ie.extract(url)
File "/usr/local/lib/python3.6/dist-packages/youtube_dl/extractor/common.py", line 507, in extract
ie_result = self._real_extract(url)
File "/usr/local/lib/python3.6/dist-packages/youtube_dl/extractor/youtube.py", line 1892, in _real_extract
raise ExtractorError(error_message, expected=True)
youtube_dl.utils.ExtractorError: "Tim Maia -..." The YouTube account associated with this video has been terminated due to multiple third-party notifications of copyright infringement.
An exception just occured, if you found this exception isn't related with any of your connection problem, please report this issue to https://github.com/bibanon/tubeup/issues
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/tubeup/__main__.py", line 91, in main
use_download_archive):
File "/usr/local/lib/python3.6/dist-packages/tubeup/TubeUp.py", line 356, in archive_urls
urls, proxy, ydl_username, ydl_password, use_download_archive)
File "/usr/local/lib/python3.6/dist-packages/tubeup/TubeUp.py", line 144, in get_resource_basenames
self.create_basenames_from_ydl_info_dict(ydl, info_dict)
File "/usr/local/lib/python3.6/dist-packages/tubeup/TubeUp.py", line 163, in create_basenames_from_ydl_info_dict
info_type = info_dict.get('_type', 'video')
AttributeError: 'NoneType' object has no attribute 'get'
from tubeup.
@rudolphos No, you can hit the Up key and look at the last commands you ran. Limited state videos still show when you dump a channels IDs, geoblocked throw up an error when you try to rip them but still show.
What you posted was a terminated channel that can't be ripped and is useless to us as an error still.
This is what I'm talking about:
tubeup --use-download-archive --metadata=collection:tubeup Z-nLNHzsAD8
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2018.10.05
[debug] Python version 3.4.9 (CPython) - Linux-4.15.18-1-pve-x86_64-with-centos-7.5.1804-Core
[debug] exe versions: ffmpeg 2.8.15, ffprobe 2.8.15
[debug] Proxy map: {}
video doesn't have subtitles
An exception just occured, if you found this exception isn't related with any of your connection problem, please report this issue to https://github.com/bibanon/tubeup/issues
Traceback (most recent call last):
File "/home/vxbinaca/.local/lib/python3.4/site-packages/tubeup/__main__.py", line 91, in main
use_download_archive):
File "/home/vxbinaca/.local/lib/python3.4/site-packages/tubeup/TubeUp.py", line 359, in archive_urls
identifier, meta = self.upload_ia(basename, custom_meta)
File "/home/vxbinaca/.local/lib/python3.4/site-packages/tubeup/TubeUp.py", line 273, in upload_ia
with open(json_metadata_filepath) as f:
FileNotFoundError: [Errno 2] No such file or directory: '/home/vxbinaca/.tubeup/downloads/le_monke_full_rap-Z-nLNHzsAD8.info.json'
Tubeup is wonky right now due to:
- File parsing errors because Tubeup needs to be gone through with a fine tooth comb.
- JSON errors that just make a video at random not upload and leaves the downloaded result in the .tubeup folder.
@rudolphos Do not comment on this issue further you're wasting our time with you purposely hiding URLs or when you do you giver us terminated channels. Just save it for next issue. Learn the rules. I assure you I run into issues before you do since I heavily use this script.
@refeed DM me your email adddress I want to add your Archive.org account to a collection I use to test Tubeup and keep things out of Community Video. I think you're testing one video at a time and I want you to start doing test rips of medium sized channels and comparing the results of whats on youtube to what was posted on IA. Theres a wonky bug that didn't exist before your re-base thats leaving videos in the tubeup folder and I want to nail this bug. I want you to start dumping into that channel so we don't piss off staff if you don't already have a collection of your own.
from tubeup.
About the .info.json
missing issue, I think it's an intermittent issue produced by youtube-dl, I still can't reproduce it until now. https://pastebin.com/DB69PNyS
We do can make tubeup
to re-run youtube-dl
until we get the .info.json
file which is crucial for ia metadata
from tubeup.
I've rerun it multiple times same result.
from tubeup.
from tubeup.
Yeah gimme a bit
from tubeup.
Any update? Also having this issue with https://www.youtube.com/user/sthetix360
from tubeup.
@ealgase use youtube-dl to output the channels videoIDs then have tubeup rip each video individually.
from tubeup.
OK.
from tubeup.
I believe this issue is fixed with the move to the new yt-dlp extractor. Please reopen if issues reoccurs with the latest version of tubeup. Thanks!
from tubeup.
Related Issues (20)
- Bug report: Rate limiting is not implemented HOT 1
- Bug report: Twitch chat at some point stopped being downloaded (RECHAT) HOT 8
- Limit video download resolution to Full HD HOT 1
- ERROR: Unable to extract uploader id HOT 5
- Proposal: Identify core/essential metadata and add upload safeties for missing MD HOT 4
- Bug report: Channels having YouTube shorts cause Tubeup to fail HOT 4
- Proposal: What to do about yt-dlps new nightly branch? HOT 4
- deleted HOT 1
- Bug report/feature request: Continue downloading other videos when one fails with a permanent error HOT 6
- Bug report: extremely slow downloads from youtube HOT 3
- Bug report: [native] nsig extraction failed HOT 4
- Possible NSIG fixes HOT 8
- Upgrade yt-dlp ASAP to at least 2023.07.06 HOT 5
- "Creator" field for Douyin needs update HOT 9
- Update internetarchive to 3.4.0/3.5.0? HOT 4
- Uploaded YT video thumbnails in .webp are not used for IA item tiles HOT 7
- PEP 668 compatability
- Add new release for 2023-08-10. HOT 2
- Bug report: Video impossible to upload when best quality stream is unavailable on the server-side HOT 11
- Bug report: Unable to archive Youtube video after premiere HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from tubeup.