soraxas / echo360 Goto Github PK
View Code? Open in Web Editor NEWCommandline tool for automated downloads of echo360 videos hosted by university
Home Page: https://cs.tinyiu.com/echo360
License: MIT License
Commandline tool for automated downloads of echo360 videos hosted by university
Home Page: https://cs.tinyiu.com/echo360
License: MIT License
Hi,
I've just stumbled upon the fact that (at lest at my university) echo360 has an RTMP stream with much higher quality available (720p instead of the 360p I get using this downloader), which can be derived from the video source URL that is extracted here:
Line 83 in 8d19dae
On our instance, the URL looks like this:
http://[...]/echo/_definst_/1815/5/35518f02-d94c-451a-8b0f-b8c84bdb0db7/mp4:audio-video-streamable.m4v/playlist.m3u8
The RTMP stream is derived like so:
rtmp://[...]/echo/_definst_/1815/5/35518f02-d94c-451a-8b0f-b8c84bdb0db7/audio-video.flv
Using that stream, the video can be downloaded e.g with ffmpeg, which the downloader is using anyway. The downside is that, since it's RTMP, the download time is exactly the video run time. One can work around this a little bit by downloading all selected lectures simultaneously.
Do you think this is a feature worth adding?
I think what happened here is that source
failed, and so run.sh
ran $PYTHON -m pip install --upgrade pip
, which bricked my system pip install.
Maybe change:
source $VENV_NAME/bin/activate
to:
source $VENV_NAME/bin/activate || exit
(or equivalent)
I tried to download videos from Echo360 platform on a PC. First it brings up a browser window but does not allow me to manually enter SSO login webpage. Then I pre-login to my school's SSO website and repeat the procedure. Now it stops at the webpage "https://echo360.org/section/my_course_uuid/syllabus which returns "404 Not Found.
The log from the prompt is shown below.
> Echo360 Cloud platform detected
> This implies setup_credential, and using web_driver
>> Please login with your SSO details and type continue when logged in.
-----------------------------------------------------------------
DevTools listening on ws://127.0.0.1:12565/devtools/browser/262b3662-5b3a-4046-981b-9078acb096f9
>> After you finished logging into echo360 cloud, the window should be automatically redirected and continued. If it got stuck, please contact the author :)
>> I'm gonna assume you are responsible enough to had finished logged in by now ;)
>> Retrieving echo360 Course Info... [2912:4844:0901/120337.411:ERROR:device_event_log_impl.cc(208)] [12:03:37.411] Bluetooth: bluetooth_adapter_winrt.cc:1074 Getting Default Adapter failed.
Traceback (most recent call last):
File "echo360.py", line 7, in <module>
sys.exit(main())
File "C:\Users\username\Documents\GitHub\echo360\echo360\main.py", line 259, in main
downloader.download_all()
File "C:\Users\username\Documents\GitHub\echo360\echo360\downloader.py", line 189, in download_all
videos = self._course.get_videos().videos
File "C:\Users\username\Documents\GitHub\echo360\echo360\course.py", line 128, in get_videos
course_data_json = self._get_course_data()
File "C:\Users\username\Documents\GitHub\echo360\echo360\course.py", line 196, in _get_course_data
raise Exception("Error: Failed to get m3u8 info for EchoCourse!")
Exception: Error: Failed to get m3u8 info for EchoCourse!
I'm trying to login to e360's cloud via Chrome, and before I get the chance to input my credentials, I receive this error:
requests.exceptions.SSLError: HTTPSConnectionPool(host='echo360.org.au', port=443): Max retries exceeded with url: /section/bed1a3e7-918b-4e9d-98a3-c07effa1fb4d/syllabus (Caused by SSLError(SSLEOFError(8, u'EOF occurred in violation of protocol (_ssl.c:590)'),))
Logging into echo360 before executing the script doesn't work either. And when I try to use --firefox, it gives me this error:
IOError: [Errno socket error] [SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:590)
Thanks in advance.
Hi
Thanks for this software!
I'm trying to use it as a teacher at Uni. Nottingham in the UK which uses the cloud version.
The problem is that the login does not work, probably because
https://echo360.org.uk/section/82a37ff6-132c-4300-91dd-48eb0a18214d/syllabus
points to
https://login.echo360.org.uk
which points to
https://login.pingone.com/ (a centralized login).
But this prompt does not show, instead a 404 error is thrown.
When using an -existing- browser on
https://echo360.org.uk/section/82a37ff6-132c-4300-91dd-48eb0a18214d/syllabus
the login goes as expected via poingone.com, the 404 is resolved and syllabus has content.
Log attached.
Is there a way to download lecture videos using the link that your friend has given you (from the same institution) but you are not enrolled in that specific course...
e.g. https://echo360.net.au/section/a25a73c6-0499-4872-b89e-8cef66532062/home
Thank you! @@
I successfully used this a couple weeks ago, but now even after fresh install it logs in, collects list of lectures, then hangs after list is complete and command line reads:
Downloading "2020-06-25 - Lecture"
Downloading audio:
Hi, would it be possible to check if files exist before downloading, and skip those that already exist?
As it is, the program terminates after downloading any video that does not exist in the directory even if several videos were supposed to be downloaded.
THANKS!
They're moving domains. Accessing https://echo360.org.au/ currently redirects to a login page on echo360.net.au.
The downloader can still be used if you can manage to get logged in on echo360.org.au without hitting that redirect. (I managed to do this by clicking a link on my course's Blackboard page.)
But if you do hit the redirect, the downloader will fail, because it doesn't think it's really logged in.
I am unable to do anything, keep getting errors:
Any ideas if this can be overcome?
Can't Download "Audio ONLY" lectures, i.e. audio-recorded lectures without video.
Downloading such lectures will give the following error:
----------------------------------------------------------------------------------------
Skipping Lecture '2020-05-27 - Lecture 31 [Recap]' as it says it does not contain any video.
----------------------------------------------------------------------------------------
Would be nice to handle such errors and allow downloading audio lectures (even maybe into mp3 format)
Program fails to download and produces the following error after login credentials provided.
` Download will use "PhantomJS" webdriver from LOCAL executable <<<
Logging into "https://canvas.sydney.edu.au/ess/portal/section/956"... Credentials needed...
Unikey: Traceback (most recent call last):
File "usydEcho360.py", line 148, in
main()
File "usydEcho360.py", line 128, in main
downloader.download_all()
File "/home/username/echo_downloader/USYDecho360-master/USYDecho360/downloader.py", line 138, in download_all
self.login()
File "/home/username/echo_downloader/USYDecho360-master/USYDecho360/downloader.py", line 73, in login
self.loginWithCredentials()
File "/home/username/echo_downloader/USYDecho360-master/USYDecho360/downloader.py", line 105, in loginWithCredentials
user_name.clear()
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webelement.py", line 95, in clear
self._execute(Command.CLEAR_ELEMENT)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webelement.py", line 628, in _execute
return self._parent.execute(command, params)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 314, in execute
self.error_handler.check_response(response)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.InvalidElementStateException: Message: {"errorMessage":"Element is not currently interactable and may not be manipulated","request":{"headers":{"Accept":"application/json","Accept-Encoding":"identity","Connection":"close","Content-Length":"81","Content-Type":"application/json;charset=UTF-8","Host":"127.0.0.1:43767","User-Agent":"selenium/3.12.0 (python linux)"},"httpVersion":"1.1","method":"POST","post":"{"sessionId": "437210c0-59bd-11e8-8cf9-6d249373a31f", "id": ":wdc:1526553007974"}","url":"/clear","urlParsed":{"anchor":"","query":"","file":"clear","directory":"/","path":"/clear","relative":"/clear","port":"","host":"","password":"","user":"","userInfo":"","authority":"","protocol":"","source":"/clear","queryKey":{},"chunks":["clear"]},"urlOriginal":"/session/437210c0-59bd-11e8-8cf9-6d249373a31f/element/:wdc:1526553007974/clear"}}
Screenshot: available via screen
`
Device: 2011 Macbook Pro (8,1)
OS: 64bit Mac OS X 10.13.6 17G14042
Kernel: x86_64 Darwin 17.7.0
Intel Core i5-2435M @ 2.40GHz
Tested on QUT's Echo system. Here are the steps that I did:
Ran:
Encountered:
(echoenv) bash-5.1$ python3 echo360.py https://echo360.net.au/section/a6e99f1d-e120-44c7-817f-595642fc6c18/home
> Echo360 Cloud platform detected
> This implies setup_credential, and using web_driver
>> Please login with your SSO details and type continue when logged in.
-----------------------------------------------------------------
>> After you finished logging in, type 'continue' in the terminal.
> Type 'continue' and press [enter]
continue
>> I'm gonna assume you are responsible enough to had finished logged in by now ;)
>> Retrieving echo360 Course Info... 19/19 videos Done!
============================================================
Course: [[UNTITLED]]
Total videos to download: 19 out of 19
============================================================
------------------------------------------------------------
Downloading "2021-09-02 - Lecture 19 [CAB240_Week7_PartC]"
> Downloading audio:
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20.0M/20.0M [00:13<00:00, 1.48MiB/s]
> Downloading video:
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 12.1M/12.1M [00:05<00:00, 2.21MiB/s]
> Converting to mp4... Traceback (most recent call last):
File "/Volumes/HDD/echo360/echoenv/lib/python3.9/site-packages/ffmpy.py", line 93, in run
self.process = subprocess.Popen(
File "/usr/local/Cellar/[email protected]/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 951, in __init__
self._execute_child(args, executable, preexec_fn, close_fds,
File "/usr/local/Cellar/[email protected]/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 1821, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'ffmpeg'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Volumes/HDD/echo360/echo360.py", line 7, in <module>
sys.exit(main())
File "/Volumes/HDD/echo360/echo360/main.py", line 356, in main
downloader.download_all()
File "/Volumes/HDD/echo360/echo360/downloader.py", line 235, in download_all
if video.download(self._output_dir, filename):
File "/Volumes/HDD/echo360/echo360/videos.py", line 269, in download
result = self.download_single(session, single_url, output_dir, new_filename, pool_size)
File "/Volumes/HDD/echo360/echo360/videos.py", line 317, in download_single
self.combine_audio_video(
File "/Volumes/HDD/echo360/echo360/videos.py", line 359, in combine_audio_video
ff.run()
File "/Volumes/HDD/echo360/echoenv/lib/python3.9/site-packages/ffmpy.py", line 98, in run
raise FFExecutableNotFoundError(
ffmpy.FFExecutableNotFoundError: Executable 'ffmpeg' not found
(echoenv) bash-5.1$
Possible guesses:
Even though I downloaded the ffmpeg it may not be placed in the right directory. https://stackoverflow.com/a/60561710/14368392
Hello this is my first issue ever :)
I tried to download my course [URL] doing following:
tester@test:~$ ./run.sh https://echo360..........
./run.sh: 43: ./run.sh: source: not found
Failed to source virtual environment
Ich checked run.sh and found the line:
source $VENV_NAME/bin/activate || error_exit "Failed to source virtual environment"
So it tries to find "$VENV_NAME/bin/activate" but it seems to be not there.
Some more info about my shell:
tester@test:~$ echo $SHELL
/bin/bash
tester@test:~$ echo $0
bash
What can I do ?
Information
OS: Linux Mint 19 Tara
Desktop-Umgebung: Cinnamon
Hardware: Thinkpad T460
Kernel: 4.15.0-20-generic
Hi!
Thanks for posting the yt video. I have tried to follow it as best i can and when running the run.bat i came up with the following:
(_echo360venv) C:\Users\Kevin\Downloads\echo360-master\echo360-master>run.bat
Requirement already satisfied: windows-curses in c:\users\kevin\downloads\echo360-master\echo360-master_echo360venv\lib\site-packages (1.0)
Traceback (most recent call last):
File "echo360.py", line 16, in
from echo360.downloader import EchoDownloader
File "C:\Users\Kevin\Downloads\echo360-master\echo360-master\echo360\downloader.py", line 6, in
from echo360.hls_downloader import Downloader
File "C:\Users\Kevin\Downloads\echo360-master\echo360-master\echo360\hls_downloader.py", line 1, in
from gevent import monkey
ModuleNotFoundError: No module named 'gevent'
not sure what's happening here, any help would be much appreciated
Thanks
My university uses SSO with the domain having .ca
without .org|.net
.
How can I use SSO?
Hello, every time when I try to download the video the progress will always be stuck at a certain percentage. I tried many times, but they all come out with the same results. Btw, I really think this program is amazing. Thx.
Done!
Retrieving echo360 Course Info... 13/13 videos Done!
============================================================
Course: FINC6000 - Quantitative Finance
Total videos to download: 13 out of 13
============================================================
Downloading "FINC6000 - 2018-06-04 - Lecture 13"
Progress: [> ] 0.19% 2/1047
(_echo360venv) C:\Users\Ray Liang\Downloads\echo360-master>
The course I am taking has two video options in the "Sources" button, video 1 (showing the materials) and video 2 (showing the instructor). Both share the same audio so I can choose any or both of them to watch online.
As for the downloaded videos, It seems that only video 1 is used for downloading. In some of the lectures, however, video 1 got some problem and the materials have been switched to video 2. Is there any option to choose which video source to download?
Thanks
Hi, I haven't had any issues until I recently started a new class and I'm getting the following error "ERROR: Failed to find audio/video m3u8... skipping this one"
Any thoughts on how to fix this? Thank you!
Here is the debug result: https://pastebin.com/q2i04Ekm
OS: Debian GNU/Linux 9.9 x86_64
Python Version: 3.5.3
Chrome Version: Chromium 73
Neat tool you've got here. I've just tried this on Linux and it works fine with Chromium.
Only problem, was that I was getting TypeError: the JSON object must be str, not 'bytes'
, due to json.loads
trying to parse, json_str
which turns out to be bytes for me.
This can be fixed by changing line 198
in echo360/course.py
to either:
json_str = r.content.decode('utf-8')
or
json_str = r.text
The tool can't connect to a webpage with an invalid TLS certificate. I got a quick solution running for the chromium webdriver from this post.
Maybe this could be introduced as an option for every driver?
This is a fantastic piece of software! I had quite a difficult time getting logged into echo360 through the selenium browser and thought I would share my solution.
After entering my username into echo360, I was being redirected to the universities SSO and the redirection was causing problems. In fact, I was not able to navigate to any webpage besides google in the chrome instance due to the ssl error below :
This site can’t provide a secure connection googleweblight.com sent an invalid response. ERR_SSL_PROTOCOL_ERROR
I tried adding various options and capabilities to the webdriver Options
from selenium.webdriver.chrome.options import Options
but nothing worked.
What solved the issue was to change change the value of self._useragent
in the EchoDownloader
class in the file echo360/downloader.py
to my user agent which I obtained from the site https://www.whatsmyua.info/
Hopefully this is helpful to someone.
When I try to download the entire lecture for a given course, I receive the following errors
Traceback (most recent call last):
File "echo360.py", line 7, in <module>
sys.exit(main())
File "C:\Users\Super\Desktop\echo360-master (1)\echo360-master\echo360\main.py", line 365, in main
downloader.download_all()
File "C:\Users\Super\Desktop\echo360-master (1)\echo360-master\echo360\downloader.py", line 259, in download_all
if video.download(self._output_dir, filename):
File "C:\Users\Super\Desktop\echo360-master (1)\echo360-master\echo360\videos.py", line 291, in download
result = self.download_single(
File "C:\Users\Super\Desktop\echo360-master (1)\echo360-master\echo360\videos.py", line 332, in download_single
audio_file = self._download_url_to_dir(
File "C:\Users\Super\Desktop\echo360-master (1)\echo360-master\echo360\videos.py", line 167, in _download_url_to_dir
os.rename(os.path.join(echo360_downloader.result_file_name), result_full_path)
FileExistsError: [WinError 183] Cannot create a file when that file already exists: 'default_out_path\\[[UNTITLED]]\\s0q1_all.m4s' -> 'default_out_path\\[[UNTITLED]]\\2021-12-02 - Lecture 56 [CS 342-37162]_audio.m4s
Even after changing the output file to a different file, I still face the same issue.
The line that I run are :
python echo360.py URL --output FILENAME
Been tried to deleted the existing file and unzip all the file. Then followed the instruction but failed. It worked few weeks ago. Not sure what happened.
Please login with your SSO details and type continue when logged in.
-----------------------------------------------------------------
=================================================================
Binary file of chromedriver not found, will initiate a download process now...
>> Downloading chromedriver binary file for "win32"
100% [..........................................................................] 3378230 / 3378230
>> Extracting archive file "chromedriver_win32.zip"
Done!
=================================================================
Traceback (most recent call last):
File "C:\download\echo360-master\echo360-master\echo360.py", line 7, in <module>
sys.exit(main())
File "C:\download\echo360-master\echo360-master\echo360\main.py", line 320, in main
course_uuid = re.search(
AttributeError: 'NoneType' object has no attribute 'group'
>> Logging into "https://echo360.net.au/ess/portal/section/xxxx"... INFO: No need to login :)
Done!
>> Retrieving echo360 Course Info... selenium cannot find given elements
Exception: Message: {"errorMessage":"Unable to find element with tag name 'pre'","request":{"headers":{"Accept":"application/json","Accept-Encoding":"identity","Content-Length":"90","Content-Type":"application/json;charset=UTF-8","Host":"127.0.0.1:50355","User-Agent":"selenium/3.141.0 (python mac)"},"httpVersion":"1.1","method":"POST","post":"{\"using\": \"tag name\", \"sessionId\": \"xxxxxx\", \"value\": \"pre\"}","url":"/element","urlParsed":{"anchor":"","query":"","file":"element","directory":"/","path":"/element","relative":"/element","port":"","host":"","password":"","user":"","userInfo":"","authority":"","protocol":"","source":"/element","queryKey":{},"chunks":["element"]},"urlOriginal":"/session/xxxx/element"}}
Hi, I've recently encountered an issue with the downloader not being able to start either Chrome or Firefox. I'm using:
When running .\run.bat --chrome <url>
:
> Echo360 Cloud platform detected
> This implies setup_credential, and using web_driver
>> Please login with your SSO details and type continue when logged in.
-----------------------------------------------------------------
=================================================================
Binary file of chromedriver not found, will initiate a download process now...
>> Downloading chromedriver binary file for "win32"
100% [..........................................................................] 3378230 / 3378230
>> Extracting archive file "chromedriver_win32.zip"
Done!
=================================================================
Traceback (most recent call last):
File "echo360.py", line 7, in <module>
sys.exit(main())
File "C:\Users\Jon\Documents\Repositories\GitHub\Clones\echo360\echo360\main.py", line 337, in main
downloader = EchoDownloader(
File "C:\Users\Jon\Documents\Repositories\GitHub\Clones\echo360\echo360\downloader.py", line 87, in __init__
self._driver = webdriver.Chrome(**kwargs)
File "C:\Users\Jon\Documents\Repositories\GitHub\Clones\echo360\_echo360venv\lib\site-packages\selenium\webdriver\chrome\webdriver.py", line 70, in __init__
super(WebDriver, self).__init__(DesiredCapabilities.CHROME['browserName'], "goog",
File "C:\Users\Jon\Documents\Repositories\GitHub\Clones\echo360\_echo360venv\lib\site-packages\selenium\webdriver\chromium\webdriver.py", line 93, in __init__
RemoteWebDriver.__init__(
File "C:\Users\Jon\Documents\Repositories\GitHub\Clones\echo360\_echo360venv\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 268, in __init__
self.start_session(capabilities, browser_profile)
File "C:\Users\Jon\Documents\Repositories\GitHub\Clones\echo360\_echo360venv\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 359, in start_session
response = self.execute(Command.NEW_SESSION, parameters)
File "C:\Users\Jon\Documents\Repositories\GitHub\Clones\echo360\_echo360venv\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 424, in execute
self.error_handler.check_response(response)
File "C:\Users\Jon\Documents\Repositories\GitHub\Clones\echo360\_echo360venv\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 247, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unknown error: cannot find Chrome binary
(Driver info: chromedriver=2.38.552522 (437e6fbedfa8762dec75e2c5b3ddb86763dc9dcb),platform=Windows NT 10.0.19042 x86_64)
And for .\run.bat --firefox <url>
:
> Echo360 Cloud platform detected
> This implies setup_credential, and using web_driver
>> Please login with your SSO details and type continue when logged in.
-----------------------------------------------------------------
=================================================================
Binary file of geckodriver not found, will initiate a download process now...
>> Downloading geckodriver binary file for "win64"
100% [..........................................................................] 1532274 / 1532274
>> Extracting archive file "geckodriver-v0.26.0-win64.zip"
Done!
=================================================================
Traceback (most recent call last):
File "echo360.py", line 7, in <module>
sys.exit(main())
File "C:\Users\Jon\Documents\Repositories\GitHub\Clones\echo360\echo360\main.py", line 337, in main
downloader = EchoDownloader(
File "C:\Users\Jon\Documents\Repositories\GitHub\Clones\echo360\echo360\downloader.py", line 99, in __init__
self._driver = webdriver.Firefox(profile, **kwargs)
File "C:\Users\Jon\Documents\Repositories\GitHub\Clones\echo360\_echo360venv\lib\site-packages\selenium\webdriver\firefox\webdriver.py", line 174, in __init__
self.service.start()
File "C:\Users\Jon\Documents\Repositories\GitHub\Clones\echo360\_echo360venv\lib\site-packages\selenium\webdriver\common\service.py", line 98, in start
self.assert_process_still_running()
File "C:\Users\Jon\Documents\Repositories\GitHub\Clones\echo360\_echo360venv\lib\site-packages\selenium\webdriver\common\service.py", line 110, in assert_process_still_running
raise WebDriverException(
selenium.common.exceptions.WebDriverException: Message: Service C:\Users\Jon\Documents\Repositories\GitHub\Clones\echo360/bin/geckodriver.exe unexpectedly exited. Status code was: 64
I found that manually downloading the latest versions of chromedriver and geckodriver and placing them in the bin
folder fixes the two errors mentioned above.
Would it be worth updating the auto-downloaded chromedriver and geckodriver versions? After looking at the supported browser versions for each, it seems that geckodriver is safe to update since it supports a wide range of Firefox versions, but the newer chromedrivers only support one major version of Chrome. Maybe the script could download the latest versions instead of using a particular version?
I tried to download a video whose URL looks like this:https://echo360.org.uk/media/$(UUID)/public. The script opened chrome and received the credentials correctly. Then the script redirects the link to https://echo360.org.uk/section/$(UUID)/home, which implies a 404 page. I am wondering whether the problem can be solved by redirecting the script to the correct URL, or if the player of ”echo360.org.uk/media/$(UUID)/public“ uses a different decoding method. Thanks!
.\run.bat https://echo360.org/sectio
n/5540db4e-4fab-4035-afb2-2ba9b93e6d06/home
=================================================================
Binary file of phantomjs not found, will initiate a download process now...
>> Downloading phantomjs binary file for "windows"
100% [..................................................................] 18193653 / 18193653
>> Extracting archive file "phantomjs-2.1.1-windows.zip"
Done!
=================================================================
>>> Download will use "PhantomJS" webdriver from LOCAL executable <<<
>> Logging into "https://echo360.org/ess/portal/section/home"... INFO: No need to login :)
Done!
>> Retrieving echo360 Course Info... selenium cannot find given elements
Exception: Message: {"errorMessage":"Unable to find element with tag name 'pre'","request":{"headers":{"Accept":"application/json","Accept-Encoding":"identity","Content-Length":"90","Content-Type":"application/json;charset=UTF-8","Host":"127.0.0.1:10290","User-Agent":"selenium/3.141.0 (python windows)"},"httpVersion":"1.1","method":"POST","post":"{\"using\": \"tag name\", \"value\": \"pre\", \"sessionId\": \"7b3317a0-fd7f-11e8-bf02-59f9a5dac269\"}","url":"/element","urlParsed":{"anchor":"","query":"","file":"element","directory":"/","path":"/element","relative":"/element","port":"","host":"","password":"","user":"","userInfo":"","authority":"","protocol":"","source":"/element","queryKey":{},"chunks":["element"]},"urlOriginal":"/session/7b3317a0-fd7f-11e8-bf02-59f9a5dac269/element"}}
Screenshot: available via screen
Any help is appreciated, thanks! Following the url I give does take me into the video files (I have already logged in).
the link no longer works, is there any ways to get access to lecture recordings??
Thank you
Hi,
I really hope you can help with this!
I have setup everything as directed and everything was running smoothly till the script asks for login details.
I have tried every permutation (including just un, domain\un, un@ etc... but with no luck.
I also tried using the chrome driver as a last resort but still no good.
From what i can see i think there may be some confusion over the userrname and how the standard LMS SSO login passes the authentication details to the LCS for authentication but i could be way off...
Im downloading the files from here at UWA
Im omitting some of the details but here is where im stuck...
(_echo360venv) C:\Users\user\Downloads\echo360-master\echo360-master\echo360-master>C:\Users\user\Downloads\echo360-master\echo360-master\echo360-master\run.bat https://prod.lcs.uwa.edu.au/ess/portal/section/XXXXXX
>>> Download will use "PhantomJS" webdriver from LOCAL executable <<<
>> Logging into "https://prod.lcs.uwa.edu.au/ess/portal/section/XXXXXX"... Credentials needed...
Unikey: USERNAME
Passowrd for USERNAME:
Failed!
> Failed to login, is your username/password correct...?
(_echo360venv) C:\Users\user\Downloads\echo360-master\echo360-master\echo360-master>C:\Users\user\Downloads\echo360-master\echo360-master\echo360-master\run.bat https://prod.lcs.uwa.edu.au/ess/portal/section/XXXXXX --chrome
=================================================================
Binary file of chromedriver not found, will initiate a download process now...
>> Downloading chromedriver binary file for "win32"
100% [..........................................................................] 3378230 / 3378230
>> Extracting archive file "chromedriver_win32.zip"
Done!
=================================================================
[0904/104906.551:ERROR:gpu_process_transport_factory.cc(1016)] Lost UI shared context.
DevTools listening on ws://127.0.0.1:12595/devtools/browser/9e62cf84-77f3-4380-a3d0-fb14ad8a88ba
>>> Download will use "ChromeDriver" webdriver from LOCAL executable <<<
>> Logging into "https://prod.lcs.uwa.edu.au/ess/portal/section/XXXXXX"... Credentials needed...
Unikey: USERNAME
Passowrd for USERNAME:
Failed!
> Failed to login, is your username/password correct...?
Please, please, please help me!!!
Downloads are successful but course names are always untitled. I am downloading lectures from UniMelb's echo360. Not really a big deal but it'd be nice if we have the option to name the course if it's too difficult to scrape the right course name
=================================================================
Course: [[UNTITLED]]
=================================================================
Successfully downloaded:
2021-07-26 - Lecture 1 [Lecture 1 - Introduction]
2021-07-27 - Lecture 2 [Lecture 2 - A flavor of Haskell]
=================================================================
Canada Echo360 doesn't have a .org
or a .net
.
I already have ffmpeg installed and verified that it was working.
I've tried using auto-downloaded chrome/firefox/phantomjs and I've even tried using the updated chromedriver.
I've tried changing the .ca
to .org.ca
and it still doesn't work.
The login window was finally able to open when I ran it with .org
with the updated chromedriver.
.\echo360-windows.exe https://echo360.org/section/0dffe11b-3cbe-45e0-b66a-b0766e3cd264/home --chrome
> Echo360 Cloud platform detected
> This implies setup_credential, and using web_driver
>> Please login with your SSO details and type continue when logged in.
-----------------------------------------------------------------
DevTools listening on ws://127.0.0.1:58991/devtools/browser/fa06fa87-ecab-4071-b79e-cabca7ddaa5c
>> After you finished logging in, type 'continue' in the terminal.
> Type 'continue' and press [enter]
[5780:21460:0413/221317.270:ERROR:device_event_log_impl.cc(214)] [22:13:17.270] USB: usb_device_handle_win.cc:1049 Failed to read descriptor from node connection: A device attached to the system is not functioning. (0x1F)
[5780:21460:0413/221317.278:ERROR:device_event_log_impl.cc(214)] [22:13:17.278] USB: usb_device_handle_win.cc:1049 Failed to read descriptor from node connection: A device attached to the system is not functioning. (0x1F)
[5780:14140:0413/221404.231:ERROR:util.cc(126)] Can't create base directory: C:\Program Files\Google\GoogleUpdater
continue
>> I'm gonna assume you are responsible enough to had finished logged in by now ;)
>> Retrieving echo360 Course Info... Traceback (most recent call last):
File "entrypoint.py", line 7, in <module>
File "echo360\main.py", line 365, in main
File "echo360\downloader.py", line 202, in download_all
File "echo360\course.py", line 139, in get_videos
File "echo360\course.py", line 216, in _get_course_data
File "json\__init__.py", line 348, in loads
File "json\decoder.py", line 337, in decode
File "json\decoder.py", line 355, in raw_decode
json.decoder.JSONDecodeError: Expecting value: line 2 column 1 (char 1)
[15864] Failed to execute script entrypoint
Sadly, it still didn't download in the end.
Here's the error when I run it through the updated chromedriver using .org instead of .ca:
echo360Downloader1.log
All the dependencies are installed and I'm running Python 3.10.4 on my Window 10 computer.
Here's what happens if I just run it with .ca
as it is supposed to:
.\echo360-windows.exe https://echo360.ca/section/0dffe11b-3cbe-45e0-b66a-b0766e3cd264/home --chrome
DevTools listening on ws://127.0.0.1:60330/devtools/browser/2069cc93-a0e4-45c9-98f2-b2e046c0ed3c
>> Logging into "https://echo360.ca/ess/portal/section/home"... INFO: No need to login :)
Done!
>> Retrieving echo360 Course Info... selenium cannot find given elements
Exception: Message: no such element: Unable to locate element: {"method":"css selector","selector":"pre"}
(Session info: headless chrome=100.0.4896.88)
Stacktrace:
Backtrace:
Ordinal0 [0x00C87413+2389011]
Ordinal0 [0x00C19F61+1941345]
Ordinal0 [0x00B0C658+837208]
Ordinal0 [0x00B391DD+1020381]
Ordinal0 [0x00B3949B+1021083]
Ordinal0 [0x00B66032+1204274]
Ordinal0 [0x00B54194+1130900]
Ordinal0 [0x00B64302+1196802]
Ordinal0 [0x00B53F66+1130342]
Ordinal0 [0x00B2E546+976198]
Ordinal0 [0x00B2F456+980054]
GetHandleVerifier [0x00E39632+1727522]
GetHandleVerifier [0x00EEBA4D+2457661]
GetHandleVerifier [0x00D1EB81+569713]
GetHandleVerifier [0x00D1DD76+566118]
Ordinal0 [0x00C20B2B+1968939]
Ordinal0 [0x00C25988+1989000]
Ordinal0 [0x00C25A75+1989237]
Ordinal0 [0x00C2ECB1+2026673]
BaseThreadInitThunk [0x7676FA29+25]
RtlGetAppContainerNamedObjectPath [0x77257A7E+286]
RtlGetAppContainerNamedObjectPath [0x77257A4E+238]
Hi
I am trying to use your program to download some lectures from echo360.org.uk site. I get these errors:
./run.sh https://echo360.org.uk/section/571de44e-ecab-4a70-8ea5-e8833577dcee/home
=================================================================
Binary file of phantomjs not found, will initiate a download process now...
>> Downloading phantomjs binary file for "macosx"
100% [........................................................................] 17148816 / 17148816
>> Extracting archive file "phantomjs-2.1.1-macosx.zip"
Done!
=================================================================
>>> Download will use "PhantomJS" webdriver from LOCAL executable <<<
>> Logging into "https://echo360.org.uk/ess/portal/section/home"... INFO: No need to login :)
Done!
>> Retrieving echo360 Course Info... selenium cannot find given elements
Exception: Message: {"errorMessage":"Unable to find element with tag name 'pre'","request":{"headers":{"Accept":"application/json","Accept-Encoding":"identity","Content-Length":"90","Content-Type":"application/json;charset=UTF-8","Host":"127.0.0.1:56131","User-Agent":"selenium/3.141.0 (python mac)"},"httpVersion":"1.1","method":"POST","post":"{\"using\": \"tag name\", \"sessionId\": \"951cc890-9901-11ea-82fb-8b7ddc49c111\", \"value\": \"pre\"}","url":"/element","urlParsed":{"anchor":"","query":"","file":"element","directory":"/","path":"/element","relative":"/element","port":"","host":"","password":"","user":"","userInfo":"","authority":"","protocol":"","source":"/element","queryKey":{},"chunks":["element"]},"urlOriginal":"/session/951cc890-9901-11ea-82fb-8b7ddc49c111/element"}}
Screenshot: available via screen
It says I don't need a password but I'm pretty sure I do. When I run with chromedriver I get this error:
./run.sh https://echo360.org.uk/section/571de44e-ecab-4a70-8ea5-e8833577dcee/home --chrome --setup-credentials
>>> Download will use "ChromeDriver" webdriver from LOCAL executable <<<
> Type 'continue' and press [enter]
continue
>> Logging into "https://echo360.org.uk/ess/portal/section/home"... INFO: No need to login :)
Done!
>> Retrieving echo360 Course Info... selenium cannot find given elements
Exception: Message: no such element: Unable to locate element: {"method":"tag name","selector":"pre"}
(Session info: chrome=81.0.4044.138)
(Driver info: chromedriver=2.38.552518 (183d19265345f54ce39cbb94cf81ba5f15905011),platform=Mac OS X 10.15.4 x86_64)
When logged in the page looks like this:
And when logged out the page looks like this:
I can upload page source code as well if you need.
Any help would be appreciated. I have a little bit of programming experience. So if you could point me to which part of the code needs modifying I can give it a go
Hey,
I just stumbled across this tool, really would like to get it working again for USYD. It looks like they no longer host echo360 themselves and so the URL's are different (echo360 cloud?).
https://echo360.org.au/section/be03853e-9c56-4080-8386-eb1384336508/home
From what I can see, I think this is the new URL format I've tried going to that directly but it asks me for a username/password of which my unikey doesn't work so I'm also assuming they're now using some kind of SSO.
Hi,
After logging in, and typing continue I get the following error:
File "/home/user/Downloads/echo360-master/echo360.py", line 7, in <module>
sys.exit(main())
File "/home/user/Downloads/echo360-master/echo360/main.py", line 355, in main
downloader._driver.set_window_size(0, 0)
File "/home/user/Downloads/echo360-master/_echo360venv/lib/python3.9/site-packages/selenium/webdriver/remote/webdriver.py", line 1376, in set_window_size
self.set_window_rect(width=int(width), height=int(height))
File "/home/user/Downloads/echo360-master/_echo360venv/lib/python3.9/site-packages/selenium/webdriver/remote/webdriver.py", line 1458, in set_window_rect
raise InvalidArgumentException("x and y or height and width need values")
selenium.common.exceptions.InvalidArgumentException: Message: x and y or height and width need values
I have also tried firefox but that crashes straight away ;(
Running Arch with chromium
The input I used was:
.\run.bat --setup-credential --after-date "2022-03-15" --before-date "2022-05-01" [link]
And I ran it on a windows laptop using both firefox and chrome. The first issue occurs after setting up the credentials where the driver would start loading lectures from the earliest date and wouldn't stop until it loaded every single lecture. The second issue occurs after all lectures have been loaded where the program wouldn't download any videos at all. Here's the output if that matters:
Retrieving echo360 Course Info... 5/5 videos Done!
============================================================
Course: [[UNTITLED]]
Total videos to download: 0 out of 5
============================================================
=================================================================
Course: [[UNTITLED]]
=================================================================
Successfully downloaded:
=================================================================
When running it on chrome it'd return comments that didn't show up when running it on firefox:
Type 'continue' and press [enter]
[16212:11272:0417/201605.122:ERROR:device_event_log_impl.cc(214)] [20:16:05.122] USB: usb_device_handle_win.cc:1049 Failed to read descriptor from node connection: A device attached to the system is not functioning. (0x1F)
[16212:11272:0417/201605.125:ERROR:device_event_log_impl.cc(214)] [20:16:05.125] USB: usb_device_handle_win.cc:1049 Failed to read descriptor from node connection: A device attached to the system is not functioning. (0x1F)
[16212:11272:0417/201605.125:ERROR:device_event_log_impl.cc(214)] [20:16:05.125] USB: usb_device_handle_win.cc:1049 Failed to read descriptor from node connection: A device attached to the system is not functioning. (0x1F)
continue
Retrieving echo360 Course Info... 1/5 videos [16212:25160:0417/201658.028:ERROR:util.cc(126)] Can't create base directory: C:\Program Files\Google\GoogleUpdater
[16520:23840:0417/201757.958:ERROR:gpu_init.cc(446)] Passthrough is not supported, GL is disabled, ANGLE is
Edit: I just tried this on my linux PC using firefox and it also had the same problems. I haven't tried it chrome on my linux PC because I'm too lazy to install it.
Hi,
I was able to make the log-in work, but I'm getting this error message when I'm trying to download. Any idea what went wrong?
Traceback (most recent call last):
File "echo360.py", line 7, in
sys.exit(main())
File "/Users/DarkKnight/Downloads/echo360-master/echo360/main.py", line 259, in main
downloader.download_all()
File "/Users/DarkKnight/Downloads/echo360-master/echo360/downloader.py", line 235, in download_all
if video.download(self._output_dir, filename):
File "/Users/DarkKnight/Downloads/echo360-master/echo360/videos.py", line 260, in download
audio_file = self._download_url_to_dir(urljoin(
File "/Users/DarkKnight/Downloads/echo360-master/echo360/videos.py", line 148, in _download_url_to_dir
echo360_downloader = Downloader(pool_size,
File "/Users/DarkKnight/Downloads/echo360-master/echo360/hls_downloader.py", line 43, in init
self.pool = Pool(pool_size)
NameError: name 'Pool' is not defined
I was trying to download video lectures from Echo360 Cloud and used ./run.sh https://echo360.org.uk/section/myCoursePage/home as suggested but still got an error after deleting the _echo360venv;
BernardoLee@DESKTOP-P7MOK6G MINGW64 ~/Desktop/echo360-master
$ ./run.sh https://echo360.org.uk/section/790d7b57-f22c-4b84-ae91-a84707c2e8c6/home
./run.sh: line 43: _echo360venv/bin/activate: No such file or directory
Failed to source virtual environment (try to delete '_echo360venv/' and re-run)
BernardoLee@DESKTOP-P7MOK6G MINGW64 ~/Desktop/echo360-master
$ ./run.sh https://echo360.org.uk/section/790d7b57-f22c-4b84-ae91-a84707c2e8c6/home
Checking pip is installed
Creating python virtual environment in _echo360venv/...
./run.sh: line 36: _echo360venv/bin/activate: No such file or directory
Failed to source virtual environment
I am actually quite new to python and command line, so probably got it wrong somewhere I am not aware of... Are there any possible ways to sort this out? I would really appreciate it :)
Hi, is it possible to download echo360.org without logging into the usual log-in interface (eg. for publicly hosted videos)? As an example, is it possible to parse(and download) videos from this portal (https://echo360.org/section/af040c4a-11d1-4bf5-856a-97485e0be2ed/home)? These videos do not need credential to log-in to echo360 cloud. Thanks for any help.
Hey man, appreciate your work. Unfortunately there seems to be an error with the encoding of unicode characters like ä,ö,ü when the script attempts to fetch course id and name.
>> Logging into "$someURL"... INFO: No need to login :)
Done!
>> Retrieving echo360 Course Info... 36/36 videos Done!
Traceback (most recent call last):
File ".\echo360.py", line 243, in <module>
main()
File ".\echo360.py", line 208, in main
downloader.download_all()
File "D:\Downloads\echo360-master\echo360-master\echo360\downloader.py", line 174, in download_all
self._course.course_id, self._course.course_name).strip())
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe4' in position 6: ordinal not in range(128)
When I use the command
run.bat URL
the following list of messages appears and the scripts stop running
'_echo360venv\Scripts\activate' is not recognized as an internal or external command,
operable program or batch file.
I have tried to delete the directory and re-run the scripts, but still facing the same issue.
Thanks for this very useful tool!
Is there an option to download a single video feed (or download feeds separately)?
Some of my lectures have two video feeds: Video 1: powerpoint/slides, Video 2: professor/class being filmed. When downloading these videos, sometimes only the 2nd feed gets downloaded. Would there be a way to select which source to download?
Thanks!
After line 5, I logged in to echo360, and then typed continue.
I also tried without the --manal flag, and could get nowhere.
I have python3 installed, but perhaps it is trying to use python 2.7?
matBo:echo360-master mat$ ./run.sh "https://echo360.org.au/lesson/1bd8f597-4d1e-4fae-a750-7af733c9b046" --firefox --output "~/Documents" --manual
> Echo360 Cloud platform detected
> This implies setup_credential, and using web_driver
>> Please login with your SSO details and type continue when logged in.
-----------------------------------------------------------------
>> After you finished logging in, type 'continue' in the terminal.
> Type 'continue' and press [enter]
continue
>> I'm gonna assume you are responsible enough to had finished logged in by now ;)
>> Retrieving echo360 Course Info... Traceback (most recent call last):
File "echo360.py", line 7, in <module>
sys.exit(main())
File "/.../echo360/main.py", line 349, in main
downloader.download_all()
File "/.../echo360/downloader.py", line 189, in download_all
videos = self._course.get_videos().videos
File "/.../echo360/echo360-master/echo360/course.py", line 129, in get_videos
course_data_json = self._get_course_data()
File "/.../echo360/echo360-master/echo360/course.py", line 197, in _get_course_data
raise Exception("Error: Failed to get m3u8 info for EchoCourse!")
Exception: Error: Failed to get m3u8 info for EchoCourse!
matBo:echo360-master mat$
I'd admit in advance that I know little about python crawler, but I can use your project and it indeed works well on echo360 cloud. After a few (about 10) successful downloads, however, it encountered [WinError 10054] "An Existing Connection was Forcibly Closed by the Remote Host." I searched online to find that if urlopen() method is used very frequently in a short period (i.e. links are opened much quicker than human can), then remote host will close the abnormal connection to avoid being attacked (by DDoS).
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 3 column 1 (char 2)
I get this error when trying to download. I'm very sorry, I'm still a little new the command line. I'm using python3 to try and run the script, I'm not sure if that's the issue? I can't seem to get any scripts to work with just python.
I have tried several times with and without specifying download folder. I both cases, the output folder is created and I can see the files created while being downloaded. Then they get immediately deleted after trying to merge audio with video.
I get the message that download is successful, but the output folder (default or specified) is empty.
So the course title is UNTITLED and every new audio and video being downloaded, it seems to clean the previous download and then starts downloading which causes only the last of video| audio being kept.
For example, the program downloads 1.m4a 2.m4a 3.m4a 4.m4a, at the end of the downloading, only 4.m4a would be kept.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.