mk-fg / fgtk Goto Github PK
View Code? Open in Web Editor NEWA set of a misc tools to work with files and processes
A set of a misc tools to work with files and processes
When the --ytdl-output-format
option is used to specify a location on a different drive to the temporary files, the file_dst_path.rename
call fails with this error:
Traceback (most recent call last):
File "C:\Users\USER\Documents\GitHub\fgtk\desktop\media\twitch_vod_fetch", line 900, in <module>
if __name__ == '__main__': sys.exit(main())
File "C:\Users\USER\Documents\GitHub\fgtk\desktop\media\twitch_vod_fetch", line 896, in main
loop, conf, vod_queue, list_formats_only=opts.ytdl_list_formats ))
File "C:\Program Files\Python36\lib\asyncio\base_events.py", line 466, in run_until_complete
return future.result()
File "C:\Users\USER\Documents\GitHub\fgtk\desktop\media\twitch_vod_fetch", line 273, in vod_fetch
try: await fetch(url, prefix, info_suffix=info_suffix)
File "C:\Users\USER\Documents\GitHub\fgtk\desktop\media\twitch_vod_fetch", line 189, in _wrapper
return await func(self, ctx, *args, **kws)
File "C:\Users\USER\Documents\GitHub\fgtk\desktop\media\twitch_vod_fetch", line 766, in get
file_dst_path.rename(file_dst_done)
File "C:\Program Files\Python36\lib\pathlib.py", line 1292, in rename
self._accessor.rename(self, target)
File "C:\Program Files\Python36\lib\pathlib.py", line 396, in wrapped
return strfunc(str(pathobjA), str(pathobjB), *args)
OSError: [WinError 17] The system cannot move the file to a different disk drive: '..\\download_temp\\Dave.mp4' ->
'C:\\Users\\USER\\Downloads\\LivestreamDownloads\\KirinDave\\20170529 - X of 2 - GAME - Chill with Minecraft & Dave
For A Bit - v148136907.mp4'
shutil.move
can be used to move a file to another location, including across drives.
vfat_shuffler crashes when listing files at entry ".", which is the current folder:
cweiske:~> sudo ./bin/vfat_shuffler.py /media/cweiske --list
[sudo] Passwort für cweiske:
Traceback (most recent call last):
File "./bin/vfat_shuffler.py", line 667, in <module>
if __name__ == '__main__': sys.exit(main())
File "./bin/vfat_shuffler.py", line 647, in main
paths = [op_navigate(fs, path)]
File "./bin/vfat_shuffler.py", line 428, in op_navigate
for k in filter(None, slugs): path = path[k]
File "./bin/vfat_shuffler.py", line 263, in __getitem__
raise KeyError(name)
KeyError: '.'
Duplicate of #7, please ignore.
This is a request for an enhancement.
I wonder if you could add an option similar to --public
, perhaps call it --private
, to produce the OpenSSL private key (whether in base64 or hex form) for convenience.
Notice that I'm making a distinction here between the private key for OpenSSH, which OpenSSL would call 'seed', and OpenSSL's own key they call 'private', derived from seed.
The reason is that right now it is convenient to isolate a private key to inject later into other OpenSSH key files, but injecting the said key into a private key file generated by OpenSSL requires deriving the private key from the hash, and it is not convenient to do by hand, or I don't know a ready-made tool to do it.
The closest I know is NewKeyFromSeed() in this Go implementation.
The RFC 8032 has an Appendix A with a detailed Python 3.2 implementation but that only implements sign, not NewKeyFromSeed (even though within sign it might get calculated).
If we could output the private key with --private
, the next step could be that -d --private
inserts the key into a OpenSSL-generated private key file instead of OpenSSH.
That'd help bridging the gap between OpenSSH and OpenSSL which is not always easy to cross.
I am trying to find or write a command line program for Windows that will sort
FAT32 drives
http://softwarerecs.stackexchange.com/q/19575
Is this possible using vfat_shuffler
as a starting point? I could use a push in the
right direction.
In the same vain as #7, the script fails when attempting to move the output file to its final destination when the destination directory does not yet exist. Understandably, you may not expect to be given a non-existing directory before you implemented --ytdl-output-format
, but now it may be desirable, seeing as said option can be used to create directories dynamically, or at least youtube-dl can be used in that way. For example, here is what I just tried:
$ twitch_vod_fetch --aria2c-opts "max-concurrent-downloads=2 lowest-speed-limit=10K rpc-listen-all" -o '%(uploader)s/2018-10-10 %(title)s.ts' https://www.twitch.tv/videos/321031441 -s 9:57 -l 19:10
2018-12-08 04:01:14 :: INFO :: --- Downloading VoD vod-001.mp4 (final-name='Nico/2018-10-10 sister nico - happy hallowmeme - !goal !sub.ts', url=https://www.twitch.tv/videos/321031441)
2018-12-08 04:01:27 :: INFO ::
------ Started 115 downloads, last gid: 000115 ------
Download Results:
gid |stat|avg speed |path/URI
======+====+===========+=======================================================
Traceback (most recent call last):
File "/home/dan/.local/apps/miniconda3/lib/python3.7/shutil.py", line 557, in move
os.rename(src, real_dst)
FileNotFoundError: [Errno 2] No such file or directory: 'vod-001.mp4' -> 'Nico/2018-10-10 sister nico - happy hallowmeme - !goal !sub.ts'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/dan/bin/vendor/twitch_vod_fetch", line 930, in <module>
if __name__ == '__main__': sys.exit(main())
File "/home/dan/bin/vendor/twitch_vod_fetch", line 926, in main
loop, conf, vod_queue, list_formats_only=opts.ytdl_list_formats ))
File "/home/dan/.local/apps/miniconda3/lib/python3.7/asyncio/base_events.py", line 573, in run_until_complete
return future.result()
File "/home/dan/bin/vendor/twitch_vod_fetch", line 271, in vod_fetch
try: await fetch(url, prefix, info_suffix=info_suffix)
File "/home/dan/bin/vendor/twitch_vod_fetch", line 189, in _wrapper
return await func(self, ctx, *args, **kws)
File "/home/dan/bin/vendor/twitch_vod_fetch", line 783, in get
shutil.move(file_dst_path, file_dst_done, copy_function=shutil.copy)
File "/home/dan/.local/apps/miniconda3/lib/python3.7/shutil.py", line 571, in move
copy_function(src, real_dst)
File "/home/dan/.local/apps/miniconda3/lib/python3.7/shutil.py", line 241, in copy
copyfile(src, dst, follow_symlinks=follow_symlinks)
File "/home/dan/.local/apps/miniconda3/lib/python3.7/shutil.py", line 121, in copyfile
with open(dst, 'wb') as fdst:
FileNotFoundError: [Errno 2] No such file or directory: 'Nico/2018-10-10 sister nico - happy hallowmeme - !goal !sub.ts'
Command exited with non-zero status 1
After this, I created the directory manually and re-ran the command, and it finished normally.
Granted, this is a minor inconvenience for my use case, as I am only creating a directory for each channel, and I understand if this feature would be out of scope for this script. Regardless, thank you for the script; it does work very nicely. 👍
Using vfat_shuffler to list files of a mounted directory works fine now (#20):
$ sudo ./bin/vfat_shuffler --umount /media/cweiske/CWEISKEOUYA/
[ 0] {name}.{ext}
[ 43075] 01 - Dont Tell Me That Its Over.ogg
[ 43419] 01 - Ich bin hier.ogg
But passing a device file does not work:
$ sudo ./bin/vfat_shuffler --umount /dev/sdb1
usage: vfat_shuffler [-h] [-l] [-s] [-o] [--rename template] [-r] [-c] [-g type] [--rename-inside]
[-x] [-u] [-n] [-v] [-d]
dev_or_vfs_path [path]
vfat_shuffler: error: Failed to resolve path to non-root mountpoint: dev/sdb1
When I add a path after the device path, it works:
$ sudo ./bin/vfat_shuffler --umount /dev/sdb1 /
[ 0] {name}.{ext}
[ 43075] 01 - Dont Tell Me That Its Over.ogg
[ 43419] 01 - Ich bin hier.ogg
The help says that path is optional, but it does not seem to be optional when a device is given.
After searching the web for hours, i was not able to find a way to convert ssh-keyphrase back into the 'BEGIN OPENSSH PRIVATE KEY' format, could someone point me in the right direction?
The script often does not append the last batch of chunks to the destination file. This causes the VoD to be truncated up to 100 seconds early with the default config. Consider the following example.
For ease of demonstration, I changed line 27 in my local copy to file_append_batch = 2
. twitch_vod_fetch output truncated for legibility
* $ twitch_vod_fetch https://www.twitch.tv/videos/347546662 10 -l 10 -k
------ Started 1 downloads, last gid: 000001 ------
* $ twitch_vod_fetch https://www.twitch.tv/videos/347546662 40 -l 40 -k
------ Started 4 downloads, last gid: 000004 ------
* $ twitch_vod_fetch https://www.twitch.tv/videos/347546662 5chunks-by2 -l 50 -k
------ Started 5 downloads, last gid: 000005 ------
40.mp4 comes out fine (and for some reason, when I tried 20- and 30-second videos, they worked, too), but 5chunks-by2.mp4's last chunk is left in the tmp folder, and poor 10.mp4 never receives its single chunk!
* $ ls -l 5chunks-by2.mp4 40.mp4 10.mp4
-rw-r--r-- 1 dan dan 0 Jan 14 07:01 10.mp4
-rw-r--r-- 1 dan dan 15039812 Jan 14 07:04 40.mp4
-rw-r--r-- 1 dan dan 15039812 Jan 14 07:10 5chunks-by2.mp4
* $ ls 5chunks-by2.tmp/
5chunks-by2.0000050000000000.mp4.chunk 5chunks-by2.aria2c_conf 5chunks-by2.m3u8 5chunks-by2.m3u8.ua 5chunks-by2.m3u8.url 5chunks-by2.pos
* $ ls 40.tmp/
40.aria2c_conf 40.m3u8 40.m3u8.ua 40.m3u8.url 40.pos
* $ ls 10.tmp/
10.0000010000000000.mp4.chunk 10.aria2c_conf 10.m3u8 10.m3u8.ua 10.m3u8.url 10.pos
I've seen this issue on many (longer) videos since the batched chunk append was implemented; I just chose this short highlight as an example.
Thanks again for your work :)
When i base_64 decode the seed to store it as binary in the database it often defaults to nothing.
base64_decode($base64string),
$base64string,
results in
["","5_DbkBn6IhczM4r-rlIIzrkoLaiNYUchVviGnNqNuf0="]
About 30% of the time the first half stays empty, otherwise it works just as intended.
And when you paste the base64 string into other converters (e.g. https://cryptii.com/base64/binary) it does not interpret it as well.
What am i missing here?
Hi,
If I run the script, and issue a "show" command in bluetoothctl, I can see the PAN uuid.
From the android phone, I have trouble keeping a connection, but when the devices are paired I see the option to use for internet access, but it fails. Any ideas how I can make this work?
I want to run a PAN server on my Ubuntu box. I want to connect to the Ubuntu box via bluetooth from iPhone and Android, and then issue REST commands. Can I use this script to accomplish that?
When running vfat_shuffler.py with the latest construct installed, I get an error:
Traceback (most recent call last):
File "./bin/vfat_shuffler.py", line 3, in
from construct import (
ImportError: cannot import name 'Embedded' from 'construct' (/usr/local/lib/python3.8/dist-packages/construct/init.py)
The reason is that "Embedded" is not part of construct anymore since v2.10.
Workaround:
pip3 install construct==2.9.52
(Info taken from rytilahti/python-yeelightbt#36)
time python ssh-keyparse.py id_25519 HOSEWmGJtkrOBOuTGGOFXsMIsMqlnQTWAGcRIWXvRqQ=
real 0m0.111s
user 0m0.047s
sys 0m0.046s
time python ssh-keyparse.py --expand-seed HOSEWmGJtkrOBOuTGGOFXsMIsMqlnQTWAGcRIWXvRqQ=
real 0m1.385s
user 0m1.317s
sys 0m0.048s
time python ssh-keyparse.py storage/app/default -d HOSEWmGJtkrOBOuTGGOFXsMIsMqlnQTWAGcRIWXvRqQ=
real 0m1.438s
user 0m1.317s
sys 0m0.080s
time ssh-keygen -t ed25519 -N '' -f ./id_ed25519
Generating public/private ed25519 key pair.
Your identification has been saved in ./id_ed25519.
Your public key has been saved in ./id_ed25519.pub.
The key fingerprint is:
SHA256:PuypdWkGucbhnnIY8eaiphD/CCPeLZNqAZFWVpJhPP0 root@localhost
The key's randomart image is:
+--[ED25519 256]--+
| .o*=. |
|o.o+.. |
|.. . . |
|. .E . |
|o oS |
| + .=o+ . |
|= o . =X = |
|o=.*o +=oB |
|.o++=o.== |
+----[SHA256]-----+
real 0m0.011s
user 0m0.005s
sys 0m0.005s
I have compared some of the timings regarding this script, and i am curious as to why the 'seed > seed + public' takes such a long time, given that ssh-keygen 'does both' in a fraction of the time taken by ssh-keyparse?
The --ytdl-opts
/--aria2c-opts
arguments aren't being passed to the youtube-dl
/aria2c
commands.
I believe the issue is here, the arguments are being stored in local variables rather than the conf
object.
To reproduce, run this command:
py twitch_vod_fetch --ytdl-opts="--verbose" --aria2c-opts="log=aria2c.log" https://www.twitch.tv/videos/148136907 "../download_temp/Dave"
When a youtube-dl
command is logged, the --verbose
option isn't present and youtube-dl
doesn't print any debugging output. The same applies to aria2c
and the log=aria2c.log
option, it's not present in the .aria2c_conf file and no aria2c.log file is created.
Randomizing a directory crashes here:
$ sudo ./bin/vfat_shuffler --umount /dev/sdb1 / -s
Traceback (most recent call last):
File "./bin/vfat_shuffler", line 667, in <module>
if __name__ == '__main__': sys.exit(main())
File "./bin/vfat_shuffler", line 659, in main
op_reorder(path, src, dry_run=opts.dry_run, **sort_opts)
File "./bin/vfat_shuffler", line 489, in op_reorder
slice_map = path.get_slice_map()
File "./bin/vfat_shuffler", line 205, in get_slice_map
return list(map(self.fs.get_slice, self.get_chain()))
TypeError: get_chain() missing 1 required positional argument: 'clidx'
Full debug log: vfat_shuffler_posarg.log
Debian stable (and Ubuntu 18.x and 20.x) automatically mount USB flash drives under /media/$username/$driveLabel
.
vfat_shuffler unmounts it, and the system automatically removes the folder. Then the vfat_shuffler wants to re-mount it, and that fails because the folder is gone:
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "./bin/vfat_shuffler.py", line 667, in <module>
if __name__ == '__main__': sys.exit(main())
File "./bin/vfat_shuffler.py", line 665, in main
if res.returncode: res.check_returncode()
File "/usr/lib/python3.8/subprocess.py", line 444, in check_returncode
raise CalledProcessError(self.returncode, self.args, self.stdout,
subprocess.CalledProcessError: Command '['mount', '/dev/sdb1', '/media/cweiske/music']' returned non-zero exit status 32.
Something is not right… I get different base64-encoded result depending on whether I use the default option or I use hex option then transform to base64 (notice the different symbols around 'D', i.e., +D+
vs -D-
).
Why is that? Is it because the base64 transform you're using is "urlsafe"? Could we not have a switch to use ordinary base64 instead?
Thanks.
ssh-keyparse.py -s AedTQX1orIJQ4lyDtM4LiY0vTiELzSiL1EmFNrdy684= AedTQX1orIJQ4lyDtM4LiY0vTiELzSiL1EmFNrdy685cD9bgfGqnSchnrugs-D-Ths6A0rVCAg7wAUX8R0Mnpw==
ssh-keyparse.py -s AedTQX1orIJQ4lyDtM4LiY0vTiELzSiL1EmFNrdy684= -x | xxd -r -p | base64 -w 0 AedTQX1orIJQ4lyDtM4LiY0vTiELzSiL1EmFNrdy685cD9bgfGqnSchnrugs+D+Ths6A0rVCAg7wAUX8R0Mnpw==
./twitch-dl --debug http://www.twitch.tv/oculus/v/17538854 oculus3
2015-09-25 16:19:46 :: main INFO :: --- Downloading VoD oculus3 (url: http://www.twitch.tv/oculus/v/17538854)
2015-09-25 16:19:46 :: main DEBUG :: Starting aria2c daemon: aria2c --summary-interval=0 --console-log-level=notice --stop-with-process=2258 --enable-rpc=true --rpc-listen-port=57329 --rpc-secret=iD079kYMUur5sogvFzFB5IIP --no-netrc --no-proxy --always-resume=false --max-concurrent-downloads=5 --max-connection-per-server=5 --max-file-not-found=5 --max-tries=8 --timeout=30 --connect-timeout=20 --lowest-speed-limit=100K --user-agent=Mozilla/5.0 (X11; Linux x86_64; rv:10.0) Gecko/20150101 Firefox/20.0 (Chrome) --on-download-complete=/tmp/.fetch_twitch_vod.2258.done.hook
2015-09-25 16:19:46 :: main DEBUG :: Starting downloads (rpc port: 57329)...
09/25 16:19:46 [NOTICE] IPv4 RPC: listening on TCP port 57329
09/25 16:19:46 [NOTICE] IPv6 RPC: listening on TCP port 57329
terminate called after throwing an instance of 'std::length_error'
what(): basic_string::_S_create
2015-09-25 16:19:47 :: requests.packages.urllib3.util.retry DEBUG :: Incremented Retry for (url='/jsonrpc'): Retry(total=4, connect=None, read=None, redirect=None)
2015-09-25 16:19:47 :: requests.packages.urllib3.util.retry DEBUG :: Incremented Retry for (url='/jsonrpc'): Retry(total=3, connect=None, read=None, redirect=None)
2015-09-25 16:19:49 :: requests.packages.urllib3.util.retry DEBUG :: Incremented Retry for (url='/jsonrpc'): Retry(total=2, connect=None, read=None, redirect=None)
2015-09-25 16:19:53 :: requests.packages.urllib3.util.retry DEBUG :: Incremented Retry for (url='/jsonrpc'): Retry(total=1, connect=None, read=None, redirect=None)
2015-09-25 16:20:01 :: requests.packages.urllib3.util.retry DEBUG :: Incremented Retry for (url='/jsonrpc'): Retry(total=0, connect=None, read=None, redirect=None)
Traceback (most recent call last):
File "./twitch-dl", line 445, in <module>
if __name__ == '__main__': sys.exit(main())
File "./twitch-dl", line 443, in main
vod_fetch(url, prefix, dl_info_suffix=info_suffix, **dl_kws)
File "./twitch-dl", line 257, in vod_fetch
if len(line_buff) >= line_buff_max: line_buff_flush()
File "./twitch-dl", line 232, in line_buff_flush
if gid_files: queue_gid_downloads(gid_files)
File "./twitch-dl", line 216, in queue_gid_downloads
for gid_str, path in gid_files ))
File "./twitch-dl", line 97, in req_jrpc
with req('post', url, json=data_req) as r: data_res = r.json()
File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
return self.gen.next()
File "./twitch-dl", line 80, in req
with closing(req.s.request(method, *args, **kws)) as r:
File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 465, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 573, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 424, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=57329): Max retries exceeded with url: /jsonrpc (Caused by ProtocolError('Connection aborted.', error(111, 'Connection refused')))
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.