int-72h / pytoast Goto Github PK
View Code? Open in Web Editor NEWA simple, low overhead content distribution system
License: BSD 3-Clause "New" or "Revised" License
pytoast's People
Contributors
Stargazers
Watchers
Forkers
fentealepytoast's Issues
Further improvements
AKA the todo list.
- Sort out schema (UUIDs, hashing etc)
- Compression algorithm since lzma for that many files is really slow
- genericize it (just toast, not OFtoast since frontend devs be frontending) and create a shoddy backronym for toast
- Signing support possibly
- make it "suck less", optimisations and unbodging (not suckless though I'm not writing this in C)
'branches' support
I.E if there's an update that's in testing/some alternate version which isn't part of the canonical revision line but branches off at a point.
multiprocessing.pool.Remotetraceback
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/usr/lib/python3.9/multiprocessing/pool.py", line 125, in worker
result = (True, func(*args, **kwds))
File "/usr/lib/python3.9/multiprocessing/pool.py", line 51, in starmapstar
return list(itertools.starmap(args[0], args[1]))
File "/media/internald/ofatomic/ofatomic/__init__.py", line 60, in download_file_multi
memfile = u.read()
File "/usr/lib/python3.9/http/client.py", line 468, in read
s = self._safe_read(self.length)
File "/usr/lib/python3.9/http/client.py", line 609, in _safe_read
data = self.fp.read(amt)
File "/usr/lib/python3.9/socket.py", line 704, in readinto
return self._sock.recv_into(b)
File "/usr/lib/python3.9/ssl.py", line 1241, in recv_into
return self.read(nbytes, buffer)
File "/usr/lib/python3.9/ssl.py", line 1099, in read
return self._sslobj.read(len, buffer)
ConnectionResetError: [Errno 104] Connection reset by peer
"""
The above exception was the direct cause of the following `exception:`
Traceback (most recent call last):
File "/usr/bin/ofatomic", line 33, in <module>
sys.exit(load_entry_point('ofatomic', 'console_scripts', 'ofatomic')())
File "/media/internald/ofatomic/ofatomic/__init__.py", line 162, in main
dpool.starmap(download_file_multi, todl)
File "/usr/lib/python3.9/multiprocessing/pool.py", line 372, in starmap
return self._map_async(func, iterable, starmapstar, chunksize).get()
File "/usr/lib/python3.9/multiprocessing/pool.py", line 771, in get
raise self._value
Store used URL to verify objects
OFLauncher-rei supports specifying a server to download the files from, which can have different revisions than the official toast.openfortress.fun.
If a user, for whatever reason, decides to change the URL after installing this can lead to files being left in the incorrect state.
To fix this I propose adding a .remote (or similar #18) that lets the program verify what remote was used last and if we need to verify the files from the first revision onwards
dedicated revision information folder
Right now only one special file is used .revision to store the current revision ID.
But in the future more information might get added that would need to be stored in a known good location.
For this I propse a .tvn folder with .revision becoming .tvn/revision
for the sake of backwards compatibility .revision can be checked for if .tvn/revision is not present.
This would require coordinated changes to both toast and int-72h/oflauncher-rei which is why I am looking for feedback on this before creating Pull Requests
toast
toast
Hashing/UUID algorithm/version
For hashing I wanted to use MD5 since SHA256 was too slow and big for simple data checking, however I did discover UUIDv5 which generates a UUID which is predictable, in a given namespace.
Instead of:
revision: uuid: filename, hash
look up revision -> look up uuid -> get file -> check against hash
You could eliminate the need to store a hash if the UUID is essentially also the hash.
Having a seperate namespace for each revision would reduce the chance of a rainbow table attack.
If this isn't feasible for some weird reason the other way around would work- just store the file as the hash.
memfile is hashed even if hashing and signing are disabled
--disable-hashing doesn't disable hashing; it just ignores a mismatch. I assume by design as the signature check also uses the new hash, but it could be skipped if signing is disabled as well.
Redundant downloading
Either re-downloads or appears to re-download files that are already there and are the same as remote db
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.