sec-ant / torrefy Goto Github PK
View Code? Open in Web Editor NEWAn ESM package that uses Web Streams API to create v1, v2 or hybrid torrents in your web browser
Home Page: https://www.npmjs.com/package/torrefy
License: MIT License
An ESM package that uses Web Streams API to create v1, v2 or hybrid torrents in your web browser
Home Page: https://www.npmjs.com/package/torrefy
License: MIT License
Vite uses esbuild to prebundle dependencies and transform source codes in dev mode, which makes it a perfect selection to test the package in browser env without losing typescript features. And Vite is fast.
Maybe I should do some tests.
Related links:
Basically, there're two functions that need implementing:
Consume a readable stream as an async iterator.
This function is called when input files are consumed as readable streams and we expect to use async generators to manipulate chunks (regulating them to equally sized chunks, applying hash algorithm on them) of the file streams.
This feature is already merged in the web streams spec, but not yet implemented by mainstream browsers:
So, a polyfill is in need for now. There're already some good implementations:
MattiasBuelens/web-streams-polyfill
Written by the spec contributor Mattias Buelens, this repo is considered canonical but it polyfills all readable stream related APIs which seems not appropriate to be used only as a polyfill of async iteration.
ThaUnknown/fast-readable-async-iterator
Written by Cas who pointed out the performance issue in using web streams in my repo and implemented async iterations for webtorrent/create-torrent. His implementation uses one-value ahead cache, for performance reasons, I think, and properly cancels the stream when the iteration is returned or thrown, which is good. But I think there're still some problems:
return
or throw
)stream.values({ preventCancel: true })
There're some other existing snippets that have similar or more significant problems, including this one by [email protected] or this one by dy.
I intend to implement a spec-compliant polyfill in typescript. Performance is my second consideration as I think browser support will eventually supersede this polyfill soon enough (hopefully). This polyfill I'm working on is Sec-ant/readable-stream. It now passes all the tests that are copied from wpt.
Construct a readable stream from an async iterator or iterator.
This function is called when we want to collect the results and produce them as a ReadableStream
, which can be later consumed as a Response
or be piped to a FileSystemWritableFileStream
.
ReadableStream.from
is the proposed API yet not merged in the spec:
Reference implementations:
I implemented a fromIterable
function in Sec-ant/readable-stream to fill in the absence of ReadableStream.from
. It now passes all the tests that are copied from wpt.
Other works left including refactoring transformers to async generators are already done in another branch of this repo. But they still need testing as I also refactored many other places.
For large files, buffering the entire thing into an ArrayBuffer or Blob for new File(...)
isn't always doable. Would it be possible to have create([my_ReadableStream])
to avoid needing to buffer everything at once?
Trying to get this module to work with a Cloudflare Worker, but I seem to always get a weird truncated response. Any ideas on what I'm doing wrong?
interface Env {
}
import { create, encode } from "torrefy";
export default {
async fetch(
request: Request,
env: Env,
ctx: ExecutionContext
): Promise<Response> {
const testFile = new File(
["Hello world. This is the test file content."],
"testfile.txt"
);
// calculate (hash) the meta info of the test file
const metaInfo = await create([testFile]);
console.debug(metaInfo);
// bencode meta info into a readable stream
const torrentStream = encode(metaInfo);
return new Response(torrentStream);
},
};
I end up getting the following data returned:
00000000: 6431 303a 6372 6561 7465 6420 6279 3133 d10:created by13
00000010: 3a63 7265 6174 696f 6e20 6461 7465 343a :creation date4:
00000020: 696e 666f 65 infoe
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.