sec-ant / torrefy Goto Github PK

Vite uses esbuild to prebundle dependencies and transform source codes in dev mode, which makes it a perfect selection to test the package in browser env without losing typescript features. And Vite is fast.

Maybe I should do some tests.

Related links:

• The algorithms are implemented iteratively, because the VM stack is faster than a JS stack.

Basically, there're two functions that need implementing:

• Consume a readable stream as an async iterator.

  This function is called when input files are consumed as readable streams and we expect to use async generators to manipulate chunks (regulating them to equally sized chunks, applying hash algorithm on them) of the file streams.

  This feature is already merged in the web streams spec, but not yet implemented by mainstream browsers:

  • spec: https://streams.spec.whatwg.org/#rs-asynciterator
  • related issue: whatwg/streams#778
  • pr: whatwg/streams#980
  • tests: https://github.com/MattiasBuelens/wpt/blob/master/streams/readable-streams/async-iterator.any.js

  So, a polyfill is in need for now. There're already some good implementations:

  • MattiasBuelens/web-streams-polyfill
    Written by the spec contributor Mattias Buelens, this repo is considered canonical but it polyfills all readable stream related APIs which seems not appropriate to be used only as a polyfill of async iteration.

  • ThaUnknown/fast-readable-async-iterator
    Written by Cas who pointed out the performance issue in using web streams in my repo and implemented async iterations for webtorrent/create-torrent. His implementation uses one-value ahead cache, for performance reasons, I think, and properly cancels the stream when the iteration is returned or thrown, which is good. But I think there're still some problems:

    • The readable stream won't be released when the iteration is normally exited (i.e., loop exits without calls of return or throw)
    • It does not support stream.values({ preventCancel: true })
    • A lack of Typescript types

  There're some other existing snippets that have similar or more significant problems, including this one by [email protected] or this one by dy.

  I intend to implement a spec-compliant polyfill in typescript. Performance is my second consideration as I think browser support will eventually supersede this polyfill soon enough (hopefully). This polyfill I'm working on is Sec-ant/readable-stream. It now passes all the tests that are copied from wpt.

• Construct a readable stream from an async iterator or iterator.

  This function is called when we want to collect the results and produce them as a ReadableStream, which can be later consumed as a Response or be piped to a FileSystemWritableFileStream.

  ReadableStream.from is the proposed API yet not merged in the spec:

  • related issue: whatwg/streams#1018
  • pr: whatwg/streams#1083
  • reference implementation: https://github.com/MattiasBuelens/streams/blob/rs-from/reference-implementation/lib/abstract-ops/readable-streams.js#L1884
  • tests: https://github.com/MattiasBuelens/wpt/blob/rs-from/streams/readable-streams/from.any.js

  Reference implementations:

  • https://github.com/MattiasBuelens/streams/blob/2badb5f15a20531a150af8bebe92daf215b9e645/reference-implementation/lib/abstract-ops/readable-streams.js#L1884
  • https://deno.land/[email protected]/streams/readable_stream_from_iterable.ts?source#L39
  • https://github.com/ThaUnknown/webtorrent/blob/b9942016ebfe8224ed0685171f57fd9734554e99/lib/file.js#L147

  I implemented a fromIterable function in Sec-ant/readable-stream to fill in the absence of ReadableStream.from. It now passes all the tests that are copied from wpt.

Other works left including refactoring transformers to async generators are already done in another branch of this repo. But they still need testing as I also refactored many other places.

For large files, buffering the entire thing into an ArrayBuffer or Blob for new File(...) isn't always doable. Would it be possible to have create([my_ReadableStream]) to avoid needing to buffer everything at once?

Trying to get this module to work with a Cloudflare Worker, but I seem to always get a weird truncated response. Any ideas on what I'm doing wrong?

interface Env {
}

import { create, encode } from "torrefy";

export default {
	async fetch(
		request: Request,
		env: Env,
		ctx: ExecutionContext
	): Promise<Response> {
		const testFile = new File(
			["Hello world. This is the test file content."],
			"testfile.txt"
		);

		// calculate (hash) the meta info of the test file
		const metaInfo = await create([testFile]);
		console.debug(metaInfo);

		// bencode meta info into a readable stream
		const torrentStream = encode(metaInfo);

		return new Response(torrentStream);
	},
};

I end up getting the following data returned:

00000000: 6431 303a 6372 6561 7465 6420 6279 3133  d10:created by13
00000010: 3a63 7265 6174 696f 6e20 6461 7465 343a  :creation date4:
00000020: 696e 666f 65                             infoe