Comments (9)
Good news!
WHATWG streams have landed in NodeJS (as experimental)
nodejs/node#39062
Think we should start using it whenever possible, might be worth looking into adding whatwg stream polyfill now after all!
from fetch-blob.
I would love to converge https://www.npmjs.com/package/web-blob with fetch-blob, however incompatibility of stream method prevents me from doing it. If I were to do the work to replace current node Readable
with web ReadableStream
is it going to fly ?
from fetch-blob.
As long as the stream is async iterable node-fetch
will be happy with it.
but i would rather have it be some optional dependency.
from fetch-blob.
Made a decision to degrade the stream()
to barely do what both node & whatwg stream have in common in the v3, which is the async iterable - this would make it work in browser without the need of nodes stream or buffer module. and make it unopinionated to not pick anyones side between node or whatwg stream
Guess It can stay like that until node implements whatwg stream
For those who want to really have a node stream can now do stream.Readable.from(blob.stream())
or whatwg equivalent upcoming version whatwg/streams#1018 (which can be polyfilled too)
...And for those who do not care about either node or whatwg stream can use
for await (const chunk of stream) { ... }
All doe async iterable ReadableStream is not implemented anywhere in any browser AFAIK
So you can't really do for await (const chunk of stream) { ... } just yet but you could polyfill it by adding ReadableStream.prototype[symbol.asyncIterator]
also
from fetch-blob.
Made a decision to degrade the
stream()
to barely do what both node & whatwg stream have in common in the v3, which is the async iterable - this would make it work in browser without the need of nodes stream or buffer module. and make it unopinionated to not pick anyones side between node or whatwg stream
In practice it is not the case:
- No browser implements / ships async iteration for streams yet.
- Streams have bunch of things that async iterables do not.
I think it would be a lot better to just not have stream()
method and have a different one if it is going to be a different thing. That way it is easy to subclass and provide web API compatible version.
from fetch-blob.
I think it would be a lot better to just not have stream() method and have a different one if it is going to be a different thing. That way it is easy to subclass and provide web API compatible version.
how would you go about subclassing this if it can't provide a stream method? slicing the blob, read each chunk as arraybuffer or something like that? similar to this pr/file? both text()
and arrayBuffer()
also depends on some form of stream()
method, but it could have a internal (private-iterator) stream instead of using the public version...
fyi.
node-fetch
is very dependent onblob.stream()
and for it being (at the very least) async iterable. hence why some form ofstream()
is needed.- plus this is mostly built for the server, so I do not know why you would use
fetch-blob
overwindow.Blob
in the browser that is fully spec compatible?
...with that in mind you would probably also have web-stream-polyfill if you run this on nodejs - which provide Symbol.asyncIterator
I can see some few solution to this...
- We could check if
globalThis.ReadableStream
exist and "upgrade" the stream method into a whatwg stream if it exist
if (globalThis.ReadableStream) {
const orig = Blob.prototype.stream
Blob.prototype.stream = function() {
const iterator = orig.call(this)
return new ReadableStream({
pull: ctrl => iterator.next().then(({value, done}) done ? ctrl.close() : ctrl.enqueue(value))
})
}
}
This could be subclassed in user-land also. it is easy to check if stream is not spec compatible and see if it's a generator function or regular function with this detection script
if (Blob.prototype.stream.constructor.name === 'AsyncGeneratorFunction') {
// not spec compatible, subclass it...
}
another solution could be to polyfill ReadableStream.prototype[Symbol.iterator]
so you could actually use the for-await-of
loop in browser (and everywhere else) also instead of relying on the full stream implementation.
// Here is a very basic one
if (!ReadableStream.prototype[Symbol.asyncIterator]) {
ReadableStream.prototype[Symbol.asyncIterator] = async function* () {
const reader = this.getReader()
while (1) {
const chunk = await reader.read()
if (chunk.done) return chunk.value
yield chunk.value
}
}
}
The benefit of this is that your for..of
code could then handle all three cases where it's either node stream, whatwg stream or a generator (or anything with iterator symbol in its prototype)
There are few requirements on this blob implementation.
node-fetch
is dependent on a asyncIterable blob version ofblob.stream()
- we can't simply remove
stream()
if whatwg streams don't exist (cuz that would break things)
- we can't simply remove
- We do not want to import/depend on a hole whatwg stream polyfill in nodejs (it's simply too large).
this was built for nodejs after all so the polyfilled version of whatwg stream and node stream are async iterator so i think it's best to use for-await-of in most cases to avoid any environmental differences
other than those two/three requirements i'm open to suggestions
from fetch-blob.
First of all thanks you for willingness to carry on this conversation, I really appreciate it. I'll try to respond inline below
How would you go about subclassing this if it can't provide a stream method?
class Blob extends FetchBlob {
stream() {
return asyncIterableToReadableStream(FetchBlob.asyncIterate(this))
}
}
Where I assume static asyncIterate(blob)
could do what blob.stream()
does currently. Alternatively it could be asyncIterableToReadableStream(blob[GOOD_NAME]())
.
node-fetch is very dependent on blob.stream() and for it being (at the very least) async iterable. hence why some form of stream() is needed.
I'm not opposed to async iteration API in blob, on the contrary (In fact I wish blob[Symbol.asyncIterator]
was standardized). I do however find incompatibly with standard stream()
API problematic.
plus this is mostly built for the server, so I do not know why you would use fetch-blob over window.Blob in the browser that is fully spec compatible?
...with that in mind you would probably also have web-stream-polyfill if you run this on nodejs - which provide Symbol.asyncIterator
I think there is a bit of misunderstanding here. I do not have a case where fetch-blob would be preferred over window.Blob
. I do however work with code few layers up that operates on Response, Request, Blob
objects and is agnostic of runtime and in fact runs in both node and browsers. Unfortunately however API incompatibility across runtimes does cause issues and introduces complexity as it is forced to detect if blob stream is actually a stream or not and if not do something else. And that bleeds all over the place. In theory one could update APIs to pass async iterables around instead, but in practice it's difficult because:
- Not all components at play are under our control.
- There are good reasons to keep blobs as you can slice, concat and upload without loading things into memory (which is tricky to do if you target both node and browsers).
More recently I've also run into the same incompatibily problem with a code which targets both node and cloudfare workers (that has native fetch and blob impls).
We could check if globalThis.ReadableStream exist and "upgrade" the stream method into a whatwg stream if it exist
This is not a great option because while global polyfills seem fine when building an app, they don't seem ok in libraries. Having to tell library users to also bring global polyfill does not seem like a great option.
I can see some few solution to this...
You are right that one could subclass fetch-blob today like this:
class MyBlob extends FetchBlob {
stream() {
return asyncIterableToReadableStream(super.stream())
}
}
And maybe it would not break any assumptions anywhere and will work.
The benefit of this is that your for..of code could then handle all three cases where it's either node stream, whatwg stream or a generator (or anything with iterator symbol in its prototype)
You are right about benefits but:
- There are also tradeoffs (especially in browser context)
- Things get complicated when dealing with components from others as you have to convince them to do the same (while they may have good reasons not to).
There are few requirements on this blob implementation.
- node-fetch is dependent on a asyncIterable blob version of blob.stream()
It is true but again node-fetch could just as well depend on any other method. In fact I have filled the issue to that end there as well node-fetch/node-fetch#1120 and wrote POC https://github.com/web-std/node-fetch/pull/2/files
* We do not want to import/depend on a hole whatwg stream polyfill in nodejs (it's simply too large).
That is very reasonable position. But again you could choose to expose same functionality with a different name. It would work in node just the same and it would make it straight forward for someone to just combine two libs to get a whatwg compliant version.
P.S.: I do not think this alone provides much without node-fetch
from fetch-blob.
For bit more context, I found incompatibilities so problematic that I end up:
- Implementing my own blob implementation https://github.com/web-std/io/tree/main/blob
- Implementing web
File
API on top of it https://github.com/web-std/io/tree/main/file - Implementing web
FormData
API https://github.com/web-std/io/tree/main/form-data FromData
support in fetch node-fetch/node-fetch#1118- Forknig fetch that uses web streams instead https://github.com/web-std/node-fetch/pull/3/files
But unfortunately it is nearly impossible to backport fixes and I would much rather contribute to the community instead of forking it.
from fetch-blob.
ping @bitinn, @Richienb, @tinovyatkin
from fetch-blob.
Related Issues (20)
- Run test against WPT
- Move check to the top
- exec `npm run test test-wpt` with github workflows
- Use stream consumers HOT 1
- require "stream/web" produces warning HOT 21
- object key naming should not contain '#' HOT 2
- Module not found: Error: Can't resolve 'stream/web' HOT 20
- text() is fatal on non UTF-8 data, while in browser it works without any issue HOT 1
- How can I implement the below code in older version ? HOT 1
- maybe use `stream.Readable.toWeb` HOT 2
- Behavior of .slice() is incorrect when using blobs backed by a file on the disk
- node failing due to unexpected character '#' HOT 8
- Possibility to remove web-streams-polyfill dependency? HOT 1
- fileFromSync just part of a file HOT 1
- The RC version should not be released as latest. HOT 1
- Adding a File class maybe?
- Doesn't work either with latest stable and beta version of node-fetch HOT 6
- Revisit DOMException
- Support Node.js v12 HOT 4
- Implementing a mutable append only blob type? HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from fetch-blob.