Git Product home page Git Product logo

fetch-extra's Introduction

Node-fetch wrapper

The goal is to add some extra functionality to node-fetch in a non-intrusive way:

  • Extra options:
    • type FetchResource: Parameters<fetch>[0]
    • type FetchState: {retryCount: number, fetchId: string, resource: FetchResource, options: FetchOptions}
    • type FetchStats: {ok, error, totalMs, sentMs, receivedMs, sent, received}
    • retry: number | ({error: Error, response: Response, state: FetchState}) => Promise<boolean | {resource: FetchResource, options: FetchOptions} (could just return state)>
      • number is just an easy option to retry with a given number of attempts
      • and function is an interface in which you can make complex rules whether to retry or not
        • returning true = retry
        • returning falsy = don't retry, fetch will throw the original error
        • returning object = retry with given parameters (allows you to modify request on retry)
        • throwing = don't retry, fetch will throw what you've thrown
    • validate: true | (response: Omit<Response, "body">, state: FetchState) => Promise<void>: throws allows to retry the request (with no body consumed yet!) on particular circumstances
      • if true, then validate: res => if (!res.ok ) throw HttpError
    • validate.json, validate.buffer, etc - same rule as validate, but with access to the parsed body (result: any, state: FetchState)
    • timeout: number: default value for all the request handling (doing the request + fetching the body); same as in node-fetch v2
    • requestTimeout = timeout for await fetch(...) (includes body upload time)
    • bodyTimeout = timeout for fetching the whole body, i.e. await res.json()
    • stallTimeout = same as body, except instead of fixed time of fetching the whole body, we count the time of download speed of 0 bytes/s
  • todo: abort controller we use for timeout handling shouldn't ignore abort controller given in the options (already put the comment in the code)
  • todo: noProgress but for file upload (request)?
  • Extra Response fields:
    • completed: Promise<FetchStats>: Promise for body completion - rejects if fetch body or validation failed
  • makeFetch with built-in async-sema options like Sema and RateLimit
  • rich logging: ?
    • request start (include the body if it's string)
    • request done/failed
    • download for larger bodies (maybe with download speed?)
    • retries
    • sema
  • tests
  • todo: download helper that will hash on fly and have retry support

Helper StatsStream: maybe exists somewhere? re-export or rebuild

fetch-extra's People

Contributors

arturkot95 avatar dzieni avatar krystianfowler avatar wmertens avatar

Watchers

 avatar  avatar  avatar

fetch-extra's Issues

`validate` does not work with undici >= 5.12

> await require('@stratokit/fetch-extra')('https://example.com', {timeout: 30000, retry: 1, validate: true}).then(r => r.text())
Uncaught TypeError: Found invalid object in transferList
    at structuredClone (node:internal/structured_clone:23:17)
    at cloneBody (/Users/michal/Projects/strato-agent/node_modules/.pnpm/[email protected]/node_modules/undici/lib/fetch/body.js:260:21)
    at cloneResponse (/Users/michal/Projects/strato-agent/node_modules/.pnpm/[email protected]/node_modules/undici/lib/fetch/response.js:346:24)
    at Response.clone (/Users/michal/Projects/strato-agent/node_modules/.pnpm/[email protected]/node_modules/undici/lib/fetch/response.js:290:28)
    at fetch (/Users/michal/Projects/strato-agent/node_modules/.pnpm/github.com+StratoKit+fetch-extra@824d3381e3d44cad3_zb6buhrpkjh63xlguhkd2olchq/node_modules/@stratokit/fetch-extra/index.js:356:51)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async REPL1:1:33 {
  code: 'ERR_INVALID_TRANSFER_OBJECT',
  stats: {
    size: undefined,
    duration: 554.7955830097198,
    speed: 0,
    attempts: 1
  }
}

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.