Git Product home page Git Product logo

miniflare's Introduction

Warning

This repository is for Miniflare 2, which is only receiving critical security updates. Miniflare 2 simulated the Workers runtime and the rest of the Cloudflare developer platform using Node.js. New versions of Miniflare can be found in the workers-sdk repository, and use the open-sourced Workers runtime workerd. This practically eliminates behaviour mismatches between development and production deployments. We recommend you migrate to Miniflare 3 now if you can. Whilst Miniflare 3 supports most of Miniflare 2's features, one key omission is the unit testing environment. We're actively working on adding support for this to Miniflare 3. Once this is supported, we're planning to deprecate Miniflare 2 and archive this repository.

πŸ”₯ Miniflare 2

Miniflare is a simulator for developing and testing Cloudflare Workers.

  • πŸŽ‰ Fun: develop workers easily with detailed logging, file watching and pretty error pages supporting source maps.
  • πŸ”‹ Full-featured: supports most Workers features, including KV, Durable Objects, WebSockets, modules and more.
  • ⚑ Fully-local: test and develop Workers without an internet connection. Reload code on change quickly.

It's an alternative to wrangler dev, written in TypeScript, that runs your workers in a sandbox implementing Workers' runtime APIs.

See https://legacy.miniflare.dev for more detailed documentation.

Features

  • πŸ“¨ Fetch Events (with HTTP(S) server and manual dispatch)
  • ⏰ Scheduled Events (with cron triggering and manual dispatch)
  • πŸ”‘ Variables and Secrets with .env Files
  • πŸ“š Modules Support
  • πŸ“¦ KV (with optional persistence)
  • πŸͺ£ R2 (with optional persistence)
  • ✨ Cache (with optional persistence)
  • πŸ“Œ Durable Objects (with optional persistence)
  • 🌐 Workers Sites
  • βœ‰οΈ WebSockets
  • πŸ›  Custom & Wrangler Builds Support
  • βš™οΈ WebAssembly Support
  • πŸ—Ί Source Map Support
  • πŸ•Έ Web Standards: Base64, Timers, Fetch, Encoding, URL, Streams, Crypto
  • πŸ“„ HTMLRewriter
  • ⚑️ Live Reload on File Changes
  • πŸ“… Compatibility Dates/Flags Support
  • πŸ”Œ Multiple Workers Support
  • 🀹 Custom Jest Environment (with isolated per-test storage)
  • πŸ’ͺ Written in TypeScript

Install

Miniflare is installed using npm:

$ npm install -g miniflare@2 # either globally..
$ npm install -D miniflare@2 # ...or as a dev dependency

Using the CLI

$ miniflare worker.js --watch --debug
[mf:dbg] Options:
[mf:dbg] - Scripts: worker.js
[mf:dbg] Reloading worker.js...
[mf:inf] Worker reloaded! (97B)
[mf:dbg] Watching .env, package.json, worker.js, wrangler.toml...
[mf:inf] Listening on :8787
[mf:inf] - http://127.0.0.1:8787

Using the API

import { Miniflare } from "miniflare";

const mf = new Miniflare({
  script: `
  addEventListener("fetch", (event) => {
    event.respondWith(new Response("Hello Miniflare!"));
  });
  `,
});
const res = await mf.dispatchFetch("http://localhost:8787/");
console.log(await res.text()); // Hello Miniflare!

CLI Reference

Usage: miniflare [script] [options]

Core Options:
 -h, --help              Show help                                                   [boolean]
 -v, --version           Show version number                                         [boolean]
 -c, --wrangler-config   Path to wrangler.toml                                        [string]
     --wrangler-env      Environment in wrangler.toml to use                          [string]
     --package           Path to package.json                                         [string]
 -m, --modules           Enable modules                                              [boolean]
     --modules-rule      Modules import rule                                 [array:TYPE=GLOB]
     --compat-date       Opt into backwards-incompatible changes from                 [string]
     --compat-flag       Control specific backwards-incompatible changes               [array]
     --usage-model       Usage model (bundled by default)                             [string]
 -u, --upstream          URL of upstream origin                                       [string]
 -w, --watch             Watch files for changes                                     [boolean]
 -d, --debug             Enable debug logging                                        [boolean]
 -V, --verbose           Enable verbose logging                                      [boolean]
     --(no-)update-check Enable update checker (enabled by default)                  [boolean]
     --repl              Enable interactive REPL                                     [boolean]
     --root              Path to resolve files relative to                            [string]
     --mount             Mount additional named workers                [array:NAME=PATH[@ENV]]
     --name              Name of service                                              [string]
     --route             Route to respond with this worker on                          [array]
     --global-async-io   Allow async I/O outside handlers                            [boolean]
     --global-timers     Allow setting timers outside handlers                       [boolean]
     --global-random     Allow secure random generation outside handlers             [boolean]
     --actual-time       Always return correct time from Date methods                [boolean]
     --inaccurate-cpu    Log inaccurate CPU time measurements                        [boolean]

HTTP Options:
 -H, --host              Host for HTTP(S) server to listen on                         [string]
 -p, --port              Port for HTTP(S) server to listen on                         [number]
 -O, --open              Automatically open browser to URL                    [boolean/string]
     --https             Enable self-signed HTTPS (with optional cert path)   [boolean/string]
     --https-key         Path to PEM SSL key                                          [string]
     --https-cert        Path to PEM SSL cert chain                                   [string]
     --https-ca          Path to SSL trusted CA certs                                 [string]
     --https-pfx         Path to PFX/PKCS12 SSL key/cert chain                        [string]
     --https-passphrase  Passphrase to decrypt SSL files                              [string]
     --(no-)cf-fetch     Path for cached Request cf object from Cloudflare    [boolean/string]
     --live-reload       Reload HTML pages whenever worker is reloaded               [boolean]

Scheduler Options:
 -t, --cron              CRON expression for triggering scheduled events               [array]

Build Options:
 -B, --build-command     Command to build project                                     [string]
     --build-base-path   Working directory for build command                          [string]
     --build-watch-path  Directory to watch for rebuilding on changes                  [array]

KV Options:
 -k, --kv                KV namespace to bind                                          [array]
     --kv-persist        Persist KV data (to optional path)                   [boolean/string]

R2 Options:
 -r, --r2                R2 bucket to bind                                             [array]
     --r2-persist        Persist R2 data (to optional path)                   [boolean/string]

Durable Objects Options:
 -o, --do                Durable Object to bind                     [array:NAME=CLASS[@MOUNT]]
     --do-persist        Persist Durable Object data (to optional path)       [boolean/string]
     --(no-)do-alarms    Enable Durable Object alarms (enabled by default)           [boolean]

Cache Options:
     --(no-)cache        Enable default/named caches (enabled by default)            [boolean]
     --cache-persist     Persist cached data (to optional path)               [boolean/string]

Sites Options:
 -s, --site              Path to serve Workers Site files from                        [string]
     --site-include      Glob pattern of site files to serve                           [array]
     --site-exclude      Glob pattern of site files not to serve                       [array]

Bindings Options:
 -e, --env               Path to .env file                                            [string]
 -b, --binding           Binds variable/secret to environment                [array:KEY=VALUE]
     --global            Binds variable/secret to global scope               [array:KEY=VALUE]
     --wasm              WASM module to bind                                 [array:NAME=PATH]
     --text-blob         Text blob to bind                                   [array:NAME=PATH]
     --data-blob         Data blob to bind                                   [array:NAME=PATH]
 -S, --service           Mounted service to bind                      [array:NAME=MOUNT[@ENV]]

Acknowledgements

Miniflare was created by Brendan Coll.

Many thanks to dollarshaveclub/cloudworker and gja/cloudflare-worker-local for inspiration.

Durable Object's transactions are implemented using Optimistic Concurrency Control (OCC) as described in "On optimistic methods for concurrency control." ACM Transactions on Database Systems. Thanks to Alistair O'Brien for helping the Miniflare creator understand this.

miniflare's People

Contributors

cameron-robey avatar cerberus avatar chase avatar cherry avatar craiggleso avatar danifoldi avatar dsergiu avatar gregbrimble avatar jahands avatar james-maher avatar jbw1991 avatar jrencz avatar jschlesser avatar jstevans avatar kikobeats avatar leaysgur avatar lukeed avatar mosch avatar mrbbot avatar notorca avatar orls avatar penalosa avatar pi0 avatar rdaniels6813 avatar robertcepa avatar sj-e2digital avatar skye-31 avatar szkl avatar threepointone avatar yusefnapora avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

miniflare's Issues

Feature: Local IP information in local cf object

Background

Deployed Cloudflare workers offer the useful cf object, which provides Cloudflare-specific data from their edge network. Among these is the ASN, ASO, country, of the remote client IP. This data is especially useful as it cuts out an extra step in look up this data with an external service.

Since miniflare is not deployed on the Cloudflare edge, it does not provide this data. In fact, this data is explicitly overridden in the file miniflare/src/index.ts (lines 375-395).

This feature seeks to explore ways to replace some of the data in the cf object.

Targeted sources files and dependencies

  • miniflare/src/index.ts
    • contains the cf object and the overridden fields
    • this feature would alter the overridden fields to values from the locally ingested ip info
  • local ip info ingestion
    • somewhere we would be allowing the ingestion of a local ip info db, such as a .mmdb
    • this could potentially be done from the command line interface like many of the other miniflare configurations as an optional CLI flag/arg

Benefits of this feature

  • build consistent systems that rely on the Workers cf object for IP geo/whois data, even when running locally with miniflare

Risks of this feature

  • the cf object is data from the Cloudflare edge network. Supplementing the data in the cf object with data sourced from elsewhere is potentially confusing at best, misrepresentation at worst.

Fields in cf object marked for IP info

Below are a list of fields within the cf object that could make use of data from a local IP geo/whois database. I have also subjectively marked them as either geo or whois as that may determine the database to be used.

  • cf.asn: whois
  • cf.asOrganization: whois
  • cf.country: geo
  • cf.city: geo
  • cf.continent: geo
  • cf.latitude: geo
  • cf.longitude: geo
  • cf.postalCode: geo
  • cf.metroCode: geo
  • cf.region: geo
  • cf.regionCode: geo
  • cf.timezone: geo

Reference materials

When using the wrangler.toml config, use the durable objects specified within.

When specifying the wrangler config and env it is also required to specify durableObjects. They are already specified in the wrangler config.

const mf = new Miniflare({
  wranglerConfigEnv: wrangler_env,
  wranglerConfigPath: wrangler_config, // has durable objects
  durableObjectsPersist: true,
  durableObjects: { // duplicate
    USER: "User",
    USERS: "Users",
    PAGE_CONFIG: "PageConfig",
  },

Error using `miniflare -w`; `FetchError: No fetch handler` after rebuild

I am on Windows. While stumbling my way to the error in #33 I also encountered this issue with Miniflare failing on rebuild with the watcher. I am including logs that demonstrate starting my worker with miniflare -w, making a request to my worker which failed due to #33, then removing the problematic caching code. At this point the worker is rebuilt automatically. The call should succeed now. When I make another request to my worker, the call fails with FetchError: No fetch handler responded and unable to proxy request to upstream: no upstream specified. When I terminate Miniflare and restart it, the same call will succeed as expected. Here are my logs:

$ miniflare -w

up to date, audited 92 packages in 672ms

8 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities

> [email protected] build
> rm -rf ./dist && rollup -c


./src/main.mjs β†’ ./dist/main.mjs...
created ./dist/main.mjs in 245ms
[mf:inf] Build succeeded
[mf:inf] Worker reloaded!
[mf:inf] Listening on :8787
[mf:inf] - http://192.168.3.211:8787
[mf:inf] - http://127.0.0.1:8787
[mf:err] TypeError: headers is not iterable
    at normaliseHeaders (C:\Users\tim\AppData\Local\Volta\tools\image\packages\miniflare\node_modules\miniflare\src\kv\cache.ts:28:30)
    at Cache.put (C:\Users\tim\AppData\Local\Volta\tools\image\packages\miniflare\node_modules\miniflare\src\kv\cache.ts:62:24)
    at C:\Users\tim\Documents\Choir\Choir-Platform\file\dist\main.mjs:1:565
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at EventsModule.dispatchFetch (C:\Users\tim\AppData\Local\Volta\tools\image\packages\miniflare\node_modules\miniflare\src\modules\events.ts:149:27)
    at Miniflare._Miniflare_httpRequestListener (C:\Users\tim\AppData\Local\Volta\tools\image\packages\miniflare\node_modules\miniflare\src\index.ts:408:20)
GET /file/instrument-samples/piano/A2.mp3 200 OK (8.34ms)

up to date, audited 92 packages in 677ms

8 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities

> [email protected] build
> rm -rf ./dist && rollup -c

[mf:err] Unable to read dist\main.mjs: Error: ENOENT: no such file or directory, open 'C:\Users\tim\Documents\Choir\Choir-Platform\file\dist\main.mjs' (defaulting to empty string)
[mf:inf] Worker reloaded!

./src/main.mjs β†’ ./dist/main.mjs...
created ./dist/main.mjs in 249ms
[mf:inf] Build succeeded
[mf:err] GET /file/instrument-samples/piano/A2.mp3: FetchError: No fetch handler responded and unable to proxy request to upstream: no upstream specified. Have you added a fetch event listener that responds with a Response?
    at EventsModule.dispatchFetch (C:\Users\tim\AppData\Local\Volta\tools\image\packages\miniflare\node_modules\miniflare\src\modules\events.ts:167:13)
    at Miniflare.dispatchFetch (C:\Users\tim\AppData\Local\Volta\tools\image\packages\miniflare\node_modules\miniflare\src\index.ts:266:39)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at Miniflare._Miniflare_httpRequestListener (C:\Users\tim\AppData\Local\Volta\tools\image\packages\miniflare\node_modules\miniflare\src\index.ts:408:20)
GET /file/instrument-samples/piano/A2.mp3 500 Internal Server Error (67.11ms)
^C

$ miniflare

up to date, audited 92 packages in 661ms

8 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities

> [email protected] build
> rm -rf ./dist && rollup -c


./src/main.mjs β†’ ./dist/main.mjs...
created ./dist/main.mjs in 249ms
[mf:inf] Build succeeded
[mf:inf] Worker reloaded!
[mf:inf] Listening on :8787
[mf:inf] - http://192.168.3.211:8787
[mf:inf] - http://127.0.0.1:8787
GET /file/instrument-samples/piano/A2.mp3 200 OK (9.51ms)

Put utility interfaces under /cdn-cgi/

All requests on Cloudflare to paths under cdn-cgi will be handled by Cloudflare itself and cannot be routed to Workers or the origin. So rather than have utility interfaces under /.mf/* (which may in the off-chance have conflicts), it may make more sense to put them under /cdn-cgi/* for consistency (for example, preview interfaces for Workers live under /cdn-cgi/workers/preview).

THANK YOU!

I know issues aren't really the right place for this, but THANKS @mrbbot and other contributors!

Miniflare has taken almost all the pain out of developing on Cloudflare.

P.S. You should open up Github Sponsorships for your account. I'd be happy to throw you some bucks!

Ability to access context of current worker

Within cloudworker you could access the context of the running work by looking at

cw.context

By modifying the context you could change env variables at run without having to re-initiate the whole worker.

GET request body failing to parse

With an API we are building on Cloudflare Workers we support setting a body on a GET request. It works when it's deployed on a Cloudflare Worker, however we don't get any response from await event.request.json() with miniflare.

Thanks for building this awesome tool!

Failure to cache response: headers is not iterable

I am developing on Windows. I am experiencing a bug that is causing Miniflare caching to be unusable. I have written a worker using the following code:

'use strict';

// Export this worker as a module.
export default {
  async fetch (req, env, ctx) {
    try {
      return handleRequest(req, env, ctx);
    } catch (error) {
      return new Response(error.message);
    }
  }
};

const modifyHeaders = function (headers, status) {
  const output = new Headers(headers);

  // Set CORS to allow the file to be fetched by any origin.
  output.set('Access-Control-Allow-Origin', '*');

  // Return the modified headers.
	return output;
};

/**
  Handle the incoming request to this worker.
*/
async function handleRequest (req, env, ctx) {

  // Attempt to retrieve the file from the cache.
  const url = new URL(req.url);
  const cache = await caches.open('file');
  let cacheResponse = await cache.match(url);
  if (cacheResponse) {
    const processedHeaders = modifyHeaders(cacheResponse.headers,
      cacheResponse.status);
    processedHeaders.set('X-Worker-Cache', true);
    return new Response(cacheResponse.body, {
      status: cacheResponse.status,
      statusText: cacheResponse.statusText,
      headers: processedHeaders
    });

  // Otherwise, the file is not present in the cache. We can try to fetch it.
  } else {
    let fileResponse;
    try {
      fileResponse = new Response(`Success!`, { status: 200 });
    } catch (error) {
      return new Response(`file::file-response::${B2_BUCKET_URL}${b2FileUrl}`, { status: 404 });
    }

    // Update headers to inform the browser to store this file for a long time.
    const processedHeaders = modifyHeaders(fileResponse.headers,
      fileResponse.status);
    fileResponse = new Response(fileResponse.body, {
      status: fileResponse.status,
      statusText: fileResponse.statusText,
      headers: processedHeaders
    });

    // Store the file in the cache.
    let clone = fileResponse.clone();
    ctx.waitUntil(cache.put(url, new Response(clone.body, {
      status: clone.status,
      statusText: clone.statusText
    })));

    // Return the file.
    return fileResponse;
  }
};

This worker runs without issue using wrangler dev.

When I use Miniflare and run my worker with miniflare from a globally-installed Miniflare instance, I am unable to simulate use of the cache. I receive logs as follows:

[mf:inf] Build succeeded
[mf:inf] Worker reloaded!
[mf:inf] Listening on :8787
[mf:inf] - http://192.168.3.211:8787
[mf:inf] - http://127.0.0.1:8787
[mf:err] TypeError: headers is not iterable
    at normaliseHeaders (C:\Users\tim\AppData\Local\Volta\tools\image\packages\miniflare\node_modules\miniflare\src\kv\cache.ts:28:30)
    at Cache.put (C:\Users\tim\AppData\Local\Volta\tools\image\packages\miniflare\node_modules\miniflare\src\kv\cache.ts:62:24)
    at C:\Users\tim\Documents\Choir\Choir-Platform\file\dist\main.mjs:1:565
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at EventsModule.dispatchFetch (C:\Users\tim\AppData\Local\Volta\tools\image\packages\miniflare\node_modules\miniflare\src\modules\events.ts:149:27)
    at Miniflare._Miniflare_httpRequestListener (C:\Users\tim\AppData\Local\Volta\tools\image\packages\miniflare\node_modules\miniflare\src\index.ts:408:20)
GET /file/instrument-samples/piano/A2.mp3 200 OK (11.59ms)

I've read through your source and the relevant portion is here, but I've not done any further digging yet:

// Normalises headers to object mapping lower-case names to single values.
// Single values are OK here as the headers we care about for determining
// cacheability are all single-valued, and we store the raw, multi-valued
// headers in KV once this has been determined.
function normaliseHeaders(headers: Headers): Record<string, string> {
  const result: Record<string, string> = {};
  for (const [key, value] of headers) {
    result[key.toLowerCase()] = value;
  }
  return result;
}

Any help would be appreciated.

node-fetch breaks due to missing lib folder

Perhaps I should have filed an issue over at https://github.com/mrbbot/node-fetch, but -since it's meant to be used specifically with miniflare- this looked like a better option.

It turns out @mrbbot/node-fetch declares its entrypoints (module, and main) as well as it's type declarations, pointing to files contained in the lib folder. These files are explicitly included in the npm package though the files entry of package.json

  "main": "lib/index",
  "types": "index.d.ts",
  "module": "lib/index.mjs",
  "files": [
    "lib/index.js",
    "lib/index.mjs",
    "lib/index.es.js",
    "index.d.ts"
  ]

However, said folder is ignored on the package's .gitignore file

# Babel-compiled files
lib

Which would work fine if the package was installed from a package registry, in which the contents of lib would positively show up. But, in Miniflare, you're requiring node-fetch from github

https://github.com/mrbbot/miniflare/blob/b17d90961e146cf674c0c5c1a4be5e1c998f60a3/package.json#L30

The final result is you get every file that the package would have stripped off, and miss the only one that matters, which is (lib) and miniflare crashes.

Error: Cannot find module '/home/ffflabs/cf-app/node_modules/@mrbbot/node-fetch/lib/index'. Please verify that the package.json has a valid "main" entry

I guess you haven't noticed because your local copy has the built main and module files already. .For the time being I forked miniflare, and installed a modified node-fetch that ensures existense of the entry points.

I have no problem showing that solution, it's just unellegant. You' ll have to consider if it's worth publishing it as an independent module, or, if it's meant for Miniflare only it makes more sente to move it to the main repo.

s-max-age support

Great project, was using it on the airplane recently :)

I noticed cache-control s-max-age is not working but max-age does.
Is this intended?

Thanks!

DurableObject newUniqueId

This method doesnt seem to be implemented or at least it doesnt seems to be able to generate a string when you call toString()) on it.

Option to intercept requests before firing event

Hey!

I've stumbled upon a possible issue, Currently if the upstream option is set the incoming host header isn't being used. We use a single worker for multiple subdomains, but still need the correct upstream URL. So we need to fetch the subdomain from the incoming request and use it as the subdomain on the upstream URL.

With dollarshaveclub/cloudworker we had to create our own HTTP server so this could be handled easily, But now we are using the server built into MiniFlare which handles things slightly different to our use case.

If we are able to modify the incoming request just before this.dispatchFetch is called it would solve our problem. As long as we had access to the original incoming request and the new request that get's passed to the worker

https://github.com/mrbbot/miniflare/blob/8010e3321d7913a206839e683ef469c22df5aedb/src/index.ts#L416

[BUG] Websocket upgrades shouldn't be possible with POST requests.

Upgrading a request to WebSockets should only be possible with GET requests. I verified this in production and took me a while to figure out that this was the thing that was going wrong. The following code sample shouldn't work.

        const res = await durableObject.fetch(
          `http://fake-host/events/messageSend?clientId=${ctx.message.id}`,
          {
            method: 'POST',
            headers: {
              Upgrade: 'websocket'
            },
          },
        )
        console.log('has WebSocket', typeof res?.webSocket !== "undefined");

Took me a while to figure out why my code wasn't working in production but was working with Miniflare. this was the reason. Will be great to have this fixed in miniflare.

I might be able to work on a PR this week. What are the preferences in terms of testing?

E.g. would a test called test("fetch: only performs web socket upgrade on GET requests") be ok?

In case I don't get round to it, I think it might be as simple as changing the following file: src/modules/standards.ts

before:

    // Handle web socket upgrades
    if (request.headers.get("upgrade") === "websocket") {

after:

    // Handle web socket upgrades, this should only for GET requests
    if (request.method === "GET" && request.headers.get("upgrade") === "websocket") {

miniflare should stringify all env vars

Hello there!

Cloudflare stringyfies all environment variables, but miniflare respects/maintains the var type.

This might cause confusion/bugs and detours from what's expected when using wrangler etc.

[vars]
MY_VAR = true

Using miniflare:

console.log(typeof env.MY_VAR) // boolean

Using wrangler:

console.log(typeof env.MY_VAR) // string

It's not a huge thing but I thought I'd mention it.

Thanks again for this project, it's been a life saver!

Missing EventTarget and Event

For compliance with Cloudflare the global object (aka globalThis et al) and WebSocket need to inherit from EventTarget and both EventTarget and Event need to be exposed globally. I did some preliminary investigation and it’s not trivial to do; WebSocket implementation etc. but it seems doable.

I don’t need this right this second, so no immediate rush, but it was discovered while trying to test our complex worker so if/when we eventually need this we’ll spend more time and potentially contribute the fixes, assuming you haven’t prioritized it yourself. Our usage is pretty advanced, so I imagine most won’t run into this.

Unable to find class DurableObjectExample for Durable Object EXAMPLE_CLASS

After unsuccessfully making rollup config work for my use case (See #49), I attempted to use esbuild but with typescript following the esbuild recipe : https://miniflare.dev/recipes/esbuild.html

I created a repo here https://github.com/wighawag/esbuild-typescript-worker

But I get the following error : Unable to find class DurableObjectExample for Durable Object EXAMPLE_CLASS

To reproduce:

  • clone repo and cd into ot
  • npm i
  • npm run dev

Any idea?

Question - Durable Objects

Firstly @mrbbot miniflare is a wonderful project, thank you so much. It's very very easy to use 🦾 !

I'm sure that I'm missing something so I apologise if it's covered in the docs already 🀦 πŸ˜„

Given

// wranger.toml
...

[durable_objects]
bindings = [
    { name = "OBJECT", class_name = "DurableObject", script_path="./src/DurableObject.mjs" }
]
// ./src/index.mjs

addEventListener('fetch', event => {
  event.respondWith(handleRequest(event.request))
})
/**
 * Respond with hello worker text
 * @param {Request} request
 */
async function handleRequest(request) {
  return new Response('Hello worker!', {
    headers: { 'content-type': 'text/plain' },
  })
}
// ./src/DurableObject.mjs

export class DurableObject {
    
}
// package.json

{
    ...,
    "type": "module",
    "module": "src/index.mjs",
    "scripts": {
        "dev": "miniflare -w -d",
        ...
    }
}

Debug Output - Error loading DO class

[mf:dbg] Options:
[mf:dbg] - Scripts: src/index.mjs
[mf:dbg] - Modules: true
[mf:dbg] - Modules Rules: {ESModule: **/*.mjs}, {CommonJS: **/*.js, **/*.cjs}
[mf:dbg] - Durable Objects: OBJECT
[mf:dbg] Reloading src/index.mjs...
[mf:err] Unable to find class DurableObject for Durable Object OBJECT node_modules/miniflare/src/log.ts:47 πŸ”₯ 
[mf:inf] Worker reloaded!
[mf:dbg] Watching .env, package.json, src/index.mjs, wrangler.toml...
[mf:inf] Listening on :8787
[mf:inf] - http://127.0.0.1:8787
[mf:inf] - http://192.168.1.71:8787

Question

Reading the miniflare durable object docs it seems like this setup should work, am I missing something πŸ€” ?

Modify index.mjs like below causes the durable object to load irrespective of script_path="..." being set in wrangler.toml

// ./src/index.mjs

// Durable Objects
export { DurableObject } from './DurableObject.mjs' ✨ 

addEventListener('fetch', event => {
  event.respondWith(handleRequest(event.request))
})
/**
 * Respond with hello worker text
 * @param {Request} request
 */
async function handleRequest(request) {
  return new Response('Hello worker!', {
    headers: { 'content-type': 'text/plain' },
  })
}
[mf:dbg] Options:
[mf:dbg] - Scripts: src/index.mjs
[mf:dbg] - Modules: true
[mf:dbg] - Modules Rules: {ESModule: **/*.mjs}, {CommonJS: **/*.js, **/*.cjs}
[mf:dbg] - Durable Objects: OBJECT
[mf:dbg] Reloading src/index.mjs...
[mf:inf] Worker reloaded!
[mf:dbg] Watching .env, package.json, src/DurableObject.mjs, src/index.mjs, wrangler.toml... πŸŽ‰ 
[mf:inf] Listening on :8787
[mf:inf] - http://127.0.0.1:8787
[mf:inf] - http://192.168.1.71:8787

Attempting to read a nonexistent KV entry results in null, where cloudflare workers throws an exception

The example bellow, when ran using miniflare main.js -k TEST_NAMESPACE1 returns null.

// main.js
addEventListener("fetch", (e) => {
  e.respondWith(
    TEST_NAMESPACE1.get("key").then((value) => new Response(value))
  );
});

But when uploaded to cloudflare workers with the KV binding, it throws an exception.
In order to get the null response in cloudflare workers, a key must be created with null value.

File type error when uploading file via formData

When trying upload file with formData miniflare is throwing following error:

/home/pagan/webdev/kvms/backend/node_modules/formdata-node/lib/FormData.js:286
      throw new TypeError(
            ^
TypeError: Failed to execute 'append' on 'FormData':  parameter 2 is not one of the following types: ReadableStream | ReadStream | Readable | Buffer | File | Blob
    at FormData.__setField (/home/pagan/webdev/kvms/backend/node_modules/formdata-node/lib/FormData.js:286:13)
    at FormData.append (/home/pagan/webdev/kvms/backend/node_modules/formdata-node/lib/FormData.js:406:17)
    at FileStream.<anonymous> (/home/pagan/webdev/kvms/backend/node_modules/@mrbbot/node-fetch/lib/index.js:463:29)
    at FileStream.emit (node:events:406:35)
    at endReadableNT (node:internal/streams/readable:1329:12)
    at processTicksAndRejections (node:internal/process/task_queues:83:21)

This error isn't showing when using wrangler dev

wrangler.toml flags

compatibility_flags = [ "formdata_parser_supports_files" ]
compatibility_date = "2021-09-09"

Simple code to test:

const handler = async (request) => {
    const fd = await request.formData()
    const filename = fd.get('file').name
    return new Response(filename,{status:200})
}

addEventListener('fetch', event => event.respondWith(handler(event.request)))

Miniflare version: 1.4.1

Response method in Miniflare doesn't sanitize html special charecters but cloudflare workers does

The following example:

// worker.js
addEventListener("fetch", (event) => {
  event.respondWith(new Response("<h1>Hello Miniflare!</h1>"));
});

when I run miniflare worker.js Miniflare displays the h1 tag. But when I deploy it to cloudflare workers it doesn't display the h1 tag. Instead it just shows:
<h1>Hello World!</h1>
Also noting cloudflare workers output is encapsulated by a <pre> tag.

[BUG] Missing removeEventListener on Websocket class.

After needing a way to remove an event listener from a Websocket, I discovered that Miniflare doesn't have removeEventListener in the Websocket class, but Cloudflare Workers in production actually have, even though it's not documented.

Here's how I tested it:

    const [client, server] = Object.values(new WebSocketPair());
    for (var test in server) {
      console.debug("[Worker] Websocket property:", test);
    }

Miniflare is not responding once

Current behavior

const { Miniflare } = require('miniflare')

;(async () => {
  const mf = new Miniflare({
    script: `
    addEventListener('fetch', async event => {
      event.respondWith(new Response('one'))
      event.respondWith(new Response('two'))
    })
    `
  })
  const res = await mf.dispatchFetch('http://localhost:8787/')
  console.log(await res.text()) // 'two', wrong!
})()

Expected behavior

CleanShot 2021-10-13 at 16 11 12@2x

Do Durable Objects not reload when `-w` is enabled?

I can see in the code that they get reset, but from my testing, they don't seem to be reloaded. So if I make a change to a Durable Object, I need to restart miniflare?

If that is indeed correct, is there a reason you don't want to reload the Durable Object? And if yes, can we have a flag to also reload DOs?

It doesn't work on windows

If I do npm install -D miniflare, I get the following error:

> [email protected] postinstall C:\Users\carlo\WebstormProjects\miniflare\node_modules\whatwg-stream-to-async-iter
> shx ls index.js >> /dev/null 2>&1 || npm run build

The system cannot find the path specified.

> [email protected] build C:\Users\carlo\WebstormProjects\miniflare\node_modules\whatwg-stream-to-async-iter
> npm run build:mjs & npm run build:cjs & wait


> [email protected] build:mjs C:\Users\carlo\WebstormProjects\miniflare\node_modules\whatwg-stream-to-async-iter
> tsc -p tsconfig.json


> [email protected] build:cjs C:\Users\carlo\WebstormProjects\miniflare\node_modules\whatwg-stream-to-async-iter
> tsc -p tsconfig.cjs.json && npm run sed && npm run mv


> [email protected] sed C:\Users\carlo\WebstormProjects\miniflare\node_modules\whatwg-stream-to-async-iter
> shx sed -i 's/\.(.*)\.js/\.$1\.cjs/g' cjs/*.js > /dev/null

The system cannot find the path specified.
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] sed: `shx sed -i 's/\.(.*)\.js/\.$1\.cjs/g' cjs/*.js > /dev/null `
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] sed script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
npm WARN Local package.json exists, but node_modules missing, did you mean to install?

npm ERR! A complete log of this run can be found in:
npm ERR!     C:\Users\carlo\AppData\Roaming\npm-cache\_logs\2021-07-09T19_04_22_743Z-debug.log
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] build:cjs: `tsc -p tsconfig.cjs.json && npm run sed && npm run mv`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] build:cjs script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
npm WARN Local package.json exists, but node_modules missing, did you mean to install?

npm ERR! A complete log of this run can be found in:
npm ERR!     C:\Users\carlo\AppData\Roaming\npm-cache\_logs\2021-07-09T19_04_22_795Z-debug.log
'wait' is not recognized as an internal or external command,
operable program or batch file.
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] build: `npm run build:mjs & npm run build:cjs & wait`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] build script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
npm WARN Local package.json exists, but node_modules missing, did you mean to install?

npm ERR! A complete log of this run can be found in:
npm ERR!     C:\Users\carlo\AppData\Roaming\npm-cache\_logs\2021-07-09T19_04_22_884Z-debug.log
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@~2.3.2 (node_modules\chokidar\node_modules\fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for [email protected]: wanted {"os":"darwin","arch":"any"} (current: {"os":"win32","arch":"x64"})
npm WARN [email protected] No description
npm WARN [email protected] No repository field.

npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] postinstall: `shx ls index.js >> /dev/null 2>&1 || npm run build`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] postinstall script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     C:\Users\carlo\AppData\Roaming\npm-cache\_logs\2021-07-09T19_04_23_100Z-debug.log

2021-07-09T19_04_23_100Z-debug.log.

It seems to be a problem with the dependency whatwg-stream-to-async-iter.

Make request headers immutable

As mentioned in #32 the request headers in miniflare aren't immutable so I was able to introduce a bug that didn't get picked up until it was deployed.

Would you add a feature request to make these immutable so that my mistakes are picked up earlier?

Thanks so much for miniflare! It's changed my life.

Edit: this was actually on the response headers so perhaps that'll be two feature requests.

How to handle modules from npm?

This might be stupid question but I'm buried deep under this problem. I have working repo, running at CF Worker.

I'm trying to use JWT package with

import JWT from '@tsndr/cloudflare-worker-jwt';

But as I start miniflare src/index.mjs -w -d I get following error:

[mf:err] MiniflareError: Unable to resolve "src/index.mjs" dependency "@tsndr/cloudflare-worker-jwt": no matching module rules

I have dependency at package.json:

    "dependencies": {
        "@cfworker/uuid": "^1.9.0",
        "@tsndr/cloudflare-worker-jwt": "^1.1.2"
    }

I am doing something wrong, but what?

KV get() causing 500 Internal Server Error

I feel like I must be missing something obvious here. Calling get() on the KV store causes an internal server error, even if the returned value is unused.

Recipe to reproduce:

$ cat worker.js
addEventListener("fetch", async (event) => {
  const val = await SCREENSHOTS.get('test');
  event.respondWith(new Response("Hello Miniflare!"));
});

$ miniflare --version
1.4.1

$ miniflare worker.js --kv SCREENSHOTS --watch --debug
[mf:dbg] Options:
[mf:dbg] - Scripts: worker.js
[mf:dbg] - KV Namespaces: SCREENSHOTS
[mf:dbg] Reloading worker.js...
[mf:inf] Worker reloaded! (149B)
[mf:dbg] Watching .env, package.json, worker.js, wrangler.toml...
[mf:inf] Listening on :8787
[mf:inf] - http://127.0.0.1:8787
[mf:inf] - http://192.168.11.8:8787
[mf:inf] - http://10.10.10.64:8787
[mf:inf] - http://10.10.10.64:8787

... then point browser (Chrome latest) at http://127.0.0.1:8787 results in this error in the console:

[mf:err] GET /: FetchError: No fetch handler responded and unable to proxy request to upstream: no upstream specified. Have you added a fetch event listener that responds with a Response?
    at ServiceWorkerGlobalScope.[dispatchFetch] (/Users/kieffer/work/miniflare/node_modules/miniflare/src/modules/events.ts:224:13)
    at processTicksAndRejections (internal/process/task_queues.js:95:5)
GET / 500 Internal Server Error (78.47ms)
[mf:err] GET /favicon.ico: FetchError: No fetch handler responded and unable to proxy request to upstream: no upstream specified. Have you added a fetch event listener that responds with a Response?
    at ServiceWorkerGlobalScope.[dispatchFetch] (/Users/kieffer/work/miniflare/node_modules/miniflare/src/modules/events.ts:224:13)
    at processTicksAndRejections (internal/process/task_queues.js:95:5)
GET /favicon.ico 500 Internal Server Error (20.69ms)

NOTE: This problem goes away if you comment out the const val = SCREENSHOTS.get()... line in worker.js.

ReferenceError: FormData is not defined

First, I wanted to say I'm very excited about Miniflare. Thanks for making this!

It looks like the FormData class is not available in Miniflare yet. My use case is just to make a POST request to an external endpoint like so:

let formData = new FormData ()
formData.append ('field', someValue)

const response = await fetch ('https://example.com/post/here', {
    method: 'POST',
    body: formData
})

This has been tested and works with the current Cloudflare Workers. Thanks!

Cache doesn't support the `cf:{cacheKey: 'cacheKeyHere'}` property of a request

An undocumented, but supported, feature of the cache API is the ability to supply custom cache keys via the CF object of a request

eg

let cacheKey = new Request(request, {
        'method': request.method,
        'headers': request.headers,
        'body': (request.method == 'GET' || request.method == 'HEAD' ) ? null : request.body,
        'cf': {
            'cacheKey': getCacheKey(request),
            'reallyUseCacheKey': true,
        }
    });

We use this pretty extensively within our workers. Happy to take a punt at adding it if you're willing to support it?

I'd understand if you wouldn't want to given it's largely undocumented support.

[Feature Request]: Intercept requests made by the worker

Hey! First thanks for this awesome project, We are currently using the one from dollar shave club but that project died :-(

We need a way to strip the added Cloudflare headers added in MiniFlare as it causes Cloudflare to block the request. We need to route to our upstream via cloudflare as there is no direct access to the servers.

So one thought was a way to have some sort of middleware we can write to strip the needed headers to make it work
Or maybe there can be an option to not add any CF headers when the worker makes fetch requests.

Looking forward to hearing your opinion!

Maximum header size hit

I am sending large headers to my cloudflare workers and want to simulate this locally using miniflare.

With normal node its possible to allow bigger headers like --max-http-header-size 15000 as an example

How can I tell miniflare to accept larger headers?

I wondered if one option was to use the sdk of miniflare and wrap it in a node app so that I then call it like node my-miniflare.js or something but thought I would ask for advice.

While I'm here, as couldn't find anything in docs:

  • I have multiple workers I want to simulate, and am currently starting them each on a different miniflare port. Is there a way I could have them at different URLs on the same port? I have one worker that supplies an HTML file that has js in, which inturn calls another worker and while they are on different ports I get a CORS issue

Miniflare Error on DO Webpack CommonJS template

When setting up this template, Miniflare throws the following error
Miniflare Error
which ultimately results in a runtime error of

[mf:err] GET /process: FetchError: No fetch handler responded and unable to proxy request to upstream: no upstream specified. Have you added a fetch event listener that responds with a Response?
    at EventTarget.[dispatchFetch] (C:\Users\tim\AppData\Local\Volta\tools\image\packages\miniflare\node_modules\miniflare\src\modules\events.ts:224:13)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
GET /process 500 Internal Server Error (87.10ms)

package that uses buffer as dependency fails

Error

[mf:err] MiniflareError: Unable to resolve "dist/index.mjs" dependency "buffer": no matching module rules
[mf:err] MiniflareError: Unable to resolve "dist/index.mjs" dependency "buffer": no matching module rules
    at ScriptLinker._linker (/home/wighawag/dev/wighawag/etherplay/conquest-agent-service/node_modules/miniflare/src/scripts.ts:126:13)
    at ModuleWrap.<anonymous> (internal/vm/module.js:321:30)
    at SourceTextModule.<computed> (internal/vm/module.js:320:36)
    at SourceTextModule.link (internal/vm/module.js:198:22)
    at ScriptBlueprint.buildModule (/home/wighawag/dev/wighawag/etherplay/conquest-agent-service/node_modules/miniflare/src/scripts.ts:42:18)
    at Miniflare._Miniflare_reloadWorker (/home/wighawag/dev/wighawag/etherplay/conquest-agent-service/node_modules/miniflare/src/index.ts:180:26)
    at Miniflare._Miniflare_watchCallback (/home/wighawag/dev/wighawag/etherplay/conquest-agent-service/node_modules/miniflare/src/index.ts:124:29)
    at OptionsWatcher._init (/home/wighawag/dev/wighawag/etherplay/conquest-agent-service/node_modules/miniflare/src/options/watcher.ts:100:16)
    at Miniflare.getOptions (/home/wighawag/dev/wighawag/etherplay/conquest-agent-service/node_modules/miniflare/src/index.ts:292:5)

Conflict in wrangler.toml between miniflare and wrangler when using esbuild

When using esbuild everything works in miniflare just fine, but when trying use wrangler dev it's showing error Error: unknown field `main`, there are no fields which relates to main in [build.upload].
According to Workers Docs field main is supported in modules format (beta for paid plan).
After deleting it miniflare stop working with [mf:err] MiniflareError: No script defined, either include it explicitly, or set build.upload.main in Wrangler configuration and wrangler dev is building and work fine.

./src/index.js

import { handler } from "./m";

addEventListener("fetch", (event) => {
  event.respondWith(handler());
});

./src/m.js

const handler = () => {
    return new Response("12345")
}

export {handler}

package.json

{
    "private": true,
    "name": "backend",
    "version": "1.0.0",
    "description": "A template for kick starting a Cloudflare Workers project",
    "main": "./dist/index.js",
    "scripts": {
        "build": "esbuild --bundle --sourcemap --outdir=dist ./src/index.js",
        "dev": "miniflare --watch --debug"
    },
    "keywords": [],
    "author": "",
    "license": "ISC",
    "devDependencies": {
        "esbuild": "^0.12.15",
        "miniflare": "^1.1.0"
    }
}

wrangler.toml

name = "backend"
type = "javascript"
account_id = "XXXXXXXX"
workers_dev = true
route = ""
zone_id = ""

[build]
command = "npm install && npm run build"

[build.upload]
format = "service-worker"
main = "index.js" ## this

[miniflare]
host = "127.0.0.1"                 ## --host
port = 8787                        ## --port
npm -v
7.19.1

node -v
v16.4.2

wrangler --version
wrangler 1.17.0

miniflare --version
1.1.0

IncomingRequestCfProperties Support

As stated in the CloudFlare documentation for Request:

In addition to the properties on the standard Request object, the request.cf object on an inbound Request contains information about the request provided by Cloudflare’s edge.

Initially, using the provided the examples to mock these properties should be sufficient.

Further enhancement could be done by using fast-geoip to return semi-accurate results for the geo-IP information.


(PS. Thanks for all your hard work on this so far!)

Esbuild doesn't rerun

When launching miniflare esbuild won't rerun on change, so old file gets loaded. I've checked my config and I don't think there's some sort of browser caching is involved.

wrangler.toml

name = "test-worker"
type = 'javascript'
account_id = ''
route = ''
zone_id = ''
usage_model = ''
workers_dev = true
compatibility_date = "2021-09-17"

[build]
watch_dir="./js"
command = "npm run buildes"
[build.upload]
# The "modules" upload format is required for all projects that export a Durable Objects class
format = "service-worker"

package.json

{
  "name": "test-worker",
  "main": "./dist/index.js",
  "scripts": {
    "build": "rollup -c",
    "buildes": "esbuild --bundle --sourcemap --minify --outfile=dist/index.js ./js/indexx.js",
    "watch": "rollup -c --watch",
    "preview": "lein compile && wrangler preview",
    "dev": "miniflare --watch --debug",
    "compileAuth": "rollup ./js/auth.js --file /target/auth.js --format umd --name 'auth'"
  },

terminal

gitpod /workspace/clojureworker $ cd worker
gitpod /workspace/clojureworker/worker $ npm run dev

> test-worker@ dev /workspace/clojureworker/worker
> miniflare --watch --debug


> buildes
> esbuild --bundle --sourcemap --minify --outfile=dist/index.js ./js/indexx.js


  dist/index.js      127.5kb
  dist/index.js.map  554.2kb

⚑ Done in 27ms
[mf:inf] Build succeeded
[mf:dbg] Options:
[mf:dbg] - Build Command: npm run buildes
[mf:dbg] - Scripts: dist/index.js
[mf:dbg] Reloading dist/index.js...
[mf:inf] Worker reloaded! (127.46KiB)
[mf:dbg] Watching .env, dist/index.js, js, package.json, wrangler.toml...
[mf:inf] Listening on :8787
[mf:inf] - http://127.0.0.1:8787
[mf:inf] - http://10.52.7.90:8787
3133
GET / 200 OK (9.27ms)
[mf:dbg] js/indexx.js changed, reloading...
[mf:dbg] Reloading dist/index.js...
[mf:inf] Worker reloaded! (127.46KiB)
3133
GET / 200 OK (2.29ms)

second console.log should be 31333

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    πŸ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. πŸ“ŠπŸ“ˆπŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❀️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.