Maximum header size hit

[amlwwalker]

I am sending large headers to my cloudflare workers and want to simulate this locally using miniflare.

With normal node its possible to allow bigger headers like --max-http-header-size 15000 as an example

How can I tell miniflare to accept larger headers?

I wondered if one option was to use the sdk of miniflare and wrap it in a node app so that I then call it like node my-miniflare.js or something but thought I would ask for advice.

While I'm here, as couldn't find anything in docs:

• I have multiple workers I want to simulate, and am currently starting them each on a different miniflare port. Is there a way I could have them at different URLs on the same port? I have one worker that supplies an HTML file that has js in, which inturn calls another worker and while they are on different ports I get a CORS issue

[mrbbot]

Hey! 👋

A bit of background: modules support currently requires the --experimental-vm-modules flag. For cross-platform compatibility, Miniflare's CLI actually spawns a new Node process with that flag set passing through other command line arguments.

Therefore, to add Node flags like --max-http-header-size, you'll need to start the script the bootstrapper starts instead by replacing miniflare <flags> with node --experimental-vm-modules --max-http-header-size 15000 ./node_modules/miniflare/dist/cli.js <flags>.

You could definitely use Miniflare's API as you suggest though, see more details here: https://miniflare.dev/api.html. You'd just need to run your script with the right flags:

// my-miniflare.mjs
import { Miniflare } from "miniflare";

const mf = new Miniflare({ ... });

node  --experimental-vm-modules --max-http-header-size 15000 my-miniflare.mjs

There isn't currently an easy way to have your workers at different URLs on the same port. You'd need to add CORS headers as in this example (https://developers.cloudflare.com/workers/examples/cors-header-proxy). If you only wanted to enable this during development, you could add an additional binding:

if(globalThis.MINIFLARE) {
  // add CORS headers
}

$ node ... ./node_modules/miniflare/dist/cli.js --binding MINIFLARE=true

You could also setup a reverse proxy in front of Miniflare, but adding CORS headers is a much simpler solution.

[amlwwalker]

hey thanks for getting back to me so fast.

So just to make sure I understand:

Option 1

Clone the repo, then run node --experimental-vm-modules --max-http-header-size 15000 ./node_modules/miniflare/dist/cli.js <flags> from within the repo root? Although, sorry my knowledge of how modules work is quite bad, this path suggests that I am running from within node_modules of another project - dist directory doesn't exist in the root of this repo it doesn't seem, which suggests I need to build it first?

Option 2

1. create a new file (my-miniflare.mjs) and call it as you suggest. This seems closer to making a custom miniflare that I can "play" with than option 1.

this part you mention

if(globalThis.MINIFLARE) {
  // add CORS headers
}

I could do that import { Miniflare } from "miniflare"; and before const mf = new Miniflare({ ... }); ? Is that the idea then to allow CORS in my custom miniflare?

EDIT: Oh Are you saying that for CORS I add the headers to my worker script and not to miniflare? If so thats cool. And then you are saying that miniflare will create the variable globalThis.MINIFLARE that will exist in simulation?

There isn't currently an easy way to have your workers at different URLs on the same port.

Ahh ok, would be cool to pass multiple worker scripts, or a directory path, and it fires them all up on url parts based on their name or something

Thanks - this tool is sweet.

[mrbbot]

For Option 1, when you install Miniflare with npm install miniflare in your project, the node_modules/miniflare/dist directory will contain all the pre-built files, so there's no need to clone the repo. You should just be able to run that node ... command as is, assuming you haven't installed Miniflare globally.

EDIT: Oh Are you saying that for CORS I add the headers to my worker script and not to miniflare? If so thats cool. And then you are saying that miniflare will create the variable globalThis.MINIFLARE that will exist in simulation?

Yep, that's exactly it. Since globalThis.MINIFLARE will only exist in the simulation, you can disable CORS in production if you don't need it there.

[amlwwalker]

Ok so I am making some good progress here. Few things not quite working for me, realise this isn't so much an "issue" with miniflare, but other user's may benefit...

The following does indeed work

My command is node --experimental-vm-modules --max-http-header-size=28000 ../node_modules/miniflare/dist/cli.js --port 8789 cookie.js

However I notice here that I should be able to set the max header size as part of the server options, with something like http.createServer({maxHeaderSize: 16384},app); - I can see from the miniflare api that there is the createServer - and the cli is probably using that in a similar manner, so I guess the first question is does miniflare take generic options in some way that I can try passing this or, do I now need to use the api in this case?

In effect I want the cli as it is with some customisations...

my-miniflare.js looks like the following, but that isn't quite the cli as it doesn't have the -- style options available albeit I did try

import { Miniflare } from "miniflare";

var myArgs = process.argv;
console.log("all args ", myArgs)
if (myArgs.length == 0) {
    console.log("need a file name")
    process.exit(1)
} else {
    console.log("running script ", myArgs[myArgs.length-1])
}
const mf = new Miniflare({
    scriptPath: myArgs[myArgs.length-1]
});

const options = await mf.getOptions();
const port = options.port ?? 5000; // Use port 5000 by default
console.log("options", options)

mf.createServer().listen(port, () => { console.log("starting server on ", port) });

my guess though is here I could pass the aforementioned option in the createserver function but using this script then loses alot of the functionality of the default client.

I also had trouble with the CORS example previously talked about. I added response.headers.set("Access-Control-Allow-Origin", url.origin) to my response and used the example if event.method == OPTION code which seems to work, but only in preflight, not in main request.

my response now looks like

  let defaultResponse = new Response(responseBody, { status, headers });
    defaultResponse.headers.set("Access-Control-Allow-Origin", "*")
  return defaultResponse;

but that doesn't seem to cut it. Any ideas? Running their example directly fails aswell btw so had to do some detective work to see what the premise of their cors changes are, but to no avail

[mrbbot]

However I notice here that I should be able to set the max header size as part of the server options, with something like http.createServer({maxHeaderSize: 16384},app);

Miniflare's createServer won't let you pass extra options like at the moment unfortunately. I'll try add it in the next version. Does using your node ... command solve all your needs or is there something else you'd like from the API?

Are there any CORS errors in your browser's JavaScript console? They may shed some light on the issue. 🕵️

[mrbbot]

Hey! 👋 The first pre-release of Miniflare 2 has just been released, allowing you to pass a http.ServerOptions or https.ServerOptions object to Miniflare#createServer. You can find the full changelog here and install it with npm i miniflare@next -D. Please let me know if you have any other issues, and feel free to ask questions in the #miniflare channel of the Cloudflare Workers Discord server.