wasm-pack should generate isomorphic code

[loynoir]

💡 Feature description

wasm-pack should generate isomorphic code

💻 Basic example

Below two targets have different output.

$ cd hello-world
$ wasm-pack build -t nodejs
$ wasm-pack build -t web

Differences are

• node don't have init, while browser have init

• node require TextDecoder from util, while browser check global

• node use readFileSync, while browser use fetch

• node generate CJS, while browser generate ESM

But,

• can embed, instead of init

• Quote https://nodejs.org/api/util.html, The TextDecoder class is also available on the global object

• And nodejs now support ESM for a long time. And now maintenance LTS is node16. Should drop CJS.

• I think, this should be implement within wasm-pack, instead of wasm-bindgen

So,

• I suggest, wasm-pack should embed the wasm generated by wasm-bindgen as base64 string, and generate same ESM code for node and web

Related

• emscripten support generate option SINGLE_FILE, which embed wasm as base64 string

• Option to embed .wasm as a base64 string in JavaScript file #831

• ESM target that is compatible with node #1253

• Add type = "module" in package.json to support nodejs in bundler mode #1039

• give option target=all which generates pkg supporting commonjs and esm #313

[cryptoquick]

Please 🙏

[currenthandle]

Bump. This would be incredible to have. Is anyone on the team working on this? Can we support in anyway? 🫡

[Denommus]

Isomorphic code would be amazing.

[loynoir]

Workaround, tested same code works on both node and browser.

wasm-pack build --target nodejs --out-dir ./dist/pkg && node ./patch.mjs

import { readFile, writeFile } from "node:fs/promises";

const name = "xxx";

const content = await readFile(`./dist/pkg/${name}.js`, "utf8");

const patched = content
  // use global TextDecoder TextEncoder
  .replace("require(`util`)", "globalThis")
  // inline bytes Uint8Array
  .replace(
    /\nconst path.*\nconst bytes.*\n/,
    `
var __toBinary = /* @__PURE__ */ (() => {
  var table = new Uint8Array(128);
  for (var i = 0; i < 64; i++)
    table[i < 26 ? i + 65 : i < 52 ? i + 71 : i < 62 ? i - 4 : i * 4 - 205] = i;
  return (base64) => {
    var n = base64.length, bytes = new Uint8Array((n - (base64[n - 1] == "=") - (base64[n - 2] == "=")) * 3 / 4 | 0);
    for (var i2 = 0, j = 0; i2 < n; ) {
      var c0 = table[base64.charCodeAt(i2++)], c1 = table[base64.charCodeAt(i2++)];
      var c2 = table[base64.charCodeAt(i2++)], c3 = table[base64.charCodeAt(i2++)];
      bytes[j++] = c0 << 2 | c1 >> 4;
      bytes[j++] = c1 << 4 | c2 >> 2;
      bytes[j++] = c2 << 6 | c3;
    }
    return bytes;
  };
})();

const bytes = __toBinary(${JSON.stringify(await readFile(`./dist/pkg/${name}_bg.wasm`, "base64"))
    });
`,
  );


// deal with `imports['__wbindgen_placeholder__']`
// TODO: optimize with `__wbg_get_imports`
const wrapped = `export default (function() {
  const module = { exports: {} };

  ${patched}

  return module.exports;
})()
`;

await writeFile(`./dist/${name}.mjs`, wrapped);

[microdou]

@loynoir Awesome work! I love how you made it isomorphic. Previously, I was doing something similar, inlining base64 to bring wasm-pack libraries (web target) into a web worker within a SvelteKit project (because compiled SvelteKit web worker has difficulty in fetching wasm file unless inlined). I didn't think it could be extended this far, so kudos for that! 🤩

I have a quick question though. Your approach allows importing the entire wasm-pack lib like so:

import wasm from "my-wasm-lib";
const { add } = wasm;

console.log(add(1, 2))

Would it be feasible to import individual functions directly? Something akin to:

import { add } from "my-wasm-lib";

console.log(add(1, 2))

[loynoir]

import { readFile, writeFile } from "node:fs/promises";

const cargoTomlContent = await readFile("./Cargo.toml", "utf8");
const cargoPackageName = /\[package\]\nname = "(.*?)"/.exec(cargoTomlContent)[1]
const name = cargoPackageName.replace(/-/g, '_')

const content = await readFile(`./dist/pkg/${name}.js`, "utf8");

const patched = content
  // use global TextDecoder TextEncoder
  .replace("require(`util`)", "globalThis")
  // attach to `imports` instead of module.exports
  .replace("= module.exports", "= imports")
  .replace(/\nmodule\.exports\.(.*?)\s+/g, "\nexport const $1 = imports.$1 ")
  .replace(/$/, 'export default imports')
  // inline bytes Uint8Array
  .replace(
    /\nconst path.*\nconst bytes.*\n/,
    `
var __toBinary = /* @__PURE__ */ (() => {
  var table = new Uint8Array(128);
  for (var i = 0; i < 64; i++)
    table[i < 26 ? i + 65 : i < 52 ? i + 71 : i < 62 ? i - 4 : i * 4 - 205] = i;
  return (base64) => {
    var n = base64.length, bytes = new Uint8Array((n - (base64[n - 1] == "=") - (base64[n - 2] == "=")) * 3 / 4 | 0);
    for (var i2 = 0, j = 0; i2 < n; ) {
      var c0 = table[base64.charCodeAt(i2++)], c1 = table[base64.charCodeAt(i2++)];
      var c2 = table[base64.charCodeAt(i2++)], c3 = table[base64.charCodeAt(i2++)];
      bytes[j++] = c0 << 2 | c1 >> 4;
      bytes[j++] = c1 << 4 | c2 >> 2;
      bytes[j++] = c2 << 6 | c3;
    }
    return bytes;
  };
})();

const bytes = __toBinary(${JSON.stringify(await readFile(`./dist/pkg/${name}_bg.wasm`, "base64"))
    });
`,
  );

await writeFile(`./dist/${name}.mjs`, patched);

[microdou]

@loynoir Works brilliantly! Thanks!

[syczuan]

@loynoir Thanks for the solution, I'm using nuxt.js and I can use it in both server and client environments with just a little code modification
.replace("require('util')", "process.client?globalThis:require('util')")

[lvauvillier]

Thanks a lot for this script. I updated it to make it work for both exported functions and classes:

The following line is too generic:

.replace(/\nmodule\.exports\.(.*?)\s+/g, "\nexport const $1 = imports.$1 ")

It produces duplicated class identifier declarations:

class MyClass { }
//...
export const MyClass = imports.MyClass = MyClass;

I added these lines before to add the suffix "Class":

// to add before
.replace(/\nclass (.*?) \{/g, "\nclass $1Class {")
.replace(/\nmodule\.exports\.(.*?) = \1;/g, "\nexport const $1 = imports.$1 = $1Class")

It produces:

class MyClassClass { }
//...
export const MyClass = imports.MyClass = MyClassClass;

[torfmaster]

FYI: I created a draft PR here which might touch this issue: rustwasm/wasm-bindgen#4065

[ahaoboy]

This is a very common use case, why is it still not supported from 2019 to now?
swc implements a similar requirement using regular expressions
https://github.com/swc-project/swc/blob/main/bindings/binding_typescript_wasm/scripts/patch.mjs

A command line tool to bundle the products of wasm-pack into a single js file
https://github.com/ahaoboy/wasm-pack-inline

Related
#1074
#831
#699