Pass memory references from Android Native (C++) to Javascript and read data.

[david-metrica]

I have a plugin that uses swig to call a method that extracts images from a video using LibAV and C++.
The pointer to the AVFrame's data can be passed to Java as a long.
Is there a way I can access this memory from javascript (ArrayBuffer) without having to copy the values?
Some videos have very high resolutions and having to send this images using the bridge + serialization affects the performance.

[breautek]

Is there a way I can access this memory from javascript (ArrayBuffer) without having to copy the values?

No not really. The JS engine memory is isolated and not accessible from outside the engine. So if wanted to actually read or manipulate the memory in JS, the memory needs to be copied. If this was a NodeJS environment, the answer will be kind of different but webview JS environments has no concept of native c/c++ addons (even with the introduction of WASM). This includes the android webview / wkwebview on iOS.

It would likely be better to keep all memory access/manipulation done on the native side. You can pass that "pointer" to the JS as a way to keep context.

If you do this, it is important to note that JS numbers are 64-bit doubles... the integer is represented with 53-bits. This means that JS engines cannot safely represent the full range of a long (64-bit) integer. Therefore if you do pass the pointer to JS, it must be passed as a string, and then when passing back to native, the native should parse the string back to a long value. Failure to do this could cause "floating-point errors" on your pointer on the JS side, resulting in the pointer becoming corrupted and pointing at some other memory space instead.

A cordova plugin could maybe be crafted so that you can have a JS representation of your context and APIs to do manipulations on that memory, without passing or copying that memory through the cordova bridge... though kinda depends on your actual use cases.

[david-metrica]

What if the memory is created using: [Android NDK SharedMemory.h]
Can that memory be accessed using WebAssembly?

[breautek]

My experience with WebAssembly is low but I don't think so.

WebAssembly still lives in an isolated VM with some key differences. One is being is that it has a super-low level format allowing several compilers to take code written for other languages and being able to compile down to WebAssembly (which I'll refer to WASM from now on, easier to type).

In otherwords, I don't believe you can take external executable and get access to memory within the WASM VM, but theoretically you compile C++ code down to WASM so that everything runs inside the WASM environment.

In WebAssembly you can allocate memory which can be shared between the WASM environment and the JS environment. And you can compile C/C++ code to WASM using Emscripten compiler. I'm not entirely sure if you'll can access raw bytes of that memory object in webassembly but I'd assume that would be possible.

There are some limitations to this however...

1. Linking against external or shared libraries isn't supported by WASM. This means you can't actually run native code from a WASM module, nor can you import and use other WASM modules from a WASM module.
2. You'll be compiling for the WASM target, it won't be code running natively on android and all code must be statically link to create the WASM module. This also means you can't rely on JNI or any other android native APIs either. If your LibAV code depends on other dependencies which you can't compile yourself, this might be problematic. The idea here is you'll be compiling all your libav code into one WASM module with a JS interface.
3. The WASM module is still a virtual machine code. It's similar to java bytecode in a way. The code is ran inside a runtime environment, which co-exists and can share memory with the JS engine. So porting JS code to WASM yields "near-native" performance and can see a performance increase. For you though, porting from actual native to WASM, you'll likely see a decrease in performance.
4. WASM environments memory spaces are 32-bit, not 64-bit which most of us have gotten used to since there isn't really many 32-bit devices nowadays. Pretty sure this means stuff like long will be a 32-bit int instead of a 64-bit int, even on 64-bit devices. Also means the maximum amount of memory that can be allocated in a WASM environment is approximately 4GB, assuming the underlying user agent has the capability of allocating 4GB.
5. WASM features that might be available in the Chrome browser may not actually be available in the android webview. e.g. WASM shared memory isn't supported in the android webview for whatever reason. Shared memory flag is to create memory and attach it to a SharedMemoryBuffer which is a JS concept for a buffer that can be used to share between web workers without copying the data within.

For what it's worth, I've dabbled with WASM for my own stuff to experiment and I've always ran into some kind of issues that made me decide it isn't worth it. No support for using WASM modules from another WASM module is obviously a huge maintainability issue for any large scale projects. Creating usable objects. I've briefly looked to see how the WASM-side can manipulate WASM memory and I can only find JS-side examples, which isn't very useful other than to read the data for results. Returning data other than JS primitives (e.g. strings or numbers) from WASM is something not really supported either, the last I've looked at WASM anyway.