fs.openAsBlob() does not work properly for files > 2GB
#52585
Comments
joelrbrandt
changed the title
fs.openAsBlob does not work properly for files > 2GBfs.openAsBlob() does not work properly for files > 2GB
|
FWIW, this also reproduces on a root@f7a53c9fe57c:/# uname -a
Linux f7a53c9fe57c 6.6.16-linuxkit #1 SMP Fri Feb 16 11:54:02 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux
root@f7a53c9fe57c:/# node -v
v20.12.2
root@f7a53c9fe57c:/# dd if=/dev/random of=random.bin bs=1048576 count=3072
3072+0 records in
3072+0 records out
3221225472 bytes (3.2 GB, 3.0 GiB) copied, 8.45959 s, 381 MB/s
root@f7a53c9fe57c:/# node
Welcome to Node.js v20.12.2.
Type ".help" for more information.
> const b = await fs.openAsBlob("random.bin")
undefined
> b
Blob { size: 3221225472, type: '' }
> b.slice(2**31-1, 2**31)
Blob { size: 0, type: '' }
> b.slice(2**31, 2**31+1)
# node[15]: static void node::Blob::ToSlice(const v8::FunctionCallbackInfo<v8::Value>&) at ../src/node_blob.cc:247
# Assertion failed: args[0]->IsUint32()
----- Native stack trace -----
1: 0xcab09c node::Assert(node::AssertionInfo const&) [node]
2: 0xc65b38 node::Blob::ToSlice(v8::FunctionCallbackInfo<v8::Value> const&) [node]
3: 0xf1f228 v8::internal::FunctionCallbackArguments::Call(v8::internal::CallHandlerInfo) [node]
4: 0xf1f9e8 [node]
5: 0xf1fe00 v8::internal::Builtin_HandleApiCall(int, unsigned long*, v8::internal::Isolate*) [node]
6: 0x189c964 [node]
----- JavaScript stack trace -----
1: slice (node:internal/blob:266:21)
2: REPL4:1:3
3: runInThisContext (node:vm:136:12)
4: defaultEval (node:repl:598:22)
5: bound (node:domain:432:15)
6: runBound (node:domain:443:12)
7: onLine (node:repl:927:10)
8: emit (node:events:530:35)
9: emit (node:domain:488:12)
10: [_onLine] (node:internal/readline/interface:416:12)
Aborted
|
VoltrexKeyva
added
the
fs
|
Hi @joelrbrandt , In the implementation of |
|
@kylo5aby thanks for your quick response (and sorry for my slow follow-up).
I'm a bit confused by this part of your response. Where is this hinted? In the node implementation? I don't see any mention of a 32-bit limitation in the File API spec. So, it doesn't seem like the Independent of the issue with Interestingly, if I get a stream of the Blob (with See below where I do the following:
$ dd if=/dev/random of=random.bin bs=1048576 count=5120
5120+0 records in
5120+0 records out
5368709120 bytes transferred in 6.613665 secs (811760063 bytes/sec)
$ openssl dgst random.bin
SHA256(random.bin)= f80fac6d9dc913e33d2c6a69783fd8fc3b4ed18f9c844aca801358e437096e6c
$ node
Welcome to Node.js v20.12.2.
Type ".help" for more information.
> b = await fs.openAsBlob("random.bin")
Blob { size: 1073741824, type: '' }
> b.size
1073741824
> const { createHash } = require("node:crypto")
undefined
> const h = createHash("sha256")
undefined
> const s = b.stream()
undefined
> for await (const a of s) {
... h.update(a);
... }
undefined
> h.digest("hex")
'f80fac6d9dc913e33d2c6a69783fd8fc3b4ed18f9c844aca801358e437096e6c'Finally, here's a codepen that can be used to check that files > 4GB work fine in all major modern browsers (open console to see output): https://codepen.io/joelrbrandt/pen/wvZRXvY |
|
Finally, if
> b = Buffer.alloc(2**33)
Uncaught:
RangeError [ERR_OUT_OF_RANGE]: The value of "size" is out of range. It must be >= 0 && <= 4294967296. Received 8_589_934_592
at Function.alloc (node:buffer:389:3) {
code: 'ERR_OUT_OF_RANGE'
} |
|
Hi @joelrbrandt, Thank you for the information.
There maybe a bug in
|
I'm not sure if this is accurate, either. Trying to construct a new Blob from a set of ArrayBuffers that total a size greater than 4GB also fails: $ node
Welcome to Node.js v18.13.0.
Type ".help" for more information.
> const arrs = []
undefined
> for (let i = 0; i < 5 ; i++) {
... arrs.push(new Uint8Array(2**30))
... }
5
> arrs.reduce((acc, cur) => acc + cur.byteLength, 0)
5368709120
> const b = new Blob(arrs)
Uncaught:
RangeError [ERR_BUFFER_TOO_LARGE]: Cannot create a Buffer larger than 4294967295 bytes
at __node_internal_captureLargerStackTrace (node:internal/errors:491:5)
at new NodeError (node:internal/errors:400:5)
at new Blob (node:internal/blob:165:13) {
code: 'ERR_BUFFER_TOO_LARGE'
}However, this code also works fine in a browser, resulting in a 5GB |
|
You are right. In versions up to v21, on 64-bit systems, the maximum size of a Buffer was 4GB. However, in the version I last used, compiled from the main branch, this limit has been exceeded, that's why I said |
PR-URL: nodejs#52588 Refs: nodejs#52585 Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com>
PR-URL: nodejs#52588 Refs: nodejs#52585 Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com>
PR-URL: nodejs#52588 Refs: nodejs#52585 Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com>
Version
v21.7.3 (and v20.12.2)
Platform
Darwin bender.local 23.4.0 Darwin Kernel Version 23.4.0: Fri Mar 15 00:10:42 PDT 2024; root:xnu-10063.101.17~1/RELEASE_ARM64_T6000 arm64
Subsystem
fs
What steps will reproduce the bug?
Steps to reproduce bug:
dd if=/dev/random of=random.bin bs=1048576 count=3072const b = await fs.openAsBlob("random.bin")b.slice(2**31-2, 2**31-1)<-- worksb.slice(2**31-1, 2**31)<-- returns a slice of zero lengthb.slice(2**31, 2**31+1)<-- crashesFull textual output of doing this given below.
Also, for files greater than 4GB, the size of the blob is wrong. It appears to be exactly
4*(2**30)less than the actual size. For example, for a5*(2**30)byte file, the blob size is reported as1073741824, which is(2**30). And, for a file that is exactly4*(2**30), it reports a size of zero.How often does it reproduce? Is there a required condition?
Always reproduces.
What is the expected behavior? Why is that the expected behavior?
Can operate on the Blob normally.
What do you see instead?
Described above. See output below.
Additional information
The text was updated successfully, but these errors were encountered: