investigate flaky test-http2-respond-file-error-pipe-offset #35881

Trott · 2020-10-30T14:14:01Z

Test: test-http2-respond-file-error-pipe-offset
Platform: several (alpine-last-latest-x64, ubuntu1804_sharedlibs_debug, probably others)
Console Output:

not ok 1252 parallel/test-http2-respond-file-error-pipe-offset
  ---
  duration_ms: 0.298
  severity: fail
  exitcode: 1
  stack: |-
    node:assert:885
        throw newErr;
        ^
    
    AssertionError [ERR_ASSERTION]: ifError got unwanted exception: EPIPE: broken pipe, write
        at /home/iojs/build/workspace/node-test-commit-linux/nodes/alpine-last-latest-x64/test/common/index.js:340:12
        at /home/iojs/build/workspace/node-test-commit-linux/nodes/alpine-last-latest-x64/test/common/index.js:377:15
        at close (node:fs:1417:11)
        at FSReqCallback.oncomplete (node:fs:171:23)
     {
      generatedMessage: false,
      code: 'ERR_ASSERTION',
      actual: [Error: EPIPE: broken pipe, write] {
        errno: -32,
        code: 'EPIPE',
        syscall: 'write'
      },
      expected: null,
      operator: 'ifError'
    }
  ...

Build Links:
https://ci.nodejs.org/job/node-test-commit-linux/nodes=alpine-last-latest-x64/37918/
test-joyent-alpine311_container-x64-1

The text was updated successfully, but these errors were encountered:

Trott · 2020-10-30T14:16:14Z

https://ci.nodejs.org/job/node-test-commit-linux-containered/23143/nodes=ubuntu1804_sharedlibs_debug_x64/console

test-joyent-ubuntu1804_sharedlibs_container-x64-4

00:14:22 not ok 1292 parallel/test-http2-respond-file-error-pipe-offset
00:14:22   ---
00:14:22   duration_ms: 0.438
00:14:22   severity: fail
00:14:22   exitcode: 1
00:14:22   stack: |-
00:14:22     node:assert:885
00:14:22         throw newErr;
00:14:22         ^
00:14:22     
00:14:22     AssertionError [ERR_ASSERTION]: ifError got unwanted exception: EPIPE: broken pipe, write
00:14:22         at /home/iojs/build/workspace/node-test-commit-linux-containered/test/common/index.js:340:12
00:14:22         at /home/iojs/build/workspace/node-test-commit-linux-containered/test/common/index.js:377:15
00:14:22         at close (node:fs:1417:11)
00:14:22         at FSReqCallback.oncomplete (node:fs:171:23)
00:14:22      {
00:14:22       generatedMessage: false,
00:14:22       code: 'ERR_ASSERTION',
00:14:22       actual: [Error: EPIPE: broken pipe, write] {
00:14:22         errno: -32,
00:14:22         code: 'EPIPE',
00:14:22         syscall: 'write'
00:14:22       },
00:14:22       expected: null,
00:14:22       operator: 'ifError'
00:14:22     }
00:14:22   ...

Trott · 2020-10-30T14:19:10Z

/ping @addaleax

Trott · 2020-10-30T14:20:51Z

/ping @tniessen Possibly a result of adding common.mustSucceed() recently? (Maybe that has uncovered a bug in the logic? Or maybe not. I'm doing a very superficial evaluation right now.)

Trott · 2020-10-30T14:22:43Z

Here's another one.

https://ci.nodejs.org/job/node-test-commit-plinux/35704/nodes=centos7-ppcle/console
test-osuosl-centos7-ppc64_le-2

00:08:00 not ok 1416 parallel/test-http2-respond-file-error-pipe-offset
00:08:00   ---
00:08:00   duration_ms: 0.146
00:08:00   severity: fail
00:08:00   exitcode: 1
00:08:00   stack: |-
00:08:00     node:assert:885
00:08:00         throw newErr;
00:08:00         ^
00:08:00     
00:08:00     AssertionError [ERR_ASSERTION]: ifError got unwanted exception: EPIPE: broken pipe, write
00:08:00         at /home/iojs/build/workspace/node-test-commit-plinux/nodes/centos7-ppcle/test/common/index.js:340:12
00:08:00         at /home/iojs/build/workspace/node-test-commit-plinux/nodes/centos7-ppcle/test/common/index.js:377:15
00:08:00         at close (node:fs:1417:11)
00:08:00         at FSReqCallback.oncomplete (node:fs:171:23)
00:08:00      {
00:08:00       generatedMessage: false,
00:08:00       code: 'ERR_ASSERTION',
00:08:00       actual: [Error: EPIPE: broken pipe, write] {
00:08:00         errno: -32,
00:08:00         code: 'EPIPE',
00:08:00         syscall: 'write'
00:08:00       },
00:08:00       expected: null,
00:08:00       operator: 'ifError'
00:08:00     }
00:08:00   ...

Trott · 2020-10-30T14:24:38Z

Last one for now.

https://ci.nodejs.org/job/node-test-commit-linuxone/23764/nodes=rhel7-s390x/consoleText
test-ibm-rhel7-s390x-4

not ok 1241 parallel/test-http2-respond-file-error-pipe-offset
  ---
  duration_ms: 0.67
  severity: fail
  exitcode: 1
  stack: |-
    node:assert:885
        throw newErr;
        ^
    
    AssertionError [ERR_ASSERTION]: ifError got unwanted exception: EPIPE: broken pipe, write
        at /home/iojs/build/workspace/node-test-commit-linuxone/nodes/rhel7-s390x/test/common/index.js:340:12
        at /home/iojs/build/workspace/node-test-commit-linuxone/nodes/rhel7-s390x/test/common/index.js:377:15
        at close (node:fs:1417:11)
        at FSReqCallback.oncomplete (node:fs:171:23)
     {
      generatedMessage: false,
      code: 'ERR_ASSERTION',
      actual: [Error: EPIPE: broken pipe, write] {
        errno: -32,
        code: 'EPIPE',
        syscall: 'write'
      },
      expected: null,
      operator: 'ifError'
    }
  ...

tniessen · 2020-10-30T14:34:13Z

/ping @tniessen Possibly a result of adding common.mustSucceed() recently? (Maybe that has uncovered a bug in the logic? Or maybe not. I'm doing a very superficial evaluation right now.)

@Trott Good question! When I updated this test, I assumed the call to writeFile is expected to succeed, which doesn't always seem to be the case. The previous mustCall() callback simply ignored the error.

If this error is expected to never occur, then switching to mustSucceed has uncovered a bug and we should investigate why this error occurs.
If this error is expected to occur sometimes, we need to switch back to mustCall(), but should add an explaining comment and an assertion regarding the type/code of the error.
If this error is expected to always occur, the test case should assert that it does.

This is now failing inconsistently across many platforms. This appears to be the result of the addition of mustSucceed being added to the test during testing refactoring. We should mark flaky until we have figured out what the issue is. Refs: nodejs#35881

This is now failing inconsistently across many platforms. This appears to be the result of the addition of mustSucceed being added to the test during testing refactoring. We should mark flaky until we have figured out what the issue is. Refs: #35881 PR-URL: #35883 Reviewed-By: Tobias Nießen <tniessen@tnie.de> Reviewed-By: Gireesh Punathil <gpunathi@in.ibm.com>

Trott · 2020-11-21T13:16:38Z

If this error is expected to never occur, then switching to mustSucceed has uncovered a bug and we should investigate why this error occurs.

If this error is expected to occur sometimes, we need to switch back to mustCall(), but should add an explaining comment and an assertion regarding the type/code of the error.

If this error is expected to always occur, the test case should assert that it does.

@addaleax Do you have any insight as to whether the error here should be expected to occur sometimes and can be ignored or not?

Trott · 2020-11-21T13:18:00Z

https://ci.nodejs.org/job/node-test-commit-linux/38406/nodes=alpine-latest-x64/consoleText

not ok 1267 parallel/test-http2-respond-file-error-pipe-offset # TODO : Fix flaky test
  ---
  duration_ms: 0.224
  severity: flaky
  exitcode: 1
  stack: |-
    node:assert:885
        throw newErr;
        ^
    
    AssertionError [ERR_ASSERTION]: ifError got unwanted exception: EPIPE: broken pipe, write
        at /home/iojs/build/workspace/node-test-commit-linux/nodes/alpine-latest-x64/test/common/index.js:340:12
        at /home/iojs/build/workspace/node-test-commit-linux/nodes/alpine-latest-x64/test/common/index.js:377:15
        at close (node:fs:1452:11)
        at FSReqCallback.oncomplete (node:fs:180:23)
     {
      generatedMessage: false,
      code: 'ERR_ASSERTION',
      actual: [Error: EPIPE: broken pipe, write] {
        errno: -32,
        code: 'EPIPE',
        syscall: 'write'
      },
      expected: null,
      operator: 'ifError'
    }
  ...

Trott · 2020-11-28T13:56:37Z

https://ci.nodejs.org/job/node-test-commit-linux/nodes=alpine-latest-x64/38508/consoleText

not ok 1267 parallel/test-http2-respond-file-error-pipe-offset # TODO : Fix flaky test
  ---
  duration_ms: 0.153
  severity: flaky
  exitcode: 1
  stack: |-
    node:assert:900
        throw newErr;
        ^
    
    AssertionError [ERR_ASSERTION]: ifError got unwanted exception: EPIPE: broken pipe, write
        at /home/iojs/build/workspace/node-test-commit-linux/nodes/alpine-latest-x64/test/common/index.js:340:12
        at /home/iojs/build/workspace/node-test-commit-linux/nodes/alpine-latest-x64/test/common/index.js:377:15
        at close (node:fs:1459:11)
        at FSReqCallback.oncomplete (node:fs:187:23)
     {
      generatedMessage: false,
      code: 'ERR_ASSERTION',
      actual: [Error: EPIPE: broken pipe, write] {
        errno: -32,
        code: 'EPIPE',
        syscall: 'write'
      },
      expected: null,
      operator: 'ifError'
    }
  ...

Trott · 2020-11-28T14:16:38Z

I'm able to replicate this locally on macOS by running many copies of the test at once:

$ tools/test.py -j96 --repeat 192 test/parallel/test-http2-respond-file-error-pipe-offset.js
=== release test-http2-respond-file-error-pipe-offset ===                     
Path: parallel/test-http2-respond-file-error-pipe-offset
node:assert:900
    throw newErr;
    ^

AssertionError [ERR_ASSERTION]: ifError got unwanted exception: EPIPE: broken pipe, write
    at /Users/trott/io.js/test/common/index.js:340:12
    at /Users/trott/io.js/test/common/index.js:377:15
    at close (node:fs:1459:11)
    at FSReqCallback.oncomplete (node:fs:187:23)
 {
  generatedMessage: false,
  code: 'ERR_ASSERTION',
  actual: [Error: EPIPE: broken pipe, write] {
    errno: -32,
    code: 'EPIPE',
    syscall: 'write'
  },
  expected: null,
  operator: 'ifError'
}
Command: out/Release/node /Users/trott/io.js/test/parallel/test-http2-respond-file-error-pipe-offset.js
=== release test-http2-respond-file-error-pipe-offset ===                     
Path: parallel/test-http2-respond-file-error-pipe-offset
node:assert:900
    throw newErr;
    ^

AssertionError [ERR_ASSERTION]: ifError got unwanted exception: EPIPE: broken pipe, write
    at /Users/trott/io.js/test/common/index.js:340:12
    at /Users/trott/io.js/test/common/index.js:377:15
    at close (node:fs:1459:11)
    at FSReqCallback.oncomplete (node:fs:187:23)
 {
  generatedMessage: false,
  code: 'ERR_ASSERTION',
  actual: [Error: EPIPE: broken pipe, write] {
    errno: -32,
    code: 'EPIPE',
    syscall: 'write'
  },
  expected: null,
  operator: 'ifError'
}
Command: out/Release/node /Users/trott/io.js/test/parallel/test-http2-respond-file-error-pipe-offset.js
=== release test-http2-respond-file-error-pipe-offset ===                     
Path: parallel/test-http2-respond-file-error-pipe-offset
node:assert:900
    throw newErr;
    ^

AssertionError [ERR_ASSERTION]: ifError got unwanted exception: EPIPE: broken pipe, write
    at /Users/trott/io.js/test/common/index.js:340:12
    at /Users/trott/io.js/test/common/index.js:377:15
    at close (node:fs:1459:11)
    at FSReqCallback.oncomplete (node:fs:187:23)
 {
  generatedMessage: false,
  code: 'ERR_ASSERTION',
  actual: [Error: EPIPE: broken pipe, write] {
    errno: -32,
    code: 'EPIPE',
    syscall: 'write'
  },
  expected: null,
  operator: 'ifError'
}
Command: out/Release/node /Users/trott/io.js/test/parallel/test-http2-respond-file-error-pipe-offset.js
[00:12|% 100|+ 189|-   3]: Done 
$

Trott · 2020-11-28T14:18:14Z

Because the test is sensitive to resources (based on #35881 (comment)) and it seems to be failing on the low-resource Alpine Linux, possible solutions (assuming the failure is expected in those situations) might be skip the test on low-resource machines or the dreaded move-to-sequential.

Trott · 2020-11-28T16:36:51Z

I think the situation we're in is this:

If this error is expected to occur sometimes, we need to switch back to mustCall(), but should add an explaining comment and an assertion regarding the type/code of the error.

It's possible for the reading end of the pipe to get the expected error
and break everything down before we're finished, so allow EPIPE but
no other errors.

Proposed fix: #36305

Fixes: #35881 PR-URL: #36305 Reviewed-By: Anna Henningsen <anna@addaleax.net>

Fixes: nodejs#35881 PR-URL: nodejs#36305 Reviewed-By: Anna Henningsen <anna@addaleax.net>

This is now failing inconsistently across many platforms. This appears to be the result of the addition of mustSucceed being added to the test during testing refactoring. We should mark flaky until we have figured out what the issue is. Refs: #35881 PR-URL: #35883 Reviewed-By: Tobias Nießen <tniessen@tnie.de> Reviewed-By: Gireesh Punathil <gpunathi@in.ibm.com>

Fixes: #35881 PR-URL: #36305 Reviewed-By: Anna Henningsen <anna@addaleax.net>

Trott added the flaky-test Issues and PRs related to the tests with unstable failures on the CI. label Oct 30, 2020

Trott added the http2 Issues or PRs related to the http2 subsystem. label Oct 30, 2020

MylesBorins mentioned this issue Oct 30, 2020

lib: add throws option to fs.f/l/statSync #33716

Closed

4 tasks

MylesBorins mentioned this issue Oct 30, 2020

test: mark test-http2-respond-file-error-pipe-offset flaky #35883

Closed

Trott mentioned this issue Nov 8, 2020

http2: move events to the JSStreamSocket #35772

Merged

3 tasks

Trott mentioned this issue Nov 28, 2020

test: fix flaky test-http2-respond-file-error-pipe-offset #36305

Merged

3 tasks

Trott closed this as completed in f178c5a Dec 7, 2020

danielleadams pushed a commit that referenced this issue Dec 7, 2020

test: fix flaky test-http2-respond-file-error-pipe-offset

a7e794d

Fixes: #35881 PR-URL: #36305 Reviewed-By: Anna Henningsen <anna@addaleax.net>

cjihrig pushed a commit to cjihrig/node that referenced this issue Dec 8, 2020

test: fix flaky test-http2-respond-file-error-pipe-offset

c9c72eb

Fixes: nodejs#35881 PR-URL: nodejs#36305 Reviewed-By: Anna Henningsen <anna@addaleax.net>

targos pushed a commit that referenced this issue Apr 25, 2021

test: fix flaky test-http2-respond-file-error-pipe-offset

cd1da67

Fixes: #35881 PR-URL: #36305 Reviewed-By: Anna Henningsen <anna@addaleax.net>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

investigate flaky test-http2-respond-file-error-pipe-offset #35881

investigate flaky test-http2-respond-file-error-pipe-offset #35881

Trott commented Oct 30, 2020 •

edited

Loading

Trott commented Oct 30, 2020

Trott commented Oct 30, 2020

Trott commented Oct 30, 2020

Trott commented Oct 30, 2020

Trott commented Oct 30, 2020

tniessen commented Oct 30, 2020 •

edited

Loading

Trott commented Nov 21, 2020

Trott commented Nov 21, 2020

Trott commented Nov 28, 2020

Trott commented Nov 28, 2020

Trott commented Nov 28, 2020

Trott commented Nov 28, 2020

investigate flaky test-http2-respond-file-error-pipe-offset #35881

investigate flaky test-http2-respond-file-error-pipe-offset #35881

Comments

Trott commented Oct 30, 2020 • edited Loading

Trott commented Oct 30, 2020

Trott commented Oct 30, 2020

Trott commented Oct 30, 2020

Trott commented Oct 30, 2020

Trott commented Oct 30, 2020

tniessen commented Oct 30, 2020 • edited Loading

Trott commented Nov 21, 2020

Trott commented Nov 21, 2020

Trott commented Nov 28, 2020

Trott commented Nov 28, 2020

Trott commented Nov 28, 2020

Trott commented Nov 28, 2020

Trott commented Oct 30, 2020 •

edited

Loading

tniessen commented Oct 30, 2020 •

edited

Loading