Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

System.Net.Http.Unit.Tests failing under jitstress #78912

Closed
jakobbotsch opened this issue Nov 28, 2022 · 5 comments · Fixed by #78930
Closed

System.Net.Http.Unit.Tests failing under jitstress #78912

jakobbotsch opened this issue Nov 28, 2022 · 5 comments · Fixed by #78930
Assignees
Labels
arch-x64 area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI blocking-clean-ci-optional Blocking optional rolling runs JitStress CLR JIT issues involving JIT internal stress modes os-linux Linux OS (any supported distro)
Milestone

Comments

@jakobbotsch
Copy link
Member

Failure: https://dev.azure.com/dnceng-public/public/_build/results?buildId=94300&view=ms.vss-test-web.build-test-results-tab
Console log: https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-main-ab9b9480b24c424cbe/System.Net.Http.Unit.Tests/1/console.eb002831.log?helixlogtype=result

After some bisection it looks related to System.Net.Http.Unit.Tests.HPack.HPackRoundtripTests:HPackEncode(System.Net.Http.Headers.HttpHeaders,System.Text.Encoding):System.Memory`1[ubyte]. The bug repros with

export DOTNET_JitStressRange=27a5bb45-ffffffff

but not with

export DOTNET_JitStressRange=27a5bb46-ffffffff

and that hash corresponds to HPackEncode. But, strangely, it also does not repro with just that hash in the range.

@ghost ghost added the untriaged New issue has not been triaged by the area owner label Nov 28, 2022
@ghost
Copy link

ghost commented Nov 28, 2022

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

Failure: https://dev.azure.com/dnceng-public/public/_build/results?buildId=94300&view=ms.vss-test-web.build-test-results-tab
Console log: https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-main-ab9b9480b24c424cbe/System.Net.Http.Unit.Tests/1/console.eb002831.log?helixlogtype=result

After some bisection it looks related to System.Net.Http.Unit.Tests.HPack.HPackRoundtripTests:HPackEncode(System.Net.Http.Headers.HttpHeaders,System.Text.Encoding):System.Memory`1[ubyte]. The bug repros with

export DOTNET_JitStressRange=27a5bb45-ffffffff

but not with

export DOTNET_JitStressRange=27a5bb46-ffffffff

and that hash corresponds to HPackEncode. But, strangely, it also does not repro with just that hash in the range.

Author: jakobbotsch
Assignees: -
Labels:

area-System.Net.Http

Milestone: -

@jakobbotsch jakobbotsch added area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI and removed area-System.Net.Http untriaged New issue has not been triaged by the area owner labels Nov 28, 2022
@jakobbotsch jakobbotsch added this to the 8.0.0 milestone Nov 28, 2022
@jakobbotsch jakobbotsch self-assigned this Nov 28, 2022
@ghost
Copy link

ghost commented Nov 28, 2022

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

Issue Details

Failure: https://dev.azure.com/dnceng-public/public/_build/results?buildId=94300&view=ms.vss-test-web.build-test-results-tab
Console log: https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-main-ab9b9480b24c424cbe/System.Net.Http.Unit.Tests/1/console.eb002831.log?helixlogtype=result

After some bisection it looks related to System.Net.Http.Unit.Tests.HPack.HPackRoundtripTests:HPackEncode(System.Net.Http.Headers.HttpHeaders,System.Text.Encoding):System.Memory`1[ubyte]. The bug repros with

export DOTNET_JitStressRange=27a5bb45-ffffffff

but not with

export DOTNET_JitStressRange=27a5bb46-ffffffff

and that hash corresponds to HPackEncode. But, strangely, it also does not repro with just that hash in the range.

Author: jakobbotsch
Assignees: -
Labels:

area-CodeGen-coreclr

Milestone: -

@jakobbotsch jakobbotsch added blocking-clean-ci-optional Blocking optional rolling runs JitStress CLR JIT issues involving JIT internal stress modes os-linux Linux OS (any supported distro) arch-x64 labels Nov 28, 2022
@jakobbotsch
Copy link
Member Author

The minimal failure range is export DOTNET_JitStressRange=27a5bb45-8d8abc88. The upper hash corresponds to System.Net.ArrayBuffer:get_AvailableSpan():System.Span`1[ubyte]:this

@jakobbotsch
Copy link
Member Author

On first glance, the following diff in get_AvailableSpan when stress is enabled looks suspicious:

 ; rbp based frame
 ; partially interruptible
 ; No PGO data
-; 1 inlinees with PGO data; 3 single block inlinees; 1 inlinees without PGO data
+; 1 inlinees with PGO data; 5 single block inlinees; 1 inlinees without PGO data
 ; Final local variable assignments
 ;
-;  V00 this         [V00,T00] (  4,  4   )   byref  ->  rdi         this single-def
-;# V01 OutArgs      [V01    ] (  1,  1   )  lclBlk ( 0) [rsp+00H]   "OutgoingArgSpace"
-;* V02 tmp1         [V02    ] (  0,  0   )  struct (16) zero-ref    do-not-enreg[SBR] multireg-ret "Return value temp for multireg return"
-;* V03 tmp2         [V03    ] (  0,  0   )  struct (16) zero-ref    multireg-ret "Inline return value spill temp"
-;  V04 tmp3         [V04,T01] (  4,  6   )     ref  ->  rsi         class-hnd single-def "Inlining Arg"
-;  V05 tmp4         [V05,T02] (  5,  5   )     int  ->  rbx         "Inlining Arg"
-;* V06 tmp5         [V06    ] (  0,  0   )  struct (16) zero-ref    ld-addr-op "NewObj constructor temp"
-;* V07 tmp6         [V07    ] (  0,  0   )  struct (16) zero-ref    ld-addr-op "Inline ldloca(s) first use temp"
-;  V08 tmp7         [V08,T03] (  3,  3   )     int  ->  r14         "Inlining Arg"
-;  V09 tmp8         [V09,T04] (  2,  2   )   byref  ->  r15         single-def "Inlining Arg"
-;* V10 tmp9         [V10    ] (  0,  0   )    bool  ->  zero-ref    "Inlining Arg"
-;* V11 tmp10        [V11    ] (  0,  0   )    bool  ->  zero-ref    "Inlining Arg"
-;* V12 tmp11        [V12    ] (  0,  0   )   byref  ->  zero-ref    V02._reference(offs=0x00) P-DEP "field V02._reference (fldOffset=0x0)"
-;* V13 tmp12        [V13    ] (  0,  0   )     int  ->  zero-ref    V02._length(offs=0x08) P-DEP "field V02._length (fldOffset=0x8)"
-;  V14 tmp13        [V14,T05] (  3,  1.50)   byref  ->  rax         V03._reference(offs=0x00) P-INDEP "field V03._reference (fldOffset=0x0)"
-;  V15 tmp14        [V15,T07] (  3,  1.50)     int  ->  rdx         V03._length(offs=0x08) P-INDEP "field V03._length (fldOffset=0x8)"
-;  V16 tmp15        [V16,T09] (  2,  1   )   byref  ->  rax         V06._reference(offs=0x00) P-INDEP "field V06._reference (fldOffset=0x0)"
-;  V17 tmp16        [V17,T10] (  2,  1   )     int  ->  rdx         V06._length(offs=0x08) P-INDEP "field V06._length (fldOffset=0x8)"
-;* V18 tmp17        [V18    ] (  0,  0   )   byref  ->  zero-ref    single-def V07._reference(offs=0x00) P-INDEP "field V07._reference (fldOffset=0x0)"
-;* V19 tmp18        [V19    ] (  0,  0   )     int  ->  zero-ref    V07._length(offs=0x08) P-INDEP "field V07._length (fldOffset=0x8)"
-;  V20 cse0         [V20,T08] (  3,  1.50)     int  ->  r14         "CSE - moderate"
-;  V21 cse1         [V21,T06] (  3,  1.50)     ref  ->  rsi         "CSE - moderate"
+;  V00 this         [V00,T02] (  3,  3   )   byref  ->  rdi         this single-def
+;  V01 tmp0         [V01,T11] (  1,  1   )     int  ->  [rbp-24H]   do-not-enreg[V] "GSCookie dummy"
+;# V02 OutArgs      [V02    ] (  1,  1   )  lclBlk ( 0) [rsp+00H]   "OutgoingArgSpace"
+;* V03 tmp2         [V03    ] (  0,  0   )  struct (16) zero-ref    do-not-enreg[SBR] multireg-ret "Return value temp for multireg return"
+;  V04 tmp3         [V04,T05] (  4,  2   )  struct (16) [rbp-38H]   do-not-enreg[SFR] multireg-ret must-init ptr "Inline return value spill temp"
+;  V05 tmp4         [V05,T00] (  4,  6   )     ref  ->  rsi         class-hnd single-def "Inlining Arg"
+;  V06 tmp5         [V06,T01] (  5,  5   )     int  ->  rbx         "Inlining Arg"
+;* V07 tmp6         [V07    ] (  0,  0   )  struct (16) zero-ref    ld-addr-op "NewObj constructor temp"
+;* V08 tmp7         [V08    ] (  0,  0   )  struct (16) zero-ref    do-not-enreg[S] ld-addr-op "Inline ldloca(s) first use temp"
+;  V09 tmp8         [V09,T12] (  3,  0   )     ref  ->  rbx         class-hnd exact single-def "NewObj constructor temp"
+;  V10 tmp9         [V10,T13] (  3,  0   )     ref  ->  rbx         class-hnd exact single-def "NewObj constructor temp"
+;  V11 tmp10        [V11,T04] (  3,  3   )     int  ->  r14         "Inlining Arg"
+;  V12 tmp11        [V12,T06] (  2,  2   )   byref  ->  r15         single-def "Inlining Arg"
+;* V13 tmp12        [V13    ] (  0,  0   )    bool  ->  zero-ref    "Inlining Arg"
+;* V14 tmp13        [V14    ] (  0,  0   )    bool  ->  zero-ref    "Inlining Arg"
+;* V15 tmp14        [V15    ] (  0,  0   )   byref  ->  zero-ref    V03._reference(offs=0x00) P-DEP "field V03._reference (fldOffset=0x0)"
+;* V16 tmp15        [V16    ] (  0,  0   )     int  ->  zero-ref    V03._length(offs=0x08) P-DEP "field V03._length (fldOffset=0x8)"
+;  V17 tmp16        [V17,T09] (  2,  1   )   byref  ->  r15         V07._reference(offs=0x00) P-INDEP "field V07._reference (fldOffset=0x0)"
+;  V18 tmp17        [V18,T10] (  2,  1   )     int  ->  r14         V07._length(offs=0x08) P-INDEP "field V07._length (fldOffset=0x8)"
+;  V19 GsCookie     [V19    ] (  1,  1   )    long  ->  [rbp-20H]   do-not-enreg[X] addr-exposed "GSSecurityCookie"
+;  V20 tmp19        [V20,T03] (  3,  3   )   byref  ->  rdi         single-def "shadowVar"
+;  V21 cse0         [V21,T08] (  3,  1.50)     int  ->  r14         "CSE - stress mode"
+;  V22 cse1         [V22,T07] (  3,  1.50)     ref  ->  rsi         "CSE - stress mode"
 ;
-; Lcl frame size = 8
+; Lcl frame size = 40
 
 G_M17271_IG01:
        push     rbp
        push     r15
        push     r14
        push     rbx
-       push     rax
-       lea      rbp, [rsp+20H]
-						;; size=12 bbWeight=1    PerfScore 5.50
+       sub      rsp, 40
+       vzeroupper 
+       lea      rbp, [rsp+40H]
+       xor      eax, eax
+       mov      qword ptr [rbp-38H], rax
+       mov      rax, 0xD1FFAB1E
+       mov      qword ptr [rbp-20H], rax
+						;; size=38 bbWeight=1    PerfScore 8.25
 G_M17271_IG02:
        mov      rsi, gword ptr [rdi]
        mov      ebx, dword ptr [rdi+0CH]
        test     rsi, rsi
-       je       SHORT G_M17271_IG06
+       je       SHORT G_M17271_IG08
 						;; size=11 bbWeight=1    PerfScore 5.25
 G_M17271_IG03:
        mov      r14d, dword ptr [rsi+08H]
        cmp      r14d, ebx
-       jb       SHORT G_M17271_IG07
+       jb       G_M17271_IG10
        mov      edi, ebx
        lea      r15, bword ptr [rsi+rdi+10H]
        sub      r14d, ebx
@@ -61,34 +67,57 @@ G_M17271_IG03:
 
        mov      rdi, rsi
        call     [System.Diagnostics.Debug:Fail(System.String,System.String)]
-						;; size=40 bbWeight=0.50 PerfScore 4.62
+						;; size=44 bbWeight=0.50 PerfScore 4.62
 G_M17271_IG04:
-       mov      rax, r15
-       mov      edx, r14d
-						;; size=6 bbWeight=0.50 PerfScore 0.25
+       mov      bword ptr [rbp-38H], r15
+       mov      dword ptr [rbp-30H], r14d
+						;; size=8 bbWeight=0.50 PerfScore 1.00
 G_M17271_IG05:
-       add      rsp, 8
+       mov      rax, bword ptr [rbp-38H]
+       mov      rdx, qword ptr [rbp-30H]
+       mov      rdi, 0xD1FFAB1E
+       cmp      qword ptr [rbp-20H], rdi
+       je       SHORT G_M17271_IG06
+       call     CORINFO_HELP_FAIL_FAST
+						;; size=29 bbWeight=1    PerfScore 6.25
+G_M17271_IG06:
+       nop      
+						;; size=1 bbWeight=1    PerfScore 0.25
+G_M17271_IG07:
+       add      rsp, 40
        pop      rbx
        pop      r14
        pop      r15
        pop      rbp
        ret      

Notice

+;  V04 tmp3         [V04,T05] (  4,  2   )  struct (16) [rbp-38H]   do-not-enreg[SFR] multireg-ret must-init ptr "Inline return value spill temp"

is marked must-init, yet we init only 8 bytes of it, and on return we load the length of this Span as if it was 8 bytes when it is actually 4 bytes.

@jakobbotsch
Copy link
Member Author

The above is probably ok since those are padding bits, so they are undefined. In reality the problem might be in the caller (HPackEncode) which generates:

       FF1566D908FF         call     [System.Net.ArrayBuffer:get_AvailableSpan():System.Span`1[ubyte]:this]
       BEFF000000           mov      esi, 255
       488BF8               mov      rdi, rax
       E8752C1C7C           call     CORINFO_HELP_MEMSET

This passes the length in the rdx register to CORINFO_HELP_MEMSET without any normalization. It seems there is a mismatch between STORE_DYN_BLK and CORINFO_HELP_MEMSET in the type of the length.

@ghost ghost added the in-pr There is an active PR which will close this issue when it is merged label Nov 28, 2022
jakobbotsch added a commit to jakobbotsch/runtime that referenced this issue Nov 28, 2022
STORE_DYN_BLK turns into a call to JIT_MemSet/JIT_MemCpy that go quite
directly to memset/memcpy, so the size is actually a native uint. This
can cause problems since the JIT does not make any normalization
guarantees above 32 bits.

Fix dotnet#78912
jakobbotsch added a commit that referenced this issue Nov 30, 2022
STORE_DYN_BLK turns into a call to JIT_MemSet/JIT_MemCpy that go quite
directly to memset/memcpy, so the size is actually a native uint. This
can cause problems since the JIT does not make any normalization
guarantees above 32 bits.

Fix #78912
@ghost ghost removed the in-pr There is an active PR which will close this issue when it is merged label Nov 30, 2022
@ghost ghost locked as resolved and limited conversation to collaborators Dec 30, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
arch-x64 area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI blocking-clean-ci-optional Blocking optional rolling runs JitStress CLR JIT issues involving JIT internal stress modes os-linux Linux OS (any supported distro)
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant