JIT: fix issue with unreachable blocks in redundant branch opt

[AndyAyersMS]

Update fgGetDomSpeculatively to return nullptr if the idom of a block is unreachable, even if other preds have refs.

Fixes #72767.

[ghost]

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Update fgGetDomSpeculatively to return nullptr if the idom of a block is unreachable, even if other preds have refs.

Fixes #72767.

Author:	AndyAyersMS
Assignees:	-
Labels:	area-CodeGen-coreclr
Milestone:	-

[AndyAyersMS]

@EgorBo PTAL
cc @dotnet/jit-contrib

[EgorBo]

@AndyAyersMS I assume a block may still have a valid real dominator since we didn't inspect other preds, but it means that current block->iDom is definitely compromised/stale, hoping it won't lead to terrible diffs 🙂

[AndyAyersMS]

@AndyAyersMS I assume a block may still have a valid real dominator since we didn't inspect other preds, but it means that current block->iDom is definitely compromised/stale, hoping it won't lead to terrible diffs 🙂

I didn't see any diffs locally but only based on a few SPMI collections as we're still rebuilding them.

There is an odd divergence between dominators and the flow graph preds in some cases. I don't yet know if it is just unreachable blocks, some issue where we don't model flow out of finallys properly, cases where can't get rid of some EH regions even if unreachable, or something else.

In this case BB07 through BB10 are unreachable and BB11 has a dominator BB06.

RBO discovers that BB04 must branch to BB05 and this makes BB06 also unreachable.

RBO then looks at BB11 and as part of that, fgGetSpeculativeDom selects BB10 as a plausible dominator since BB06 has no preds left. But the IR in BB10 has been skipped over during value numbering. When we try and unpack BB10's relop VN we get the assert above.

My fix stops RBO from looking at the VNs in BB10. It might be more robust to just screen out potential doms with NoVNs.

-----------------------------------------------------------------------------------------------------------------------------------------
BBnum BBid ref try hnd preds           weight    lp [IL range]     [jump]      [EH region]         [flags]
-----------------------------------------------------------------------------------------------------------------------------------------
BB01 [0000]  1                             1       [000..007)-> BB03 ( cond )                     i 
BB02 [0001]  1       BB01                  1       [007..010)-> BB04 (always)                     i idxlen 
BB03 [0002]  1       BB01                  1       [010..011)                                     i 
BB04 [0003]  2       BB02,BB03             1       [011..01E)-> BB06 ( cond )                     i 
BB05 [0004]  1       BB04                  1       [01E..01F)        (return)                     i bwd-target 
BB06 [0006]  1       BB04                  1       [023..033)-> BB11 (always)                     i 
BB07 [0009]  1  0    BB10                  0       [037..04A)                 T0      try { }     keep i try rare Loop hascall gcsafe bwd 
BB08 [0016]  1       BB07                  0       [???..???)-> BB14 (callf )                     i internal rare gcsafe 
BB09 [0017]  1       BB14                  0       [???..???)-> BB10 (ALWAYS)                     i internal rare Loop gcsafe KEEP 
BB10 [0011]  1       BB09                  0       [062..06A)-> BB07 ( cond )                     i rare gcsafe bwd 
BB11 [0013]  2       BB06,BB10             1       [06A..073)-> BB13 ( cond )                     i idxlen 
BB12 [0014]  1       BB11                  1       [073..07D)                                     i 
BB13 [0015]  2       BB11,BB12             1       [07D..07E)        (return)                     i 
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ funclets follow
BB14 [0010]  2     0 BB08                  0       [04A..062)        (finret)    H0 F finally { } keep i rare gcsafe flet bwd 
-----------------------------------------------------------------------------------------------------------------------------------------

[AndyAyersMS]

Failures all seem to be from #76041

[AndyAyersMS]

No diffs.