NativeAOT: Silent unwind info corruption #88292
Comments
ghost
added
the
|
Tagging subscribers to this area: @agocke, @MichalStrehovsky, @jkotas Issue DetailsOn macOS the unwind information is stored as the compact unwinding encoding and the DWARF EH encoding. The compact unwinding serves as a lookup table to the DWARF section (if the whole unwinding cannot be expressed using compact code, which NativeAOT doesn't currently produce). The "hint offset" into the DWARF table is 24-bit on both ARM64 and x64. Turns out, if the offset is longer, then it gets silently truncated and results in incorrect pointers into the DWARF section. This in turn results in unwinding not working properly and app freeze due to live lock between stuck Example stack trace: The
|
|
I guess we have three options how to deal with it:
|
|
Another option is to compensate for this issue in the libunwind implementation. We can loop over all candidates that match the hint. |
I don't think that's possible. The offset points into middle of a stream so it's essentially decoding garbage. Sometimes the garbage can make sense, sometimes not, but it's not easy to tell whether it's a false hit. |
|
Can we build our own hint table from the broken hint table? Something like: If the dwarf stream is more than 16MB:
This would fix our unwinder, but it would not fix other unwinders. For example, I would expect C++ EH to be still broken. |
|
We can probably build our own hint table from scratch during compilation. It needs to cover only "managed code" section and hence doesn't need much of a linker input, as long as the DWARF section is preserved in one piece (I think it is). Reconstructing the hints at runtime from the linker output may be possible but at that point you get a penalty similar to not using the hints at all and just creating the cache by sequentially reading the DWARF section. That's incredibly slow even on small executables though.
That's a fair point. I didn't consider other unwinders. If we want other unwinders to work then we basically have to either 1) generate compact unwinding codes where possible (needs codegen changes), 2) implement "compression" for DWARF by identifying common prolog sequences and sharing their code (possible in the DWARF format but difficult to implement), 3) fix it the linker output to not use the hints when they overflow (the slow lookup is too slow for NativeAOT purposes thought). I didn't check what the new Apple linker (ld-prime) produces in this case. |
The hint table is not big and the reconstructed hint table can be cached. I think the penalty would be fairly small. |
In the executable from OP the size of |
|
On second thought I don't think it's even reliably possible to reconstruct the DWARF offsets solely from |
|
There’s potentially an easy win in terms of the ARM64 DWARF size with folding the extremely common sequences into a DWARF CIE and referencing that. That’s a variation of the “DWARF compression” strategy mentioned earlier, just restricted to specific known sequences. For example, the prolog for frame with no callee saved registers (except LR and FP) is this: The code looks like this: This repeats 20000+ times in the OP executable. Besides being foldable in the DWARF codes it’s also likely expressible as compact unwind code with no codegen changes. We would still need to implement special prolog treatment for asynchronous unwinding with the compact unwind codes though, so the DWARF way could be easier (and benefit other platforms too). |
|
I tried to replace the empty frame DWARF sequence with compact unwinding and it saves 32% of the DWARF section size for this particular executable. Similar savings are present for empty iOS app from template ( |
On macOS the unwind information is stored as the compact unwinding encoding and the DWARF EH encoding. The compact unwinding serves as a lookup table to the DWARF section (if the whole unwinding cannot be expressed using compact code, which NativeAOT doesn't currently produce). The "hint offset" into the DWARF table is 24-bit on both ARM64 and x64. Turns out, if the offset is longer, then it gets silently truncated and results in incorrect pointers into the DWARF section. This in turn results in unwinding not working properly and app freeze due to live lock between stuck
FindMethodInfoand GC suspensions.Example stack trace:
The
fdeSectionOffsetHint=851328is0xCFD80. The DWARF dump is a bit too big too upload but 0xCFD80 points into a middle of a record. There is, however, a start of record at 0x10CFD80 and it matches the PC 0x10338DE20 from the stack trace:The text was updated successfully, but these errors were encountered: