Implement printing of stack traces on LLVM segfaults and aborts #80182

in42 · 2020-12-19T08:54:50Z

Implement #79153

Based on discussion, try to extend the rust_backtrace=1 feature to handle segfault or aborts in the llvm backend

rust-highfive · 2020-12-19T08:54:53Z

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @matthewjasper (or someone else) soon.

If any changes to this PR are deemed necessary, please add them as extra commits. This ensures that the reviewer can see what has changed since they last reviewed the code. Due to the way GitHub handles out-of-date commits, this should also make it reasonably obvious what issues have or haven't been addressed. Large or tricky changes may require several passes of review and changes.

Please see the contribution instructions for more information.

tmandry · 2021-01-09T02:03:31Z

r? @tmandry

Just saw this. I'll give it a full review next week. Thanks!

tmandry · 2021-01-09T02:05:52Z

@in42 It looks like you'll want a #[cfg(unix)] around this specific implementation, at least.

in42 · 2021-01-09T02:55:01Z

This is a very basic implementation, the stack trace printed is barely readable, I will improve it this weekend.

tmandry · 2021-01-13T01:17:28Z

This is a very basic implementation, the stack trace printed is barely readable, I will improve it this weekend.

That's why I was thinking we might want to use LLVM's hooks when using it as a backend; we already get all the nice stuff "for free." Not sure if it's actually feasible and reachable through a public API, though I remember thinking it was.

in42 · 2021-01-13T15:42:34Z

Yeah basically there is no point in re implementing all of the functionality in rust - will look into how the llvm functions can be used.

Aaron1011 · 2021-01-16T16:00:35Z

You can also use std::backtrace::Backtrace instead of the libc backtrace function.

in42 · 2021-01-16T19:46:44Z

You can also use std::backtrace::Backtrace instead of the libc backtrace function.

Ok will look into this

in42 · 2021-01-17T14:23:32Z

You can also use std::backtrace::Backtrace instead of the libc backtrace function.

I tried to use it in the following way:

#[cfg(unix)]
fn print_stack_trace(_: libc::c_int) {
    let backtrace = std::backtrace::Backtrace::capture();
    println!("{}", backtrace);
}

and run the compiler like this:

$ RUST_BACKTRACE=1 build/x86_64-unknown-linux-gnu/stage1/bin/rustc -Aunused -g a.rs
[1]    2202274 segmentation fault (core dumped)  RUST_BACKTRACE=1 build/x86_64-unknown-linux-gnu/stage1/bin/rustc -Aunused -g

but no backtrace is getting printed.
Probably I will need to make the backtrace variable static. However, Backtrace::capture() does not take a reference to an existing backtrace variable for the output.

in42 · 2021-01-17T14:28:07Z

Using libc::backtrace function, the backtrace being printed was of this form:

$ build/x86_64-unknown-linux-gnu/stage1/bin/rustc -Aunused -g a.rs
[                                                                                                                                                                                                       
    0x00007f31c3ad5cfa,
    0x00007f31c312a6a0,                                                                                                                                                                                 
    0x00007f31c0cbc312,                                                                                                                                                                                 
    0x00007f31c0cbb619,
    ....
]

tmandry · 2021-01-26T03:24:32Z

Hm, maybe if we get something like #77384 merged we could use that.

in42 · 2021-01-26T22:28:31Z

Hm, maybe if we get something like #77384 merged we could use that.

Yeah, that will be better, I won’t have to implement the backtrace myself. 😃

tmandry · 2021-02-06T01:56:24Z

Talking with @yaahc gave me another idea: call the panic hook! The default panic hook prints a backtrace without allocating. If we call that we'll get a backtrace, and then we can abort.

The compiler does install another panic hook, but saves a handle to the default hook here. So we can call it directly from librustc_driver.

Worth a shot, I think..

in42 · 2021-02-13T04:57:37Z

Talking with @yaahc gave me another idea: call the panic hook! The default panic hook prints a backtrace without allocating. If we call that we'll get a backtrace, and then we can abort.

The compiler does install another panic hook, but saves a handle to the default hook here. So we can call it directly from librustc_driver.

Worth a shot, I think..

ok will try this

tmandry · 2021-02-24T23:40:47Z

@in42 following up here, have you had a chance to try the panic hook approach?

in42 · 2021-02-25T09:24:51Z

@in42 following up here, have you had a chance to try the panic hook approach?

Sorry was a little busy these last few weeks, will try this weekend.

in42 · 2021-03-06T22:27:03Z

Hi, I had a doubt, will the DEFAULT HOOK able to print backtrace if the functions are not from rust but from cpp part of the code? Also I am having a little trouble understanding how to populate the PanicInfo struct to send to the DEFAULT_HOOK.

in42 · 2021-03-06T22:32:42Z

On the other hand, I found out I just need to port this function to rust and I will able to print backtrace with function names based on the approach I was following earlier.

It uses the llvm-symbolizer binary to symbolize the backtrace. I am not sure whether it will be available for rustc though. Or I can try to directly call the llvm library function from rust code. The relevant code is present here.

in42 · 2021-03-07T13:33:50Z

@bjorn3 informed backtrace uses addr2line on zulip. The backtrace crate is used by libstd to provide backtraces for RUST_BACKTRACE=1.

bjorn3 · 2021-03-07T13:57:09Z

compiler/rustc_driver/src/lib.rs

+fn print_stack_trace(_: libc::c_int) {
+    unsafe {
+        static mut STACK_TRACE: [*mut libc::c_void; 256] = [std::ptr::null_mut(); 256];
+        let _depth = libc::backtrace(STACK_TRACE.as_mut_ptr(), 256);


Libc's backtrace function does a naive walk of frame pointers. Rustc and LLVM are normally compiled with frame pointers omitted, meaning that only DWARF based unwinding is reliable.

https://www.gnu.org/software/libc/manual/html_node/Backtraces.html

[...] frame pointer elimination will stop backtrace from interpreting the stack contents correctly.

In addition it is not safe to call it from asynchronous signals like SIGSEGV.

[...] AS-Unsafe init heap dlopen plugin lock [...]

Yeah I noticed inside the llvm code, using the backtrace c api was one of the ways they were generating the backtrace. I will try to add the other ways too.

@bjorn3 Here, llvm uses the backtrace function to get the backtrace. If it fails it tries to use _Unwind_Backtrace.

tmandry · 2021-03-09T02:38:36Z

Hi, I had a doubt, will the DEFAULT HOOK able to print backtrace if the functions are not from rust but from cpp part of the code? Also I am having a little trouble understanding how to populate the PanicInfo struct to send to the DEFAULT_HOOK.

Sorry, forgot about that. It's a bit hacky, but it seems like the most straightforward way to follow this approach would be, roughly:

reinstall the default hook (change the type of DEFAULT_HOOK so we can get it back somehow.. probably wrap it in Mutex<Option<..>>)
actually panic.

We can exit the process afterward so don't have to worry about cleaning up.

in42 · 2021-03-14T23:01:36Z

Ok as far as I have understood the panic hook approach, the other panic hook the compiler installs calls the default hook here, so I do not reinstall it. I can actually just panic, and the default hook will be called. So I tried that using this code:

fn print_stack_trace(_: libc::c_int) {
    panic!("help");
    // unsafe {
    //     static mut STACK_TRACE: [*mut libc::c_void; 256] = [std::ptr::null_mut(); 256];
    //     let depth = libc::backtrace(STACK_TRACE.as_mut_ptr(), 256);
    //     if depth == 0 {
    //         return;
    //     }
    //     backtrace_symbols_fd(STACK_TRACE.as_ptr(), depth, 2);
    // }
}

and ran the compiler like this:

$ RUST_BACKTRACE=1 build/x86_64-unknown-linux-gnu/stage1/bin/rustc -Aunused -g a.rs
thread '<unnamed>' panicked at 'help', compiler/rustc_driver/src/lib.rs:1333:5
stack backtrace:
[1]    3509283 segmentation fault (core dumped)  RUST_BACKTRACE=1 build/x86_64-unknown-linux-gnu/stage1/bin/rustc -Aunused -g

However, as you can see, the backtrace is not getting printed.

tmandry · 2021-06-16T01:49:29Z

I took the liberty of squashing the intermediate commits and adding a few small fixes. With that I think this PR is ready to land.

@bors r+

tmandry · 2021-06-21T17:21:07Z

@bors r+

tmandry · 2021-06-21T17:28:08Z

@bors r+

bors · 2021-06-21T17:28:09Z

📌 Commit ec6a85a has been approved by tmandry

Implement printing of stack traces on LLVM segfaults and aborts Implement rust-lang#79153 Based on discussion, try to extend the rust_backtrace=1 feature to handle segfault or aborts in the llvm backend

bors · 2021-06-22T06:44:17Z

⌛ Testing commit ec6a85a with merge 12a28971e51436adf4d00c42bc7ddae1017f99e9...

bors · 2021-06-22T07:01:49Z

💔 Test failed - checks-actions

tmandry · 2021-07-02T04:39:24Z

@bors r+ rollup=iffy

bors · 2021-07-02T04:39:26Z

📌 Commit 162ed4d has been approved by tmandry

bors · 2021-07-02T05:40:57Z

⌛ Testing commit 162ed4d with merge 1aa6c7c...

bors · 2021-07-02T08:25:43Z

☀️ Test successful - checks-actions
Approved by: tmandry
Pushing 1aa6c7c to master...

rust-highfive assigned matthewjasper Dec 19, 2020

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Dec 19, 2020