Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Implement instruction_addr_adjustment #42533

Merged
merged 11 commits into from
Feb 1, 2023
Merged

Conversation

Swatinem
Copy link
Member

Implements handling of instruction_addr_adjustment and attaching appropriate per-frame attributes when sending stack traces to symbolicator.

It implements the Sentry and Profiling side of RFC 43

The Symbolicator side is implemented in getsentry/symbolicator#948.
Well, instruction_addr_needs_adjustment is indeed a mouthful :-D Should we use a shorter name for the flag?

Implements handling of `instruction_addr_adjustment` and attaching
appropriate per-frame attributes when sending stack traces to symbolicator.

It implements the Sentry and Profiling side of [RFC 43](getsentry/rfcs#43)
Copy link
Contributor

@loewenheim loewenheim left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should the possible values for "adjustment" be documented somewhere?

@github-actions
Copy link
Contributor

This pull request has gone three weeks without activity. In another week, I will close it.

But! If you comment or otherwise update it, I will reset the clock, and if you label it Status: Backlog or Status: In Progress, I will leave it alone ... forever!


"A weed is but an unloved flower." ― Ella Wheeler Wilcox 🥀

viglia

This comment was marked as duplicate.

src/sentry/profiles/task.py Outdated Show resolved Hide resolved
@Swatinem
Copy link
Member Author

Swatinem commented Jan 26, 2023

There was a concern from profiling folks that this approach might introduce the slight chance that one deduplicated frame in the big list of frames can be both a leaf and non-leaf.

That happens when the leaf (top most) frame of a stack happens to be the instruction following a call. So if the stack is sampled right after a ret, as in this example:

stack: [0x000a, 0x000f]
-> cpu executes `ret`, and jumps from `0x000f` to `0x000a` (`0x000a` was popped from stack memory)
stack: [0x000a]

In this edge case, the same addr should be adjusted in the first stack, but not the second.

We could solve this in two ways:

  1. De-normalize / materialize all the stacks. This means that instead of a single large list of frames, we send individual stacks. This means that payloads from profiling to symbolicator increase in size, and that symbolicator has more work to do. Symbolication itself is fast, though applying source context might be slow. (NOTE to self: we might want to add a request option to symbolicator to not apply source context if it is not needed)

  2. Duplicate just the leaf frames, and mark them explicitly as not needing adjustment.

In this proposal, we would still send a single large list to symbolicator, and only M leaf frames (for M stacks) are duplicated.
The single large list would look like this: [0..N, 0..M], where 0..N is the large list of frames as it exists now.
We would then just append 0..M at the end, duplicating each leaf frames and marking it as adjust: false.
As M is the number of stacks, we can just pop that appended sub-list off the end and use that for the leafs. Non-leafs are being looked up in the 0..N list based on their index just like now.

@loewenheim
Copy link
Contributor

Proposal 2 sounds reasonable. Should we implement this in Symbolicator first?

@Swatinem
Copy link
Member Author

Symbolicator does not need to know about that, this is purely work in the profiling processor.

@loewenheim
Copy link
Contributor

Derp, right. Sorry, had a brain fart.

},
"debug_meta": {"images": []},
}

_, stacktraces = _prepare_frames_from_profile(profile)
assert profile["profile"]["stacks"] == [[3, 0], [4, 1, 2]]
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are correctly asserting that _prepare_frames_from_profile is mutating its parameter which is nice.
However, is that mutation also being observed end to end when the request comes back from symbolicator?

frames.append(frame)
stack[0] = len(frames) - 1

stacktraces = [{"registers": {}, "frames": [dict(frame) for frame in frames]}]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not transform the frame to a dict in the append above so you don't have to loop once more?

]
stacktraces = []
for s in profile["sampled_profile"]["samples"]:
frames = [dict(frame) for frame in s["frames"]]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure why you're casting as a dict again, it's already a dict.

@loewenheim
Copy link
Contributor

@phacops I removed the dict-casting.

@Swatinem Swatinem merged commit ce0bf92 into master Feb 1, 2023
@Swatinem Swatinem deleted the feat/instr-adjustment branch February 1, 2023 08:00
mikejihbe pushed a commit that referenced this pull request Feb 6, 2023
Implements handling of `instruction_addr_adjustment` and attaching
appropriate per-frame attributes (`adjust_instruction_addr`) when sending stack traces to
symbolicator.

It implements the Sentry and Profiling side of [RFC
43](getsentry/rfcs#43)

The Symbolicator side is implemented in
getsentry/symbolicator#948.

---------

Co-authored-by: Sebastian Zivota <loewenheim@mailbox.org>
@github-actions github-actions bot locked and limited conversation to collaborators Feb 16, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Scope: Backend Automatically applied to PRs that change backend components
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants