blog: reducing tail latencies with auto yielding #422

carllerche · 2020-04-01T05:06:58Z

Rendered: https://deploy-preview-422--tokio.netlify.com/blog/2020-04-preemption/

Darksonn

Looks good! I have a few nitpicks on spelling and such:

content/blog/2020-04-preemption.md

Darksonn · 2020-04-01T10:27:35Z

content/blog/2020-04-preemption.md

+under load and adding threads would make the situation much worse. To combat
+this, the .NET thread pool uses [hill climbing][hill].


I feel like a few more words can be added about the hill climbing heuristic they use? I know what hill climbing is as I specialize in OR, but even that doesn't let me guess any further details on what they are measuring here.

I found https://mattwarren.org/2017/04/13/The-CLR-Thread-Pool-Thread-Injection-Algorithm/ which seems like a pretty good discussion of the specific hill-climbing approach used in CLR (at a glance). May be good as a second reference?

Darksonn · 2020-04-01T10:29:46Z

content/blog/2020-04-preemption.md

+the order of micro seconds to tens of milliseconds at most. In this case, any
+stutttering problem from a heuristic based scheduler will result in far greater
+latency variations.


Suggested change

the order of micro seconds to tens of milliseconds at most. In this case, any

stutttering problem from a heuristic based scheduler will result in far greater

latency variations.

the order of microseconds to tens of milliseconds at most. In this case, any

stuttering problem from a heuristic based scheduler will result in far greater

latency variations.

Darksonn · 2020-04-01T10:31:25Z

content/blog/2020-04-preemption.md

+<div style="text-align:right">&mdash;Carl Lerche</div>
+
+
+[0.2.14]: #


Remember to update this once it has been released.

Thanks for reminding me, i had already forgotten... I'll probably forget again anyway 😆

content/blog/2020-04-preemption.md

jonhoo · 2020-04-01T12:39:58Z

content/blog/2020-04-preemption.md

+variance.
+
+Currently, the answer to this problem is that the user of Tokio is responsible
+for adding yield points every so often. In practice, very few actually do this


Should we link to yield_now, and maybe also rust-lang/futures-rs#2047 ?

jonhoo · 2020-04-01T12:40:38Z

content/blog/2020-04-preemption.md

+for adding yield points every so often. In practice, very few actually do this
+and end up being vulnerable to this sort of problem.
+
+A common solution to this problem is preemption. OS threads will interrupt


"With normal OS threads, the kernel will interrupt..."

content/blog/2020-04-preemption.md

jonhoo · 2020-04-01T12:45:34Z

content/blog/2020-04-preemption.md

+task. Each Tokio resource (socket, timer, channel, ...) is now aware of this
+budget. As long as the task as budget remaining, the resource operates as it did
+previously. Each asynchronous operation (actions that users must `.await` on)
+decrement the task's budget. Once the task is out of budget, all resources will


This isn't really true though. It's only true if they await a tokio resource (the sentence says "Each asynchronous operation"). And I guess we also don't want to get into the details of how it's really every poll call, not every .await.

How would you update it... I guess I can say "all tokio resources".

content/blog/2020-04-preemption.md

jonhoo · 2020-04-01T12:54:24Z

This looks good overall! I think it'd be good to include a paragraph on "next steps", which would include:

We'd like for third-party resources to be able to participate.
We'd like to provide "sub-budgets" for sub-executors/manual poll impls.
It'd be cool if there was a way to extend this mechanism so that all futures could take advantage of it, even with custom executors.

Could even mention that the docs have already been written for the first two, and that they are just not exposed out of caution in case experience will make us want to change them.

LucioFranco · 2020-04-01T14:19:30Z

I took a read overall reads well, I agree with some of jon's points but overall +1 from my end!

hawkw

This looks really good! I gave it a copyediting pass and left suggestions on some minor grammar nits and typos.

Also, since this post has a lot of discussion of prior art & comparisons with other approaches, it would be nice if there were more references for statements about other schedulers. If it's not a lot of effort, I would love to see more links.

Otherwise, looking good!

hawkw · 2020-04-01T15:30:23Z

content/blog/2020-04-preemption.md

+Tokio's scheduler requires that the generated task state machine yields control
+back to the scheduler in order to multiplex tasks. Each `.await` call is an


TIOLI: I might rephrase this like

Suggested change

Tokio's scheduler requires that the generated task state machine yields control

back to the scheduler in order to multiplex tasks. Each `.await` call is an

In order to multiplex tasks, Tokio's scheduler requires that the generated task

state machine yields control back to the scheduler. Each `.await` call is an

Might also consider reframing this as a requirement of Rust's futures model, rather than of Tokio's scheduler in particular?

Note overlap with this.

content/blog/2020-04-preemption.md

hawkw · 2020-04-01T16:06:16Z

content/blog/2020-04-preemption.md

+system calls. This is roughly equivalent to the Tokio APIs
+[`spawn_blocking`][spawn_blocking] and [`block_in_place`][block_in_place].


The difference is that Go does this in the standard library, right?

Tokio does as well... for example tokio::fs. The difference being that Tokio provides access to these fns as it doesn't preempt.

hawkw · 2020-04-01T16:06:54Z

content/blog/2020-04-preemption.md

+scheduler automatically detect blocked tasks?". The short answer is: no. Doing
+so would result in the same stuttering problems as mentioned above. Also, Go has
+no need to have generalized blocked task detection because Go is able to
+preempt. What the Go scheduler **does** do is annotate potentially blocking


Is there something we can link to for more information on how Go annotates potentially blocking calls?

Also, doesn't Go inject yield points as well? Good references here are golang/go#10958 and golang/go#24543.

I mostly got this by reading the source...

I'm not sure what I can ref.

content/blog/2020-04-preemption.md

Co-Authored-By: Alice Ryhl <alice@ryhl.io>

Co-Authored-By: Eliza Weisman <eliza@buoyant.io>

Co-Authored-By: Alice Ryhl <alice@ryhl.io>

Co-Authored-By: Jon Gjengset <jon@thesquareplanet.com>

Co-Authored-By: Eliza Weisman <eliza@buoyant.io>

Co-Authored-By: Eliza Weisman <eliza@buoyant.io> Co-Authored-By: Jon Gjengset <jon@thesquareplanet.com> Co-Authored-By: Alice Ryhl <alice@ryhl.io>

hawkw

This looks good to me! I had some last notes that may be useful.

content/blog/2020-04-preemption.md

Co-Authored-By: Eliza Weisman <eliza@buoyant.io>

sfackler · 2020-04-01T22:52:38Z

content/blog/2020-04-preemption.md

+resources will again function normally.
+
+Let's go back to the echo server example from above. When the task is scheduled, it
+is assigned a budget of 128 operations pr "tick". The number 128 was picked


per, not pr

blog: reducing tail latencies with auto yielding

b224416

Darksonn reviewed Apr 1, 2020

View reviewed changes

jonhoo reviewed Apr 1, 2020

View reviewed changes

content/blog/2020-04-preemption.md Outdated Show resolved Hide resolved

jonhoo reviewed Apr 1, 2020

View reviewed changes

content/blog/2020-04-preemption.md Outdated Show resolved Hide resolved

jonhoo reviewed Apr 1, 2020

View reviewed changes

content/blog/2020-04-preemption.md Outdated Show resolved Hide resolved

jonhoo reviewed Apr 1, 2020

View reviewed changes

content/blog/2020-04-preemption.md Outdated Show resolved Hide resolved

jonhoo reviewed Apr 1, 2020

View reviewed changes

content/blog/2020-04-preemption.md Outdated Show resolved Hide resolved

hawkw reviewed Apr 1, 2020

View reviewed changes

carllerche and others added 14 commits April 1, 2020 12:18

Update content/blog/2020-04-preemption.md

c06a859

Co-Authored-By: Alice Ryhl <alice@ryhl.io>

Update content/blog/2020-04-preemption.md

8ae63c5

Co-Authored-By: Alice Ryhl <alice@ryhl.io>

Update content/blog/2020-04-preemption.md

d4e9232

Co-Authored-By: Alice Ryhl <alice@ryhl.io>

Update content/blog/2020-04-preemption.md

a10af3c

Co-Authored-By: Alice Ryhl <alice@ryhl.io>

Update content/blog/2020-04-preemption.md

6a2931f

Co-Authored-By: Eliza Weisman <eliza@buoyant.io>

Update content/blog/2020-04-preemption.md

fe6e915

Co-Authored-By: Eliza Weisman <eliza@buoyant.io>

Update content/blog/2020-04-preemption.md

3eb6853

Co-Authored-By: Alice Ryhl <alice@ryhl.io>

Update content/blog/2020-04-preemption.md

e25321e

Co-Authored-By: Alice Ryhl <alice@ryhl.io>

Update content/blog/2020-04-preemption.md

f1717c7

Co-Authored-By: Jon Gjengset <jon@thesquareplanet.com>

Update content/blog/2020-04-preemption.md

56e4561

Co-Authored-By: Eliza Weisman <eliza@buoyant.io>

Apply suggestions from code review

8782f9d

Co-Authored-By: Eliza Weisman <eliza@buoyant.io> Co-Authored-By: Jon Gjengset <jon@thesquareplanet.com> Co-Authored-By: Alice Ryhl <alice@ryhl.io>

more tweaks

9cffc8d

tweaks

382e9e4

more conclusion

27a770d

hawkw approved these changes Apr 1, 2020

View reviewed changes

content/blog/2020-04-preemption.md Outdated Show resolved Hide resolved

content/blog/2020-04-preemption.md Outdated Show resolved Hide resolved

content/blog/2020-04-preemption.md Outdated Show resolved Hide resolved

content/blog/2020-04-preemption.md Outdated Show resolved Hide resolved

carllerche and others added 2 commits April 1, 2020 14:14

even more conclusion

944ceb6

Apply suggestions from code review

f543416

Co-Authored-By: Eliza Weisman <eliza@buoyant.io>

ref release

92408bd

carllerche merged commit fff01c5 into master Apr 1, 2020

sfackler reviewed Apr 1, 2020

View reviewed changes

carllerche deleted the preemption-blog branch July 21, 2020 20:01

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

blog: reducing tail latencies with auto yielding #422

blog: reducing tail latencies with auto yielding #422

carllerche commented Apr 1, 2020 •

edited

Loading

Darksonn left a comment

Darksonn Apr 1, 2020 •

edited

Loading

hawkw Apr 1, 2020

Darksonn Apr 1, 2020

Darksonn Apr 1, 2020

carllerche Apr 1, 2020

jonhoo Apr 1, 2020

jonhoo Apr 1, 2020

jonhoo Apr 1, 2020

carllerche Apr 1, 2020

jonhoo commented Apr 1, 2020 •

edited

Loading

LucioFranco commented Apr 1, 2020

hawkw left a comment

hawkw Apr 1, 2020

hawkw Apr 1, 2020

Darksonn Apr 1, 2020

hawkw Apr 1, 2020

carllerche Apr 1, 2020

hawkw Apr 1, 2020

jonhoo Apr 1, 2020

carllerche Apr 1, 2020

carllerche Apr 1, 2020

hawkw left a comment

sfackler Apr 1, 2020

		under load and adding threads would make the situation much worse. To combat
		this, the .NET thread pool uses [hill climbing][hill].

		<div style="text-align:right">—Carl Lerche</div>


		[0.2.14]: #

		Tokio's scheduler requires that the generated task state machine yields control
		back to the scheduler in order to multiplex tasks. Each `.await` call is an

		system calls. This is roughly equivalent to the Tokio APIs
		[`spawn_blocking`][spawn_blocking] and [`block_in_place`][block_in_place].

blog: reducing tail latencies with auto yielding #422

blog: reducing tail latencies with auto yielding #422

Conversation

carllerche commented Apr 1, 2020 • edited Loading

Darksonn left a comment

Choose a reason for hiding this comment

Darksonn Apr 1, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jonhoo commented Apr 1, 2020 • edited Loading

LucioFranco commented Apr 1, 2020

hawkw left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hawkw left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

carllerche commented Apr 1, 2020 •

edited

Loading

Darksonn Apr 1, 2020 •

edited

Loading

jonhoo commented Apr 1, 2020 •

edited

Loading