-
Notifications
You must be signed in to change notification settings - Fork 0
/
index.html
executable file
·965 lines (837 loc) · 34.9 KB
/
index.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Load Testing: A balanced approach you can actually do</title>
<meta name="description" content="Presentation from Pittsburgh Techfest 2015">
<meta name="author" content="Chris Winters">
<meta name="apple-mobile-web-app-capable" content="yes" />
<meta name="apple-mobile-web-app-status-bar-style" content="black-translucent" />
<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no">
<link rel="stylesheet" href="css/reveal.css">
<link rel="stylesheet" href="css/theme/sky.css" id="theme">
<!-- For syntax highlighting -->
<link rel="stylesheet" href="lib/css/zenburn.css">
<!-- Printing and PDF exports -->
<script>
var link = document.createElement( 'link' );
link.rel = 'stylesheet';
link.type = 'text/css';
link.href = window.location.search.match( /print-pdf/gi ) ? 'css/print/pdf.css' : 'css/print/paper.css';
document.getElementsByTagName( 'head' )[0].appendChild( link );
</script>
<style type="text/css">
html.img-left div.slide-background.present {
background-position: left bottom;
}
.lightly {
color: #cccccc;
}
</style>
<!--[if lt IE 9]>
<script src="lib/js/html5shiv.js"></script>
<![endif]-->
</head>
<body>
<div class="reveal">
<!-- Any section element inside of this container is displayed as a slide -->
<div class="slides">
<section>
<h1>Load Testing: Finding a workable balance</h1>
<p>
Chris Winters (<a href="http://twitter.com/cwinters">@cwinters</a>) - Turnitin
</p>
<p align="center">
<img src="lib/images/turnitin.png" alt="Turnitin logo" width="106" height="35"/>
</p>
<p>
Pittsburgh Techfest 2015 - <a href="http://goo.gl/MTZId4"><b>goo.gl/MTZId4</b></a>
</p>
</section>
<section>
<h2>Black magic</h2>
<p>
Is it?
</p>
<aside class="notes">
<p>Load testing has this aura of black magic around it. I think that
makes it difficult for normal humans (like me) to approach, much
less for us to integrate into our everyday practices.</p>
<p>I'm no expert, and I get things wrong all the time, and I have so much
to learn. But that doesn't mean we can't talk about what we know!</p>
</aside>
</section>
<section>
<h2>A story</h2>
<p>
<img src="lib/images/tell_me_a_mitzi.jpg" alt="Tell me a Mitzi" />
</p>
<aside class="notes">
<p>You have a great idea for a social recipes app. Like all good startups
you start with functionality -- you release it to friends and family, and
as you improve it you expand your circles of users, and expand it again,
and again. There's enough enthusiasm from everyone that there's a voice
in the back of your head saying, "How will we deal with half a million
users? Five million?" but you brush it aside as something that will
probably never happen anyway.</p>
<p>But it does. Not all at once, there's no Oprah Moment. But you grow in
fits and starts and before you know it you've got 600k active users. And
the system has been exhibiting weird slowdowns and failures every now and
then, but it's mostly manageable and there's still *so much to do* you
can't believe it. With many more people comes many more feature ideas --
good ones! -- and new ways to monetize the idea, which is good because
growth is expensive! So you focus on functionality because you need to
keep the doors open, and you size up your EC2 and RDS instances and
things get a little better.</p>
<p>And then you look up and you have 2m active users. And the slowdowns
aren't so weird anymore, but routine, even to the point where it's a
thing among your customers. (Instead of a fail whale it's a fail basket.)
They still dig you, but you feel like they'll move on given much more
trouble.</p>
<p>... are there good endings for this?</p>
</aside>
</section>
<section>
<h2>All about trade-offs</h2>
<p align="center">
<img src="lib/images/Fred_Brooks.jpg" alt="Fred Brooks" />
</p>
<aside class="notes">
<p>My wife is <b>so tired</b> of me saying this. But it's even more obvious in
software development than other areas of life because it's even part of
our software development process language. It's been clear for years
that there is No Silver Bullet. And it's easy to tut-tut at stories
like this and say it's obvious they should have done capacity planning,
and load testing.</p>
<p>But there are trade-offs for doing that early, we usually call those
opportunity costs. (Not to mention features that are cut-off because they're
not initially performant.) Also, people who understand performance of
many different parts of the system are <b>really</b> hard to find.
(Probably because most organizations don't value them well, but that's
another talk.)</p>
<p>I'm a classic fumbler with performance as well. So like most of us when
there are problems I tinker and poke and experiment. And that all takes
time, time that we're not building features, time that we're not figuring
out how to make money. It's a trade-off, and an easy one to get wrong.</p>
</aside>
</section>
<section>
<h2>Today: Learn enough to get by</h2>
<p>
Run through some types and expectations
</p>
<p>
Talk about generating and testing load
</p>
<p>
Sprinkled with side-effects.
</p>
<aside class="notes">
So what does that mean for today? We'll learn enough to get by. I'll
talk about some basics of load testing and describe some different
approaches, focusing on one that fits a lot of applications that many
standard tools don't, is deterministic, straightforward, and flexible but
not all-things-to-all-people.
</aside>
</section>
<section>
<h2>Virus: behavior and motivation</h2>
<aside class="notes">
<p>Which follows the other? Can you start with either?</p>
<p>It sounds inscrutable or pop-culturey, but there's a point. We'll
revisit. But it's something to keep in the back of your head: how
do things you do (practices) affect how you think about things (motivation)?
</p>
</aside>
</section>
<section>
<h2>Not an expert!</h2>
<p align="center">
<img src="lib/images/homer_simpson_pointing_by_frasier_and_niles.jpg" alt="Homer pointing" />
</p>
<aside class="notes">
Standard disclaimer. Also, note that I'll be talking about tools that
exercise your application via HTTP rather than automating a browser.
</aside>
</section>
<section>
<p>
<img src="lib/images/bunny_falling_over_at_desk.gif" alt="Awww...." />
</p>
<p>
<img src="lib/images/buffy-three-frame-closeup-glare.gif" alt="Look!" />
</p>
<aside class="notes">Take a drink of water!</aside>
</section>
<section>
<h2>Different types of load testing</h2>
<p>
<img src="lib/images/calvin_and_hobbes_19861126_load_limit_on_bridges.gif"
width="900" height="283"
alt="Calvin and Hobbes knows all" />
</p>
</section>
<section>
<h2>Type: Random access</h2>
<p>Every action is separate from every other action</p>
<p class="fragment">Examples: catalog, search, blog</p>
<p class="fragment">This is great! (if you can get it)</p>
<aside class="notes">
<p>Probably most popular type, and the one that most people
have experience with.</p>
<p>It's certainly possible to build stateful services supporting these
examples, but it's not an intrinsic part of the domain.</p>
<p>Stateless services are popular for a reason.</p>
</aside>
</section>
<section>
<h2>Random access ~~ Stress test</h2>
<p>
<img src="lib/images/stress_test_dolphin_jumping.jpg"
alt="What are the differences between the dolphins?" />
</p>
<aside class="notes">
<p>(Image - from site that labels this as a stress test: "Here's an image
with two dolphins; if they look different to you then you're
stressed.)</p>
<p>These are often known as stress testers. They're fairly
easy to setup -- in fact, many times all you need is to give them the
number of clients and a URL.</p>
</aside>
</section>
<section>
<h2>Random access tools: so many</h2>
<ul>
<li>Vegeta</li>
<li>http-perf</li>
<li>ab</li>
<li>wrk</li>
<li>Siege</li>
<li>Bees with machine guns</li>
<li>blitz.io</li>
<li>...</li>
</ul>
<aside class="notes">
<p>Very mature area of testing, with some tools approaching 20 years old.</p>
<p>New ones created all the time because it's easy and an interesting
problem to cut your concurrency teeth on.</p>
<p>Of particular interest are those that conveniently and transparently
marshal cloud resources to hit your app from many machines at once.</p>
<p>Types of reporting varies, but for the simple uses you want
pretty basic information.</p>
</aside>
</section>
<section>
<h2>Type: Varied workflow</h2>
<p>Every action may change based on the results of previous actions</p>
<p class="fragment">Examples: hospital admission, adaptive math</p>
<p class="fragment">Difficult but fun; potentially brittle</p>
<aside class="notes">
<p>Next-level load testing, required if you have a stateful application where
each result depends on earlier actions. You also work with users as <b>users</b>
vs treating their activity as a random bag of URLs.</p>
<p>Explain modern hospital admission form and the presence of rules engines
that generate different screens based on the combinations of factors;
similar for adaptive math.</p>
<p>Tests can be brittle because the representation of the logic to move
through the workflow is almost certainly separate from the business
logic.</p>
</aside>
</section>
<section>
<h2>Varied workflow ~~ simulation with smarts</h2>
<p>
<img src="lib/images/crazy_complicated_process.gif" alt="Crazy complicated process" />
</p>
<aside class="notes">
<p>It's possible for these to be straightforward, but they can also be an
Effort Pit. You're creating a simulation, and a simulation is code like
everything else.</p>
<p>State managment becomes an issue -- how much does your session have to
remember from action to action? Are you remembering facts or actions
(that can change in the presence of different facts.)</p>
<p>Simulating users in particular can be difficult -- you may have to
code state management that end users find trivial -- example of AccuNurse
here with interaction updates (meals?) coming in from other devices?
(maybe hospital admission: "if the patient has been in residence longer
than a week with a foo infection then the dietary protocol must change to
low-sodium...")</p>
<p>To be useful you have to know your users and their domain pretty well.</p>
</aside>
</section>
<section>
<h2>Varied workflow tools</h2>
<ul>
<li>Gatling</li>
<li>JMeter</li>
<li>Locust</li>
<li>Pylot</li>
<li>Multi-mechanize</li>
</ul>
<aside class="notes">
<p>Typically scripted in some manner (DSL, XML) with state management. There
are fewer of these because there's a lot more to them, and it's trickier
to get your abstractions right.</p>
<p>I've written three of these a long, long time ago (internal).</p>
</aside>
</section>
<section>
<h2>Type: Predictable workflow</h2>
<p>Every action may change but in ways knowable before running the test</p>
<p class="fragment">Examples: shopping checkout, student writing</p>
<p class="fragment">More straightforward</p>
<aside class="notes">
And then there are a lot of applications that are more workflow-y but not
as dynamic. The tool you use depends on how much state you need to carry
around, reports you want to generate, how much you love XML... Can work
well with authentication since you're treating users as users.
</aside>
</section>
<section>
<h2>Predictable workflow ~~ Scripted session simulation</h2>
<p>
<img src="lib/images/rail_sequence.jpg" alt="Wakeboarder rides rail" />
</p>
<aside class="notes">
This middle ground forces you to generate sessions, or steps of actions,
but the smarts to do this is typically far simpler than the runtime
simulation. It's also deterministic.
</aside>
</section>
<section>
<h2>Predictable workflow tools</h2>
<p>Both and neither...?</p>
<p class="fragment">
Random access tools support sessions poorly, if at all.
</p>
<p class="fragment">
Varied workflow tools can be complicated to use and brittle, <br />
possibly overkill for your app.
</p>
<aside class="notes">
<p>"support sessions poorly" - Random access (stress test) tools treat every URL as separate from
every other, and many just take a set of URLs an pound them randomly.
They also generally do not support per-request headers (Vegeta
excepted, see
<a href="https://github.com/tsenart/vegeta/pull/97">PR 97</a> and
<a href="https://github.com/tsenart/vegeta/pull/98">98</a>).
They're also focused on generating data for the entire activity vs
per session -- maybe your goal is to have a max time of activity per
session as opposed to a max time per individual transaction?</p>
<p>"brittle" - Much discussed earlier; idea of overkill is tricky, because you can
always take some of the strategies we talk about here and generate a
'static' script instead of relying on smarts.</p>
</aside>
</section>
<section>
<h2>As it happens...</h2>
<p>Scratching an itch is a thing</p>
<p class="fragment">
<img src="lib/images/korra_season_4.jpg" height="600" width="429" alt="Korra" />
</p>
<aside class="notes">
Korra is a fork of Vegeta, building on its concurrency model but adding
the idea of scripted sessions. These include the standard commands from
Vegeta along with the ability to pause the session between actions and
poll a URL until you get a desired response.
</aside>
</section>
<section>
<p>
<img src="lib/images/cat_jumps_at_scary_toast.gif" alt="Pop!" />
<br />
<img src="lib/images/cat_busted_taking_stuff_out_of_drawer.gif" alt="Busted!" />
</p>
<aside class="notes">Hydration is important!</aside>
</section>
<section>
<h2>What does engineering actually want?</h2>
<p>Tool/s that:</p>
<ul>
<li>allows us to simulate many users moving through this flow,</li>
<li>tells us where bottlenecks exist so that we can</li>
<li>drill down into problems using profiling or other tools</li>
</ul>
<aside class="notes">
<p>For us these are <b>ends</b> in and of themselves, partly because
we've internalized the side effects of these. Simulating users is an end
because we know what enables us to do other great things.</p>
</aside>
</section>
<section>
<h2>What does the world actually want?</h2>
<p>Tool/s that:</p>
<ul>
<li><strike>allows us to simulate many users moving through this flow,</strike></li>
<li><b>tells us where bottlenecks exist so that we can FIX THEM</b></li>
<li><strike>drill down into problems using profiling or other tools</strike></li>
</ul>
</section>
<section>
<h2>Our sample app: student writing</h2>
<p>This may sound familiar...</p>
<p class="fragment">...it's actually a good example!</p>
<p class="fragment">Plus, I'm lazy</p>
<aside class="notes">
<p>Our app is an Angular application, the backend just serves API
requests and has almost nothing to do with the front end</p>
</aside>
</section>
<section>
<h2>Who are the main users?</h2>
<ul>
<li>Our case is pretty easy - students</li>
<li class="fragment">Yours might not be -- maybe time for user research?
Did you see Stephanie's talk?</li>
<li class="fragment">Start somewhere! Simple means it's ok to toss</li>
</ul>
<aside class="notes">
<p>"ok to toss" - similar to code retreat; it's a valuable lesson -- Eric Evans
<a href="http://www.se-radio.net/2015/05/se-radio-episode-226-eric-evans-on-domain-driven-design-at-10-years/">talks</a>
about how "iteration" is not the same as "incremental". Iteration means
you do something, then throw it away and redo the same thing.
</p>
</aside>
</section>
<section>
<h2>What does a session look like?</h2>
<ol>
<li>Student logs in and gets list of assignments</li>
<li>Student chooses an assignment</li>
<li>Student writes incrementally over a session</li>
<li>Student checks score twice during session, including once at end</li>
</ol>
<aside class="notes">
<p>These are actions from the user point of view. It's useful to map
these out before doing anything technically -- before thinking about
implemetation, state management, anything.</p>
<p>Because at its core load testing is simulation. And you have to
have something to simulate</p>
</aside>
</section>
<section>
<h2>Simulating, step one</h2>
<p>1. Student logs in and gets list of assignments</p>
<p><img src="lib/images/ra_student_01_login.png" alt="login page"
width="440" height="286"/></p>
<ul>
<li>What API calls does this user action trigger?</li>
</ul>
</section>
<section>
<h2>Finding triggered calls</h2>
<p>
<img src="lib/images/ra_student_with_xhr_requests.png" alt="page with xhr requests"
width="850" height="501" />
</p>
<aside class="notes">
<p>Most people know about these, but it's worth bringing up as a tracing
tool so you can tie user actions in the front end to the API calls they
trigger along with the order.</p>
<p>Be sure to keep the window open to capture any periodic polls!</p>
</aside>
</section>
<section>
<h2>Simulating, step one</h2>
<p>1. <b>Student logs in</b> and gets list of assignments</p>
<ul>
<li>We're not actually doing this; we assume auth token exists</li>
<li>==> Our login process could be a bottleneck AND WE WOULDN'T KNOW</li>
</ul>
<aside class="notes">
<p>Our script generation process can either use an existing token or
create a new one</p>
<p>...of course, we can always test the login
process separately, just with a very short script that distributes
logins over a period of time</p>
<p>Why not just login as part of this? Korra can't remember state from
one request to another; it's a design constraint -- at least for now.</p>
</aside>
</section>
<section>
<h2>Simulating, step one</h2>
<p>1. Student logs in and <b>gets list of assignments</b></p>
<ul>
<li><code>/pages/students/assignments</code></li>
<li>'Fat' API call, lots of data</li>
</ul>
</section>
<section>
<h2>Simulating, step one</h2>
<p>
<img src="lib/images/ra_student_02_assignments.png"
width="879" height="573" alt="assignments" />
</p>
</section>
<section>
<h2>Simulating, step one</h2>
<pre>
GET /pages/students/assignments
<b>Authorization: Token {{student_token}}</b>
GET /api/classes/enrolled
Authorization: Token {{student_token}}
</pre>
<aside class="notes">
<p>1..n headers can be passed with every request</p>
</aside>
</section>
<section>
<h2>Concern one: thundering herd?</h2>
<p>Do we want all the students to login at once?</p>
<p>Hello <code>PAUSE</code></p>
</section>
<section>
<h2>Simulating, step one</h2>
<p>Stagger start over two minutes (120 sec)</p>
<pre>
PAUSE {{random (<b>0..120</b>) * 1000}}
GET /pages/students/assignments
Authorization: Token {{student_token}}
GET /api/classes/enrolled
Authorization: Token {{student_token}}
</pre>
<aside class="notes">
<p>We allow millisecond precision to allow for brief delays between
automated actions</p>
</aside>
</section>
<section>
<p>
<img src="lib/images/wookie-dance.gif" alt="Dance!" />
<br />
<img src="lib/images/West-Side-Story-snapping-fingers.gif" alt="Snap!" />
</p>
</section>
<section>
<h2>Simulating, step two</h2>
<p>2. <b>student chooses assignment</b> which loads previous drafts and scores</p>
<ul>
<li>Embedded as <code>{{assignment_id}}</code> in URLs</li>
</ul>
</section>
<section>
<h2>Simulating, step two</h2>
<p>2. student chooses assignment which <b>loads previous drafts and scores</b></p>
<pre>
GET /api/assignments/{{assignment_id}}/share?token={{assignment_token}}
Authorization: Token {{student_token}}
GET /pages/students/assignment_answers/{{assignment_id}}
Authorization: Token {{student_token}}
</pre>
</section>
<section>
<h2>Feedback into API design</h2>
<ul>
<li>These calls weren't always this way...</li>
<li>Second call originally ~~ 100x calls (WAY too granular)</li>
<li class="fragment">Side effect of building our simulations</li>
<li class="fragment">Make sure you broadcast nuggets like this!</li>
</ul>
<aside class="notes">
<p>"Side effect" - whenever side effects are virtuous I think I'm
on the right path. (This could be confirmation bias.) Another example
is Docker.</p>
<p>"Broadcast" -- good example of behavior leading to motiviation leading
to behavior. If you saw Carol and Jenny's talk this morning you saw
the importance of communicating these engineering concepts to the rest
of the business. Keeping a constant stream of subliminal successes is
a good way to do this, especially if you put it in terms of averted risk/danger.</p>
</aside>
</section>
<section>
<h2>Presenting results, tips</h2>
<ul>
<li>Always record the "before" for comparison</li>
<li class="fragment">Record it in the tool, ensure in ballpark of NewRelic (etc)</li>
<li class="fragment">Both "before" and "after" should be a representative sample, not
worst/best</li>
<li class="fragment">Some people think graphically, some with tables</li>
<li class="fragment">Boil down into user experience where possible</li>
<!-- TODO: spreadsheet from https://docs.google.com/a/iparadigms.com/spreadsheets/d/1BY9ZG2JhSVPOTYya9-BbSBT1UhH0DGq_CKjgKJTdVgE/edit?usp=sharing ?? -->
</ul>
<aside class="notes">
<p>
</aside>
</section>
<section data-background="lib/images/ella_writing.jpg"
data-background-size="500px" data-state="img-left">
<h2>Simulating, step three</h2>
<p>3. <b>Student writes incrementally over a session</b></p>
<ul>
<li>We autosave every eight secs...<b>if</b> anything has changed</li>
<li>Humans do not write at a uniform speed, it's bursty</li>
</ul>
<!--<img src="lib/images/ella_writing.jpg" alt="kid writing" />-->
</section>
<section>
<h2>Simulating, step three</h2>
<p>
<img src="lib/images/ra_student_04_writing_draft.png"
width="879" height="573" alt="writing draft" />
</p>
</section>
<section>
<h2>Simulating, step three</h2>
<p>For every handful of words, we'll generate:</p>
<pre>
PAUSE {{random (0..45) * 1000}}
POST /api/assignments/{{assignment_id}}/autosave
Authorization: Token {{student_token}}
Content-Type: application/json
<b>@posts/{{student_id}}/{{assignment_id}}_{{count}}.json</b>
</pre>
<aside class="notes">
<p>Content-Type is a new one, important to represent for back-end routing</p>
<p>The POST-ed content sits in the filesystem, which means it can be
zipped up with the script and sent to another client</p>
</aside>
</section>
<section>
<h2>So far...</h2>
<ul>
<li><code>PAUSE</code> at start to spread</li>
<li><code>GET</code> assignments and classes</li>
<li><code>GET</code> assignment details and answers</li>
<li><code>PAUSE</code> and <code>POST</code> autosave a number of times</li>
</ul>
</section>
<section>
<p>
<img src="lib/images/supercut_eyeglance_sideways.gif" alt="I see you!" />
<br />
<img src="lib/images/sewing_kitten.gif" alt="sew sew sew" />
</p>
</section>
<section>
<h2>Simulating, step four</h2>
<p>4. Student checks score twice during session, including once at end</p>
<p>
<img src="lib/images/ra_student_03_writing_signals.png"
width="879" height="573" alt="results of checking score" />
</p>
</section>
<section>
<h2>Simulating, step four</h2>
<pre>
PAUSE {{random (0..120) * 1000}}
POST /api/assignments/{{assignment_id}}/share
Authorization: Token {{student_token}}
Content-Type: application/json
@posts/{{student_id}}/{{assignment_id}}_signal_check.json
<b>POLL GET /pages/students/signal_check/latest
[Wait=2000 Count=5 Status=200]</b>
</pre>
<aside class="notes">
<p>If you've built a 'modern' app then you've got asynchronous calls somewhere.
And if you have those, you're probably polling. You're trying to simulate
how users impact the site, so you have to poll too.</p>
<p>Polling -- like infinite loops -- should have a stop condition. This has
two, but note that neither requires us to look at (or parse) the body.
Again, this is a constraint of Korra.</p>
</aside>
</section>
<section>
<h2>All together now (1/2)</h2>
<pre>
PAUSE {{random (0..120) * 1000}}
GET /pages/students/assignments
Authorization: Token {{student_token}}
GET /api/classes/enrolled
Authorization: Token {{student_token}}
GET /api/assignments/{assignment_id}/share?token={assignment_token}
Authorization: Token {{student_token}}
GET /pages/students/assignment_answers/{assignment_id}
Authorization: Token {{student_token}}
</pre>
</section>
<section>
<h2>All together now (2/2)</h2>
<pre>
for each handful of words in sample_essay:
PAUSE {{random (0..45) * 1000}}
POST /api/assignments/{{assignment_id}}/autosave
Authorization: Token {{student_token}}
Content-Type: application/json
@posts/{{student_id}}/{{assignment_id}}_{{count}}.json
for each signal_check in ['midway', 'end']:
PAUSE {{random (0..120) * 1000}}
POST /api/assignments/{{assignment_id}}/share
Authorization: Token {{student_token}}
Content-Type: application/json
@posts/{{student_id}}/{{assignment_id}}_signal_check.json
POLL GET /pages/students/signal_check/latest
[Wait=2000 Count=5 Status=200]
</pre>
</section>
<section>
<h2>What is success? Failure?</h2>
<ul>
<li>Before you do anything, define desired outcomes with your team</li>
<li class="fragment">What transaction times are you shooting for?</li>
<li class="fragment">...probably better, what times are you scared of?</li>
<li class="fragment">Be wary of load testing without these: effort pit!</li>
</ul>
</section>
<section>
<h2>So you've run some tests...</h2>
<ul>
<li>Results often communicated as average.</li>
<li class="fragment">Average is a lie!</li>
<li class="fragment">18 @ 100ms, 2 @ 2000ms = 5800ms / 20 = 290ms avg</li>
<li class="fragment">...but 10% of your transactions were ~7x the average!</li>
</ul>
</section>
<section>
<h2>If not average...</h2>
<ul>
<li>Instead use percentiles: 75, 95, 99 are common</li>
<li class="fragment">Be able to explain percentiles to normal people</li>
<li class="fragment">Distribution of experience, though sometimes you focus</li>
<li class="fragment">...plus the 95% is a leader to other problems</li>
</ul>
<aside class="notes">
<p>"be able to explain" - for: 75% at 60ms, 95% at 200ms, and 99% at
3200ms you can say that 75/100 people got a response in 60 ms or less,
95/100 got a response in 200ms or less, and only 1 or 2 got a response
in 3.2 seconds.</p>
<p>"distribution of experience" - you might be very happy with 9/10 of
your users getting a good experience; you might be happy with 75/100,
or 95/100; it's really about your tolerance for risk and how much users
are exposed to this. Maybe it's only during a 10 minute period of each
weekday when classes across the country are finishing classes before
lunch. It's annoying, but so are a lot of other things</p>
<p>"leader to other problems" - 95% may be an indicator not only that
your service is constrained, but something your service calls is
constrained (database, search index)</p>
</aside>
</section>
<section>
<h2>Hey, this looks ok!</h2>
<p>
<img src="lib/images/graph_nodes_to_median.png" alt="nice graph" />
</p>
<aside class="notes">
</aside>
</section>
<section>
<h2>Sad panda...</h2>
<p>
<img src="lib/images/graph_nodes_to_95.png" alt="ugh graph" />
</p>
<aside class="notes">
</aside>
</section>
<section>
<h2>Results</h2>
<ul>
<li>Results from tools can be rudimentary</li>
<li>...though they do include network latency</li>
<li class="fragment" data-fragment-index="1">Server-based tools can also be great - NewRelic</li>
<li class="fragment" data-fragment-index="1">Not perfect, but tools doing 80% with little work better than doing
100% with lots of work</li>
<li class="fragment">They have thought way more about this than you have</li>
<li class="fragment">Outsourcing tools scales your team and focuses your effort</li>
</ul>
</section>
<section>
<h2>Behavior and motivation stuff</h2>
<ul>
<li>Load testing is a virus (like many good practices)</li>
<li class="fragment">Do it all the time, you'll get better</li>
<li class="fragment">Learn more about how your app works</li>
<li class="fragment">Get a better feel for what's costly by screwing things up all the time</li>
<li class="fragment">Probably also learn more about profiling</li>
</ul>
<aside class="notes">
<p>"virus" - the behavior of load testing leads to other behaviors, which
change your views on performance</p>
<p>"do it all the time" - or as
<a href="http://martinfowler.com/bliki/FrequencyReducesDifficulty.html">Martin Fowler puts it</a>:
"If it hurts, do it more often"</p>
<p>"better feel" - aka, intuition - did you see the keynote this morning
about embracing failure?</p>
</aside>
</section>
<section>
<h2>Scaling</h2>
<p>Scale your team</p>
<p class="fragment" data-fragment-index="1">Scale yourself</p>
<p class="fragment" data-fragment-index="2">Agile, Devops, Lean, Startup</p>
<p class="fragment" data-fragment-index="2">Tools are not replacements for thinking</p>
<aside class="notes">
<p>Team: Provide environment for people to understand the impact of the change they're making.</p>
<p>Self: "Serial specialization" Manda mentioned during panel discussion before lunxh</p>
</aside>
</section>
<section data-background="lib/images/korra_looks_over_republic_city.png">
<h1 style="color: #000000; font-weight: bold">So much to do!</h1>
<aside class="notes">
<p>distributed clients and coordination</p>
<p>web interface for running</p>
<p>API for running (agent model)</p>
<p>reporting and visualization</p>
</aside>
</section>
<section>
<h2>Thank you!</h2>
<p><img src="lib/images/whiskeytime.gif" alt="that's it!" /></p>
<p>
Chris Winters (<a href="http://twitter.com/cwinters">@cwinters</a>)
<br />
Turnitin
</p>
<p align="center">
<img src="lib/images/turnitin.png" alt="Turnitin logo" width="106" height="35"/>
</p>
<p>
Permanent home: <a href="http://goo.gl/MTZId4"><b>goo.gl/MTZId4</b></a>
</p>
</section>
<section>
<h3>Image attribution</h3>
<ul>
<li><a href="https://mutteringretreat.wordpress.com/2014/03/18/tell-me-a-mitzi-by-lore-segal-illustrated-by-harriet-pincus/">Tell me a Mitzi</a>
from <a href="http://www.amazon.com/Tell-Me-Mitzi-Lore-Segal/dp/0590338013/">the book</a>
illustrated by Harriet Pincus and written by Lore Segal</li>
<li><a href="http://en.wikiquote.org/wiki/Fred_Brooks">Fred Brooks</a></li>
<li><a href="http://frasier-and-niles.deviantart.com/art/Homer-Simpson-06-Simpsons-196269904">Homer Simpson pointing</a></li>
<li><a href="http://www.chiro.org/LINKS/ABSTRACTS/Stress_Test.shtml">Stress test (dolphin and cow)</a></li>
<li><a href="http://blueballfixed.ytmnd.com/">Blue Ball Machine</a></li>
<li><a href="http://www.wakeskating.com/showarticle.php?id=18">Wakeboard rail sequence</a></li>
<li><a href="http://www.nick.com/legend-of-korra/">"Korra The Legend of Korra" by "The Legend of Korra" and all characters are owned by Nickelodeon Studios. - From Nickelodeon Studios.</a></li>
<li><a href="http://en.wikipedia.org/wiki/File:Korra_(Season_4)_appearance.jpg#/media/File:Korra_(Season_4)_appearance.jpg">"Korra (Season 4) appearance" by Source (WP:NFCC#4). Licensed under Fair use via Wikipedia</a></li>
<li><a href="http://en.wikipedia.org/wiki/File:Korra_The_Legend_of_Korra.jpg#/media/File:Korra_The_Legend_of_Korra.jpg">Korra shows her stuff; At Wikipedia via Nickelodeon</a></li>
</ul>
</section>
</div>
</div>
<script src="lib/js/head.min.js"></script>
<script src="js/reveal.js"></script>
<script>
// Full list of configuration options available here:
// https://github.com/hakimel/reveal.js#configuration
Reveal.initialize({
width: 1280,
height: 800,
controls: true,
progress: true,
history: true,
center: false,
theme: Reveal.getQueryHash().theme, // available themes are in /css/theme
transition: Reveal.getQueryHash().transition || 'default', // default/cube/page/concave/zoom/linear/fade/none
// Parallax scrolling
// parallaxBackgroundImage: 'https://s3.amazonaws.com/hakim-static/reveal-js/reveal-parallax-1.jpg',
// parallaxBackgroundSize: '2100px 900px',
// Optional libraries used to extend on reveal.js
dependencies: [
{ src: 'lib/js/classList.js', condition: function() { return !document.body.classList; } },
{ src: 'plugin/markdown/marked.js', condition: function() { return !!document.querySelector( '[data-markdown]' ); } },
{ src: 'plugin/markdown/markdown.js', condition: function() { return !!document.querySelector( '[data-markdown]' ); } },
{ src: 'plugin/highlight/highlight.js', async: true, callback: function() { hljs.initHighlightingOnLoad(); } },
{ src: 'plugin/zoom-js/zoom.js', async: true, condition: function() { return !!document.body.classList; } },
{ src: 'plugin/notes/notes.js', async: true, condition: function() { return !!document.body.classList; } }
]
});
</script>
</body>
</html>