Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Gracefully drain connections when stopping the gateway #1203

Merged
Changes from 6 commits
Commits
Show all changes
44 commits
Select commit Hold shift + click to select a range
6f5e4da
Fix gateway's preStop hook: curl does not exist (anymore?) in envoy i…
norbjd Feb 1, 2024
3e3e933
Before stopping the gateway, wait until requests are finished on all …
norbjd Feb 1, 2024
659fc4e
Drain listeners with appropriate endpoint
norbjd Feb 2, 2024
d10d790
Simpler drain + sleep
norbjd Feb 2, 2024
5be667d
Remove PARENT_SHUTDOWN_TIME_SECONDS and terminationGracePeriodSeconds
norbjd Feb 2, 2024
ff62994
Use a perl script (no need to open the admin HTTP interface!)
norbjd Feb 2, 2024
423a6da
Use bash instead of perl in preStop hook
norbjd Mar 10, 2024
2b19b5d
Merge branch 'knative-extensions:main' into gateway-prestop-hook-wait…
norbjd Apr 7, 2024
223b4ac
Review @skonto comments: use socket address for admin cluster
norbjd Apr 7, 2024
c5a0b98
[WIP] add graceful shutdown test and tweak CI to just run that test
norbjd Apr 7, 2024
140d05f
[WIP] Fix gracefulshutdown_test.go
norbjd Apr 7, 2024
00b30cd
[WIP] try to fix race condition and lint
norbjd Apr 7, 2024
8099a30
[WIP] use initialTimeout + debug
norbjd Apr 7, 2024
77b783e
[WIP] fix gracefulshutdown_test.go logic
norbjd Apr 7, 2024
f319b6c
[WIP] refacto and add some comments to clarify
norbjd Apr 7, 2024
6ce6d58
[WIP] fix lint
norbjd Apr 7, 2024
c8523a9
[WIP] reintroduce kind-e2e-upgrade.yaml
norbjd Apr 7, 2024
95d7576
[WIP] add test case when request takes a little longer than the drain…
norbjd Apr 7, 2024
fd3a88d
[WIP] fix compilation issue
norbjd Apr 7, 2024
efd64ba
[WIP] FIx compilation issue (again)
norbjd Apr 7, 2024
1f421a6
[WIP] hopefully fix data race
norbjd Apr 7, 2024
3234543
[WIP] refacto and hopefully fix race condition (use sync.Map)
norbjd Apr 7, 2024
ca85970
[WIP] fix compilation issue
norbjd Apr 7, 2024
474cc29
[WIP] Handle EOF
norbjd Apr 7, 2024
c2d1697
[WIP] check gateway pod has been removed + manual debugging
norbjd Apr 7, 2024
471c694
[WIP] debugging
norbjd Apr 7, 2024
fd1d993
[WIP] more debugging
norbjd Apr 7, 2024
acb1cba
[WIP] more debugging
norbjd Apr 7, 2024
cfa1b7a
[WIP] increase livenessProbe failure threshold as I'm not sure it sho…
norbjd Apr 7, 2024
0aaa024
[WIP] remove debugging related stuff
norbjd Apr 7, 2024
631f8ce
Revert all unnecessary changes made for testing
norbjd Apr 7, 2024
0e62771
Revert unnecessary change (livenessProbe)
norbjd Apr 7, 2024
4cc6b26
Scale to 1 replica
norbjd Apr 7, 2024
ed9ed64
Typo
norbjd Apr 7, 2024
eb1dd7a
Run gracefulshutdown test first (speed up feedback loop)
norbjd Apr 7, 2024
b7ca108
Merge branch 'main' into gateway-prestop-hook-wait-until-all-incoming…
norbjd Apr 14, 2024
184c0e9
Merge branch 'main' into gateway-prestop-hook-wait-until-all-incoming…
norbjd Apr 27, 2024
7170cb6
Add a comment for terminationGracePeriodSeconds
norbjd Apr 27, 2024
63e1131
Don't update deployment twice
norbjd Apr 27, 2024
44ff38c
Fix bad patch
norbjd Apr 27, 2024
ba83fe9
Run gracefulshutdown test at the end
norbjd Apr 27, 2024
b908c32
Fix gracefulshutdown test
norbjd Apr 27, 2024
4ca0a27
Fix gracefulshutdown test
norbjd Apr 27, 2024
86fcaeb
Lint
norbjd Apr 27, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 26 additions & 1 deletion config/300-gateway.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -48,8 +48,13 @@ spec:
- --base-id 1
- -c /tmp/config/envoy-bootstrap.yaml
- --log-level info
- --drain-time-s $(DRAIN_TIME_SECONDS)
norbjd marked this conversation as resolved.
Show resolved Hide resolved
- --drain-strategy immediate
command:
- /usr/local/bin/envoy
env:
- name: DRAIN_TIME_SECONDS
value: "15"
image: docker.io/envoyproxy/envoy:v1.26-latest
name: kourier-gateway
ports:
Expand Down Expand Up @@ -85,7 +90,27 @@ spec:
lifecycle:
preStop:
exec:
command: ["/bin/sh","-c","curl -X POST --unix /tmp/envoy.admin http://localhost/healthcheck/fail; sleep 15"]
command:
- perl
norbjd marked this conversation as resolved.
Show resolved Hide resolved
- -e
- |-
use strict;
use warnings;
use IO::Socket qw(AF_UNIX SOCK_STREAM SHUT_WR);

my $client = IO::Socket->new(
Domain => IO::Socket::AF_UNIX,
Type => IO::Socket::SOCK_STREAM,
Peer => "/tmp/envoy.admin"
) || die "Can't open socket: $IO::Socket::errstr";
$client->autoflush(1);

$client->send("POST /drain_listeners?graceful HTTP/1.1\nHost: localhost\nConnection: close\n\n");
$client->shutdown(IO::Socket::SHUT_WR);

$client->close();

sleep $ENV{DRAIN_TIME_SECONDS};
readinessProbe:
httpGet:
httpHeaders:
Expand Down
Loading