Robustness tests should validate if etcd binary supports gofail

[serathius]

What would you like to be added?

Etcd robustness tests scenarios that require gofail should validate that etcd binary was complied with gofail enabled and if it's not it should provide a verbose error with instructions how to compile etcd.

Why is this needed?

It's not obvious for new users that linearizability tests require etcd build with gofail enabled.
Error is not very verbose: "connection refused" error from a goPanicFailpoint.

#15237 (comment)

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.

[preetham]

@serathius I would like to pick this up, Can you please help me with any information required?

[serathius]

Sure, feel free to reach me on Slack.

[preetham]

@serathius Can you please check the PR: #15963? Does this line up with the expectation?

[jmhbnz]

Discussed during sig etcd triage meeting. It's possible this is already resolved via #16698.

Assigned @moficodes to check and close if so.

[moficodes]

@jmhbnz will take a look Thanks.

[moficodes]

Tried with gofail-disable and build.

Was hoping test-robustness would fail. It did not. Might be doing something wrong.

user@vs-code:~/documents/github.com/moficodes/etcd$ make gofail-disable && make build

cd tools/mod; go install go.etcd.io/gofail@v0.1.0
gofail disable server/etcdserver/ server/storage/backend/ server/storage/mvcc/ server/storage/wal/
cd ./server && go mod tidy
cd ./etcdutl && go mod tidy
cd ./etcdctl && go mod tidy
cd ./tests && go mod tidy
GO_BUILD_FLAGS=" -v -mod=readonly" ./scripts/build.sh
Running etcd_build
% 'rm' '-f' 'bin/etcd'
% (cd server && 'env' 'CGO_ENABLED=0' 'GO_BUILD_FLAGS= -v -mod=readonly' 'GOOS=linux' 'GOARCH=amd64' 'go' 'build' '-v' '-mod=readonly' '-trimpath' '-installsuffix=cgo' '-ldflags=-X=go.etcd.io/etcd/api/v3/version.GitSHA=904c0769e' '-o=../bin/etcd' '.')
% 'rm' '-f' 'bin/etcdutl'
% (cd etcdutl && 'env' 'GO_BUILD_FLAGS= -v -mod=readonly' 'CGO_ENABLED=0' 'GO_BUILD_FLAGS= -v -mod=readonly' 'GOOS=linux' 'GOARCH=amd64' 'go' 'build' '-v' '-mod=readonly' '-trimpath' '-installsuffix=cgo' '-ldflags=-X=go.etcd.io/etcd/api/v3/version.GitSHA=904c0769e' '-o=../bin/etcdutl' '.')
% 'rm' '-f' 'bin/etcdctl'
% (cd etcdctl && 'env' 'GO_BUILD_FLAGS= -v -mod=readonly' 'CGO_ENABLED=0' 'GO_BUILD_FLAGS= -v -mod=readonly' 'GOOS=linux' 'GOARCH=amd64' 'go' 'build' '-v' '-mod=readonly' '-trimpath' '-installsuffix=cgo' '-ldflags=-X=go.etcd.io/etcd/api/v3/version.GitSHA=904c0769e' '-o=../bin/etcdctl' '.')
SUCCESS: etcd_build (GOARCH=amd64)

user@vs-code:~/documents/github.com/moficodes/etcd$ make test-robustness

PASSES="robustness" ./scripts/test.sh 
Running with --race
Starting at: Fri 24 Nov 2023 06:13:55 AM UTC

'robustness' started at Fri 24 Nov 2023 06:13:55 AM UTC
% (cd tests && 'env' 'ETCD_VERIFY=all' 'go' 'test' 'go.etcd.io/etcd/tests/v3/robustness' '-timeout=30m')
ok      go.etcd.io/etcd/tests/v3/robustness     (cached)
'robustness' PASSED and completed at Fri 24 Nov 2023 06:13:59 AM UTC
SUCCESS

[jmhbnz]

Hey @moficodes - To get robustness tests running you will need to construct a full testing command, something like below for example:

 james  ~  Documents  etcd   main                                                                                                                                              21:13:30 
 ➜ GO_TEST_FLAGS="-v --count 12 --timeout 30m --failfast --run TestRobustness" 
 ➜ ETCD_BRANCH=main EXPECT_DEBUG=true GO_TEST_FLAGS=${GO_TEST_FLAGS} RESULTS_DIR=/tmp/results make test-robustness       

You can see how these test commands are put together looking at the github actions workflow: https://github.com/etcd-io/etcd/blob/main/.github/workflows/robustness-template.yaml

Perhaps in future we could add some sane defaults when running make test-robustness if nothing is provided in advance.

[moficodes]

@jmhbnz
With gofail disabled test still passed.

ok      go.etcd.io/etcd/tests/v3/robustness     456.528s
'robustness' PASSED and completed at Fri 24 Nov 2023 02:07:55 PM UTC
SUCCESS

[serathius]

Oops, if you run GO_TEST_FLAGS='-v' make test-robustness to get verbose logs:

cluster.go:432: please run 'make gofail-enable && make build' before running the test

[serathius]

#16698 allowed to skip e2e tests when running binary without failpoints.

This is nice for local development, however for CI we should have a way to ensure we run all tests. For example a flag --skip-gofail.

[moficodes]

@serathius yep. I see the log. I thought it was going to fail the test.

[moficodes]

@jmhbnz If just the log is intended behavior I can confirm this log print happens.

[jmhbnz]

Discussed during sig-etcd triage meeting. @serathius can you add some more context here on what the desired next step is?

[serathius]

#16698 Has implemented this issue as a side effect. So I would consider this issue closed. There is one possible followup, with automated test skipping it will be easy to introduce a bug that would unintentionally skip test. We should have some validation in CI or improve visibility into skipped tests.