tests: Add support for lazyfs

[serathius]

Signed-off-by: Marek Siarkowicz siarkowicz@google.com

Please read https://github.com/etcd-io/etcd/blob/main/CONTRIBUTING.md#contribution-flow.

[ptabor]

@serathius: Is it still intentionally draft ?

[serathius]

Yes, I only implemented building lazyfs. I still need to configure it and implement cache clear on crash.

[codecov-commenter]

Codecov Report

Merging #14691 (08cb83a) into main (08cb83a) will not change coverage.
The diff coverage is n/a.

❗ Current head 08cb83a differs from pull request most recent head 3ecf291. Consider uploading reports for the commit 3ecf291 to get more accurate results

@@           Coverage Diff           @@
##             main   #14691   +/-   ##
=======================================
  Coverage   75.50%   75.50%           
=======================================
  Files         457      457           
  Lines       37423    37423           
=======================================
  Hits        28257    28257           
  Misses       7387     7387           
  Partials     1779     1779           

Flag	Coverage Δ
all	75.50% <0.00%> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.

[serathius]

Still want to revisit this. Problem is integrating lazyfs in a user approachable way. We need to run separate lazyfs process for each etcd instance. It might require non-trivial rewrite.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.

[serathius]

PR is ready to review. There is one concern on my side. LazyFS uses a lot of CPU, in environment with limited resources like CI it might be too much. Will need to validates if it doesn't cause flakes.

cc @ahrtr @ptabor @jmhbnz @fuweid @chaochn47

[fuweid]

LazyFS uses a lot of CPU, in environment with limited resources like CI it might be too much.

I run this branch in my local with 3 cores. The failure is about traffic.go:91: Requiring minimal 200.000000 qps for test results to be reliable, got 147.115802 qps. I did see the lazyfs uses more CPU, which might cause the flake

[serathius]

Need to think how to integrate the lazyfs into the test scenarios. Should we lower the qps requirement, run separate scenarios with lower qps, or only enable on LowTraffic. Let me know if you have any suggestions.

[fuweid]

run separate scenarios with lower qps, or only enable on LowTraffic

I am trying to enable grpc-only endpoint for the test. update it later.

[fuweid]

@serathius My devbox is kind of similar to github action vm. I run the branch with 3 CPU cores. I think the HighTraffic.minQPS should be changed to 120 if enableLazyfs is true. I was trying to use different block size but it doesn't help. The fsync operation might require more CPU based on my testing.

BTW, compactAfterCommitBatch becomes flaky after enable lazyfs. After trigger compact, the total number of keys isn't more enough to trigger batch commit. Hope that information is help.

[fuweid]

I didn't see any qps issue in LowTraffic or Kubernetes profiles. For the change, it looks good to me.

[fuweid]

2023-07-25T09:21:06.6384677Z     logger.go:130: 2023-07-25T09:21:04.266Z	INFO	Triggering failpoint	{"failpoint": "compactBeforeCommitBatch"}
2023-07-25T09:21:06.6385599Z     logger.go:130: 2023-07-25T09:21:04.266Z	INFO	Setting up gofailpoint	{"failpoint": "compactBeforeCommitBatch"}
2023-07-25T09:21:06.6386514Z     logger.go:130: 2023-07-25T09:21:04.272Z	INFO	Triggering gofailpoint	{"failpoint": "compactBeforeCommitBatch"}
2023-07-25T09:21:06.6387728Z /home/runner/work/etcd/etcd/bin/etcd (TestRobustnessKubernetesLowTrafficClusterOfSize1LazyFS-test-0) (11989): {"level":"info","ts":"2023-07-25T09:21:04.390478Z","caller":"mvcc/index.go:194","msg":"compact tree index","revision":73}
2023-07-25T09:21:06.6389417Z /home/runner/work/etcd/etcd/bin/etcd (TestRobustnessKubernetesLowTrafficClusterOfSize1LazyFS-test-0) (11989): {"level":"info","ts":"2023-07-25T09:21:04.390729Z","caller":"mvcc/kvstore_compaction.go:69","msg":"finished scheduled compaction","compact-revision":73,"took":"177.702µs","hash":2606726685}
2023-07-25T09:21:06.6391083Z /home/runner/work/etcd/etcd/bin/etcd (TestRobustnessKubernetesLowTrafficClusterOfSize1LazyFS-test-0) (11989): {"level":"info","ts":"2023-07-25T09:21:04.390754Z","caller":"mvcc/hash.go:143","msg":"storing new hash","hash":2606726685,"revision":73,"compact-revision":-1}
2023-07-25T09:21:06.6392842Z     logger.go:130: 2023-07-25T09:21:04.391Z	INFO	Waiting for member to exit	{"member": "TestRobustnessKubernetesLowTrafficClusterOfSize1LazyFS-test-0"}

That is flaky case I mentioned. The compact doesn't commit after enable failpoint.

REF: https://github.com/etcd-io/etcd/actions/runs/5654876623/job/15318829070?pr=14691

[fuweid]

stderr: # go.etcd.io/etcd/tests/v3/framework/e2e
Error: stderr: framework/e2e/cluster.go:415:34: too many arguments in call to NewEtcdProcess
stderr: have (testing.TB, *EtcdServerProcessConfig)
stderr: want (*EtcdServerProcessConfig)
Error: stderr: framework/e2e/cluster.go:835:34: too many arguments in call to NewEtcdProcess
stderr: have (testing.TB, *EtcdServerProcessConfig)
stderr: want (*EtcdServerProcessConfig)
Error: stderr: framework/e2e/cluster.go:864:34: too many arguments in call to NewEtcdProcess
stderr: have (testing.TB, *EtcdServerProcessConfig)
stderr: want (*EtcdServerProcessConfig)
Error: stderr: framework/e2e/cluster_proxy.go:43:9: cannot use NewProxyEtcdProcess(cfg) (value of type *proxyEtcdProcess) as type EtcdProcess in return statement:
stderr: *proxyEtcdProcess does not implement EtcdProcess (missing LazyFS method)
Error: stderr: framework/e2e/cluster_proxy.go:47:34: not enough arguments in call to NewEtcdServerProcess
stderr: have (*EtcdServerProcessConfig)
stderr: want (testing.TB, *EtcdServerProcessConfig)

need to support proxy part.

[serathius]

need to support proxy part.

I know :(. I always neglect cluster_proxy as it's inconvenient to run locally.

[serathius]

PR passed tests. We have a working automatic disk failure injection testing! Same as Jepsen

Please take a look @fuweid @chaochn47 @jmhbnz @ahrtr

[jmhbnz]

Nice work getting this going team!

Some feedback below, nothing too major.

[fuweid]

LGTM!

[jmhbnz]

LGTM