contrib/grpc: attempt to fix flaky tests

[RomainMuller]

What does this PR do?

• AppSec tests are occasionally receiving an unexpected io.EOF error from a gRPC stream send, which according to the documentation happens if the error does not originate from the client (and in those cases, blocking causes the error server-side)
• Some tests were locally flaking due to expecting a certain count of security events being discovered, but this sometimes was not met due to the WAF timeout (this likely would not happen in CI, where we seront a higher-than-default timeout via environment)
• Some other (non-AppSec) tests fail (possibly due to AppSec being there), but would not reproduce locally... so this PR replaces some calls to require with calls to assert to allow collection of more information about what's happened; and adds some new assert checks at key locations where errors could have been silently ignored previously.

Reviewer's Checklist

• Changed code has unit tests for its functionality at or near 100% coverage.
• System-Tests covering this feature have been added and enabled with the va.b.c-dev version tag.
• There is a benchmark for any new code, or changes to existing code.
• If this interacts with the agent in a new way, a system test has been added.
• Add an appropriate team label so this PR gets put in the right place for the release notes.
• Non-trivial go.mod changes, e.g. adding new modules, are reviewed by @DataDog/dd-trace-go-guild.

Unsure? Have a question? Request a review!

[pr-commenter]

Benchmarks

Benchmark execution time: 2024-09-17 13:58:07

Comparing candidate commit 0ca4e94 in PR branch romain.marcadier/fix-flakes with baseline commit 1f0966d in branch main.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 59 metrics, 0 unstable metrics.

[RomainMuller]

/merge

[dd-devflow]

🚂 MergeQueue: waiting for PR to be ready

This merge request is not mergeable yet, because of pending checks/missing approvals. It will be added to the queue as soon as checks pass and/or get approvals.
Note: if you pushed new commits since the last approval, you may need additional approval.
You can remove it from the waiting list with /remove command.

Use /merge -c to cancel this operation!

[dd-devflow]

⚠️ MergeQueue: This merge request was unqueued

This merge request was unqueued

If you need support, contact us on Slack #devflow!

[RomainMuller]

/merge

[dd-devflow]

🚂 MergeQueue: pull request added to the queue

The median merge time in main is 0s.

Use /merge -c to cancel this operation!

[dd-devflow]

🚨 MergeQueue: This merge request is in error

mergequeue build completed successfully, but the github api returned an error while merging the pr.
It's probably because:

• repository configuration requires approval by code owners, but no code owner approved this PR
• target branch of PR is restricted to only allow up-to-date branches, but the pr is now outdated

Details

Error: PUT https://api.github.com/repos/DataDog/dd-trace-go/pulls/2872/merge: 405 Waiting on code owner review from DataDog/apm-go and/or DataDog/apm-idm-go. 5 of 5 required status checks are expected. []

FullStacktrace:
activity error (type: github.GithubService_MergePullRequest, scheduledEventID: 41, startedEventID: 42, identity: 1@github-worker-7b8757757d-bs564@): PUT https://api.github.com/repos/DataDog/dd-trace-go/pulls/2872/merge: 405 Waiting on code owner review from DataDog/apm-go and/or DataDog/apm-idm-go. 5 of 5 required status checks are expected. [] (type: ErrorResponse, retryable: true)

If you need support, contact us on Slack #devflow with those details!