Fix flaky PitMultiNodeTests including testCreatePitWhileNodeDropWithAllowPartialCreationFalse

[dbwiddis]

Signed-off-by: Daniel Widdis widdis@gmail.com

Description

#4030 introduced a Point In Time Node Stats API and added stats validation to many PitMultiNodeTests.

Thorough debugging of flaky testCreatePitWhileNodeDropWithAllowPartialCreationFalse() revealed that the transLog consistently had a pending commit when the test failed (repro 1/500 runs)

Flushing the index prior to collecting/validating stats resolves this failure (tested 1000 runs) and probably prevents other flaky test results.

Flushing frequently should be considered a best practice for test hygiene.

Issues Resolved

Fixes #4259 (which was previously closed associated with an unrelated fix).
Fixes comments #4089 (comment) although the issue itself was fixed by a different PR.

Check List

• New functionality includes testing.
  • All tests pass
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.action.admin.cluster.node.tasks.ResourceAwareTasksTests.testTaskResourceTrackingDuringTaskCancellation

• URL: https://build.ci.opensearch.org/job/gradle-check/8583/
• CommitID: 646f423
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[codecov-commenter]

Codecov Report

Merging #5655 (646f423) into main (d388051) will increase coverage by 0.07%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##               main    #5655      +/-   ##
============================================
+ Coverage     71.01%   71.08%   +0.07%     
- Complexity    58562    58639      +77     
============================================
  Files          4760     4760              
  Lines        279514   279514              
  Branches      40348    40348              
============================================
+ Hits         198501   198701     +200     
+ Misses        64815    64629     -186     
+ Partials      16198    16184      -14     

Impacted Files	Coverage Δ
...main/java/org/opensearch/node/NodeMocksPlugin.java	0.00% <0.00%> (-100.00%)	⬇️
...port/ResponseHandlerFailureTransportException.java	0.00% <0.00%> (-60.00%)	⬇️
...cluster/coordination/PendingClusterStateStats.java	20.00% <0.00%> (-48.00%)	⬇️
...opensearch/persistent/PersistentTasksExecutor.java	22.22% <0.00%> (-44.45%)	⬇️
...pensearch/action/ingest/DeletePipelineRequest.java	31.25% <0.00%> (-37.50%)	⬇️
.../java/org/opensearch/index/reindex/RemoteInfo.java	50.45% <0.00%> (-29.36%)	⬇️
...g/opensearch/geometry/utils/StandardValidator.java	33.33% <0.00%> (-21.22%)	⬇️
...pensearch/index/shard/SearchOperationListener.java	72.72% <0.00%> (-20.00%)	⬇️
...search/search/suggest/phrase/PhraseSuggestion.java	70.00% <0.00%> (-20.00%)	⬇️
...org/opensearch/test/engine/MockInternalEngine.java	65.21% <0.00%> (-17.40%)	⬇️
... and 494 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.