The current shard follow mechanism is complex and does not give us easy ways the have visibility into the system (e.g. why we are falling behind).
The main reason why it is complex is because the current design is highly asynchronous. Also in the current model it is hard to apply backpressure
other than reducing the concurrent reads from the leader shard.

This PR has the following changes:
* Rewrote the shard follow task to coordinate the shard follow mechanism between a leader and follow shard in a single threaded manner.
  This allows for better unit testing and makes it easier to add stats.
* All write operations read from the shard changes api should be added to a buffer instead of directly sending it to the bulk shard operations api.
  This allows to apply backpressure. In this PR there is a limit that controls how many write ops are allowed in the buffer after which no new reads
  will be performed until the number of ops is below that limit.
* The shard changes api includes the current global checkpoint on the leader shard copy. This allows reading to be a more self sufficient process;
  instead of relying on a background thread to fetch the leader shard's global checkpoint.
* Reading write operations from the leader shard (via shard changes api) is a seperate step then writing the write operations (via bulk shards operations api).
  Whereas before a read would immediately result into a write.
* The bulk shard operations api returns the local checkpoint on the follow primary shard, to keep the shard follow task up to date with what has been written.
* Moved the shard follow logic that was previously in ShardFollowTasksExecutor to ShardFollowNodeTask.
* Moved over the changes from elastic#31242 to make shard follow mechanism resilient from node and shard failures.

Relates to elastic#30086

…anges api and

LuceneChangesSnapshot to allow a response to not return up to `toSeqNo`.

… do a single read.

if the shard changes response returns no hits and had no target seqno then
it should schedule coordinate reads phase.

…ollowNodeTask

…ite_2

* elastic/ccr: (30 commits)
  Enable setting client path prefix to / (elastic#30119)
  [DOCS] Secure settings specified per node (elastic#31621)
  has_parent builder: exception message/param fix (elastic#31182)
  TEST: Randomize soft-deletes settings (elastic#31585)
  Mute 'Test typed keys parameter for suggesters' as we await a fix.
  Build test: Thread linger
  Fix gradle4.8 deprecation warnings (elastic#31654)
  Mute FileRealmTests#testAuthenticateCaching with an @AwaitsFix.
  Mute TransportChangePasswordActionTests#testIncorrectPasswordHashingAlgorithm with an @AwaitsFix.
  Build: Fix naming conventions task   (elastic#31681)
  Introduce a Hashing Processor (elastic#31087)
  Do not check for object existence when deleting repository index files (elastic#31680)
  Remove extra check for object existence in repository-gcs read object (elastic#31661)
  Support multiple system store types (elastic#31650)
  [Test] Clean up some repository-s3 tests (elastic#31601)
  [Docs] Use capital letters in section headings (elastic#31678)
  muted tests that will be replaced by the shard follow task refactoring: elastic#31581
  [DOCS] Add PQL language Plugin (elastic#31237)
  Merge AzureStorageService and AzureStorageServiceImpl and clean up tests (elastic#31607)
  TEST: Fix test task invocation (elastic#31657)
  ...

…ite_2

… response after

it replication has been completed

the maxOperationSizeInBytes limit has been met

…ite_2

…eSize into maxBatchOperationCount

…ite_2

…rites() is always safe