[rush] Refactor build cache to reduce critical path length #4275
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Refactors the legacy skip logic and the build cache to each be their own
PhasedOperationPlugin
.Schedules reading from and writing to the build cache as separate operations in the queue.
This allows for cache reads to happen earlier in the pipeline, and allows for cache writes to occur in parallel with operations that consume the output.
The skip detection logic and build cache will now never coexist in the same execution.
Details
Added new hooks to
PhasedOperationHooks
:beforeExecuteOperation
- Executed immediately prior to the operation executing. If this returns anOperationStatus
, the operation will be skipped, and the status will be the result of the operation.afterExecuteOperation
- Executed immediately following the operation's execution. If this returns anOperationStatus
, it will override the existing status from execution. This hook does not get called ifbeforeExecuteOperation
returned a value.Skip detection is implemented by tapping
beforeExecuteOperation
and injecting a check if the operation can be skipped. It also tapsafterExecuteOperation
to propagate whether or not skipping is allowed.The build cache taps
createOperations
and injects additional synthetic operations before (if incremental builds are enabled) and after (if cache writes are enabled) each normal operation in the graph. It uses custom runners for cache read and write that communicate with taps on thebeforeExecuteOperation
andafterExecuteOperation
hooks.With this change, if a cache read fails, the build will log a warning and proceed to try executing the operation normally. If a cache write fails, consumers of the operation being written to the cache will not be blocked, though the build will still log a failure at the end.
Adds a new parameter to the callback in
Async.forEachAsync
to allow deferral of a currently executing task. The expected use case for this feature is to support #3949, wherein Rush needs to be able to defer execution while waiting for the results of a remotely executing operation.How it was tested
Local invocation in both incremental and non-incremental mode.
Unit tests for the
Async.forEachAsync
code.Impacted documentation
Any documentation about how cache reads/writes are scheduled.
The hooks for the build cache.