Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Adding client-side request cancellation support and testing (#6383)
* Add L0_request_cancellation (#6252) * Add L0_request_cancellation * Remove unittest test * Add cancellation to gRPC server error handling * Fix up * Use identity model * Add tests for gRPC client-side cancellation (#6278) * Add tests for gRPC client-side cancellation * Fix CodeQL issues * Formatting * Update qa/L0_client_cancellation/client_cancellation_test.py Co-authored-by: Ryan McCormick <rmccormick@nvidia.com> * Move to L0_request_cancellation * Address review comments * Removing request cancellation support from asyncio version * Format * Update copyright * Remove tests * Handle cancellation notification in gRPC server (#6298) * Handle cancellation notification in gRPC server * Fix the request ptr initialization * Update src/grpc/infer_handler.h Co-authored-by: Ryan McCormick <rmccormick@nvidia.com> * Address review comment * Fix logs * Fix request complete callback by removing reference to state * Improve documentation --------- Co-authored-by: Ryan McCormick <rmccormick@nvidia.com> --------- Co-authored-by: Ryan McCormick <rmccormick@nvidia.com> * Fixes on the gRPC frontend to handle AsyncNotifyWhenDone() API (#6345) * Fix segmentation fault in gRPC frontend * Finalize all states upon completion * Fixes all state cleanups * Handle completed states when cancellation notification is received * Add more documentation steps * Retrieve dormant states to minimize the memory footprint for long streams * Update src/grpc/grpc_utils.h Co-authored-by: Ryan McCormick <rmccormick@nvidia.com> * Use a boolean state instead of raw pointer --------- Co-authored-by: Ryan McCormick <rmccormick@nvidia.com> * Add L0_grpc_state_cleanup test (#6353) * Add L0_grpc_state_cleanup test * Add model file in QA container * Fix spelling * Add remaining subtests * Add failing subtests * Format fixes * Fix model repo * Fix QA docker file * Remove checks for the error message when shutting down server * Fix spelling * Address review comments * Add schedulers request cancellation tests (#6309) * Add schedulers request cancellation tests * Merge gRPC client test * Reduce testing time and covers cancelling other requests as a consequence of request cancellation * Add streaming request cancellation test --------- Co-authored-by: Iman Tabrizian <iman.tabrizian@gmail.com> Co-authored-by: Ryan McCormick <rmccormick@nvidia.com> Co-authored-by: Jacky <18255193+kthui@users.noreply.github.com>
- Loading branch information