Skip to content

Commit

Permalink
Added traffic validation document (#20271)
Browse files Browse the repository at this point in the history
* Added traffic validation document

* Update per the feedback
  • Loading branch information
raych1 authored Aug 18, 2022
1 parent 7a2aa12 commit 08560e0
Show file tree
Hide file tree
Showing 2 changed files with 28 additions and 0 deletions.
7 changes: 7 additions & 0 deletions documentation/ci-fix.md
Original file line number Diff line number Diff line change
Expand Up @@ -165,6 +165,13 @@ This validator is to ensure the cadl & swagger files in PR are consistent and th
|InConsistentSwagger| the generated swagger is inconsistent with the swagger in PR, so you need to re-generate swagger from cadl, and check in it |
|SwaggerNotExistInPR| the occurs when there is cadl file in the PR but the swagger is not present in the PR, so you need to add the swagger to the PR |

## Traffic Validation

It generates traffics for all operations defined in swaggers under default tag of readme.md by using [RESTLer](https://github.com/microsoft/restler-fuzzer). Then it validates the request and response pairs from these traffics against correspondent swaggers. Finally, it provides a html report that reflects swagger accuracy.

### How to understand and improve the report
Please refer to [swagger-accuracy-report](./swagger-accuracy-report.md).

## Suppression Process

In case there are validation errors reported against your service that you believe do not apply, we have a suppression process you can follow to permanently remove these reported errors for your specs. Refer to [Swagger Suppression Process](https://dev.azure.com/azure-sdk/internal/_wiki/wikis/internal.wiki/85/Swagger-Suppression-Process) for detailed guidance.
21 changes: 21 additions & 0 deletions documentation/swagger-accuracy-report.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
# Swagger Accuracy Report
## What is this report about
[@Jeffrey to add details]

## How to improve report
There're two parts of the report need to be improved if it has content.
* First part is `Failed Operations` which includes the schema validation errors for listed operations, all the errors are supposed to be fixed. From the error code you could browse to the linked document about the error defintion and fix tips.
* Second part is `Untested Operations` which includes the operations which don't have the traffic generated by RESTler for validation. It could be because the dependent operations fails in validation in which case please fix failed operation firstly so that this untested operation will be unlocked automatically.

## How to improve Untested Operations
All the traffics for each operation are generated by RESTler automatically, if the result says the operation is untested then you could try to modify RESTler config files to improve RESTler generation coverage.
[@Marina to add details on how to modify configs]

* Download and modify each RESTler config
[to add details]

* Debug locally with updated RESTler configs
[to add details]

* Check in updated RESTler configs to be used in swagger PR pipeline and re-run
[to add details]

0 comments on commit 08560e0

Please sign in to comment.