rest-api-specs are out of sync with the code base

[olcbean]

Currently it is very easy for the rest-api-spec to go out of sync with the code base. For example: the code base is changed, the docs are updated but the rest-api-spec still refers to no-longer-existing parameter names (#27124, d6af6697766 and 51d53791fe7). Or a required path part is still wrongly marked as not required (#25988)

Writing explicit tests is error prone - forgetting to add a new parameter or change an existing one is way too common.

We should have a more robust approach to verify that the rest-api-specs remain valid. It is not possible to gurantee that all parameters are exposed to the rest-api-apec but at least we should attempt to check that every parameter in the rest-api-spec can actually be used.

Has this been discussed before / is there any prior art related to this style of tests?

[jasontedor]

At some point we intend to generate the specification from the server code. This issue would then be addressed by that effort, and I don't think we are going to make a concerted effort now otherwise.