Skip running tests when the only file changed is markdown

[ramya-rao-a]

Take this PR for example: Azure/azure-sdk-for-js#15523.

All it does is update the date in the changelog before a release. All tests associated with that folder is run in the PR CI which adds no value. Is there a way for the PR CI to skip the tests the only files changed are markdown files?

What I am looking for ways to reduce the wait time on PRs that are only trying to update markdown files. No issues in running the link verififcation and other checks.

(Not sure if other language repos have use for running the tests when markdown files are changed)

[ramya-rao-a]

Same for the yml files!! See Azure/azure-sdk-for-js#15146

cc @praveenkuttappan, @chidozieononiwu

[kurtzeborn]

In the teams discussion about this, you (@benbp) proposed that we could have detection in the matrix generation that would inspect the changed files and generate an empty test matrix for folders/libraries where only .md or .yml files changed.

[benbp]

I was thinking about adding some sort of filtering to the matrix config, e.g. GenerateOnFilesChanged or something like that:

  - name: MatrixConfigs
    type: object
    default:
      - Name: Js_live_test_base
        Path: eng/pipelines/templates/stages/platform-matrix.json
        Selection: sparse
        GenerateVMJobs: true
        GenerateOnFilesChanged:
          - *.js
          - *.ts
          - *.json
          # and so on...seems brittle

The more that I think about it, it may be better to have this kind of logic be in its own step, which could include scripts for various kinds of skip/run hook inputs and outputs, e.g.:

# step generates some variable that can be used later on
- template: set_test_job_condition.yml
- template: archetype-sdk-tests-generate.yml
  parameters:
    Condition: variables['GENERATE_TESTS']

# in the generate job:
job:
  condition: eq(${{ parameters.Condition }}, true)

[praveenkuttappan]

I like above approach. We can set a runtime pipeline variable based on type of files in PR Skip.Test and skip tests if no file requires testing. Individual language repo can control the types of files which doesn't need any testing. One more suggestion is to configure test exclusion types instead of test required types.

[benbp]

@praveenkuttappan by exclusion do you mean: if a file change is present, skip all tests even if other files match?

Also, if I've learned anything from adding sparse checkout to CI+live tests and java releases, there may be many unanticipated edge cases across languages, so I think my aim is to lay out the generic mechanism here and show the language teams how to use it. @ramya-rao-a would you be able to get someone from the js team to incorporate this config feature once I add it and update/test the necessary language-specific file filters?

[ramya-rao-a]

would you be able to get someone from the js team to incorporate this config feature once I add it and update/test the necessary language-specific file filters

What would this involve? Updating the yml files based on your recommendation? Or more?

[benbp]

would you be able to get someone from the js team to incorporate this config feature once I add it and update/test the necessary language-specific file filters

What would this involve? Updating the yml files based on your recommendation? Or more?

Yes, it should be pretty simple, I think: updating config targets and mainly validating that tests behave properly and we aren't skipping anything unexpectedly.

[ramya-rao-a]

Sure, we can do that

[weshaggard]

@benbp if we are going that fair we should have the job be the startup job for the entire pipeline and not just the tests. It can be a job that can setup variables based on various common inputs such as file globs but also potentially even custom tasks that can check for other conditions as needed. We can also use this logic for doing thing like setting the dev version or default branch or other common global properties we setup dynamically.

[benbp]

@weshaggard I agree, the way I would do this for live tests would be extensible to account for other job-type conditions as well. I think starting small and just hardcoding evaluations for build|analyze|test is a good first step to solve problems like this, and we can later expand on that common scripts to start to enable more complex conditions with dependencies.

[ramya-rao-a]

Any updates here?

It is code complete week again and we have PRs that only update the changelogs going out.
Each of these PRs are waiting on CI to complete running all the tests :(

[weshaggard]

No real updates here yet. We have some ideas but nothing concrete but even the simple ideas require a fair amount of design and implementation so I don't expect we will have a solution here for awhile.

[benbp]

Coming back to this @weshaggard. I think the live test handling is a common enough case that we should support setting global pipeline variables via the generator and/or generating an empty test set to solve this problem today.

The more I think about a common startup job, the more I worry about the added agent queue time that such a job will add to every pipeline run.

[weshaggard]

@benbp for the general case perhaps we can come up with a way to user a server job.

[benbp]

@weshaggard digging through the agentless job docs, I don't see any way to feed data back to the agent context (in this case, variables). Rather, the agentless jobs seem to be basically pass/fail gates for a pipeline as well as a way to trigger side effects external to the pipeline.

[kurtzeborn]

Copying this comment over from Azure/azure-sdk-for-js#18465 which was a duplicate:

This should be done in all of our language repos, not jus the JS repo.

The reason we haven't supported this is because DevOps doesn't really support it. @benbp has suggested that adding a feature to support this in the test matrix generation scripts might be a possible way to tackle this.

Related question is whether GitHub actions supports this (something we'll be investigating more over the next 6 months).