Plan for Improved Testability of Helix Python Scripts

[missymessa]

Ensuring all our code is testable is a team principle. Therefore, we need to come up with a plan to improve our testing of our Helix Python scripts as we make changes to them to support test retries.

At a minimum, let's come up with plans for the following:

• Unit testing: Preferably local unit testing, instructions for devs to be able to do it locally, be able to run these unit tests in PRs.
• Scenario testing: Results should be discoverable, consolidated, meaningful.

[missymessa]

Notes from our meeting regarding a better testing strategy:

• Scenario testing

  • Stand up endpoints: Service Bus, Event Hub, Storage
  • Use Docker for running the tests so that the scripts don't mess up the state of your local computer
    • Be sure to use the "lowest" common version of Python
  • Mount your local scripts into the Docker container
  • May not need to build the WHL in order to locally test
  • Should be able to use this process for local tests, PR/CI tests
  • Use Server Core for testing Windows with Docker; should be able to use any flavor of Linux
  • (Can't do nested Docker containers)
  • Create some kind of thing like "Arcade Validation" to run bigger scenario tests (for the retry tests feature)

• Unit testing

  • Too much module-level code that would require a greater re-write of the scripts in order to make unit testing feasible

• Existing testability

  • If the scripts change, the tests break
  • Locally, leaky python executables
  • Validate stage in dotnet-helix-machines pipeline

[missymessa]

Add this to a new epic for Helix scripts tech debt rewrite.