chore(build): run QE tests during PR workflow

[mirekdlugosz]

The main goal is to run QE tests on ephemeral environment as part of PR workflow.

Right now there's one such tests that goes to subscription table and verifies it is not displayed. Later we will figure out how to create data inside ephemeral env, and we'll work on extending suite of tests.

I've got this running when issuing commands manually from my machine. Let's see if I can get it running from PR check as well.

[codecov-commenter]

Codecov Report

Merging #1008 (ff7466a) into dev (4e97ff0) will not change coverage.
The diff coverage is n/a.

Additional details and impacted files

@@           Coverage Diff           @@
##              dev    #1008   +/-   ##
=======================================
  Coverage   93.58%   93.58%           
=======================================
  Files         122      122           
  Lines        4081     4081           
  Branches     1646     1646           
=======================================
  Hits         3819     3819           
  Misses        245      245           
  Partials       17       17           

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4e97ff0...ff7466a. Read the comment docs.

[mirekdlugosz]

OK, so this passed. If you go to "details" under ci.int.devshift.net PR build and log in, you can click "Test Results" and see breakdown of tests that were run. Which is nice.

I am pretty sure accessing ci.int.devshift.net requires VPN.

Some time breakout of this specific run:

• running unit tests, building container, pushing it to quay: 02:49
• deploying new ephemeral environment with subscriptions, including backend and dependencies: 10:31
• deploying pod with QE tests and running them: 02:06
• the whole thing: 16:08

GitHub tells me GitHub action took 8 minutes, so App SRE workflow doubles that.

For the reference, backend team needs around 20 minutes to complete their PR job on App SRE infra. We'll probably need a little more, because UI tests are always going to be slower than API. Obviously I don't want to make that too slow, I think 25 minutes is reasonable, 30 minutes is absolute maximum. We will have to cut the scope of UI tests to meet this time requirement - we will run fewer tests than backend team does.

[cdcabrera]

@mirekdlugosz would removing the redundancy of running the GUI tests make much of a difference (removing the $ yarn test:ephemeral and related node scripts in package.json)?

If we haven't already removed the redundant unit test checks, It would be preferable if we could and focus on just QE's facet.

Most of it is we still use GitHub actions for

• not having to login to review unit test results on a public project
• codecov aspect of actions, unclear if this support works or is even provided with the behind-the-scenes checks. we generate the coverage, just unclear where it goes?
• a more in-depth aspect of dev integration checks. we had to disable parts of those checks to get them to pass for the behind-the-scenes checks
• commit linting
• and generally just easier to find documentation =)

[cdcabrera]

Posted a non-blocking question/comment. Just need a rebase and we should be good

[cdcabrera]

As of 20221130 holding PR. It's recommend we hold until there are more tests associated with this check

[san7ket]

/retest

[san7ket]

/retest

[san7ket]

/retest

[san7ket]

/retest

[san7ket]

/retest

[mirekdlugosz]

/retest

[san7ket]

/retest

[ntkathole]

/retest

[san7ket]

Closing this in favor of #1072