stress test staging #731
stress test staging #731
Conversation
|
This looks like it had fewer concurrent cluster than I often see in prod:
e.g., here's the last 2 days in prod:
Does CI still start a new run on each push of new commits to a branch? |
It should, yes. On prod it is still possible to go higher than the per-job limits since GH allows 20 concurrent jobs in total. Every ordinary CI job is using 4 processes / concurrent Coiled clusters. With multiple branches we can easily go higher than this. A/B tests are what's most important since every A/B tests is running 5 concurrent CI jobs with 4 processes per job |
AB_environments/config.yaml
Outdated
| ci_jobs: 5 | ||
| ci_jobs: 20 | ||
| # Number of parallel test_*.py modules per A/B test job. | ||
| # Each module typically spawns one Coiled cluster at a time. | ||
| # Set to 1 to disable pytest-xdist. | ||
| pytest_workers_per_job: 4 |
There was a problem hiding this comment.
I'm not sure if the gh actions runner manage to spin up 8 processes (I noticed pytest-xdist not handle scenarios very well if there are more jobs requested than there are CPUs) but this should produce 160 concurrent clusters
There was a problem hiding this comment.
oh right, plus the ordinary tests, so I expect something above 200 ¯_(ツ)_/¯
This should run the tests against staging. I intend to let a couple of A/B tests run with high concurrency