CI: separate JJBB definitions for pull requests, PR merge and external e2e calls

[pazone]

What does this PR do?

Creates 3 separate JJBB workflows:

1. e2e-testing-mbp.yml - For external e2e tests executions ONLY. When another pipeline triggers the tests
2. e2e-testing-pr-mbp.yml - For pull requests to this repo ONLY
3. e2e-testing-pr-merge-mbp.yml - For PR merges to main ,8.x or 7.17

Why is it important?

Single Jenkinsfile for all operations is overcomplicated

Related issues

• Relates Use preinstalled dependencies for ubuntu #3169

[mergify]

This pull request does not have a backport label. Could you fix it @pazone? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-v./d./d./d is the label to automatically backport to the 7./d branch. /d is the digit
  NOTE: backport-skip has been added to this pull request.

[elasticmachine]

❕ Build Aborted

There is a new build on-going so the previous on-going builds have been aborted.

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2022-12-02T11:35:15.431+0000

• Duration: 4 min 33 sec

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

[v1v]

there is no need to backport the jjb files

[v1v]

What's the reason for using two new MBPs rather than the traditional MBP with the same pipeline for branches and PRs?

[pazone]

What's the reason for using two new MBPs rather than the traditional MBP with the same pipeline for branches and PRs?

@v1v To have separate Jenkinsfiles. Can we use one jjb for multiple pipeline scripts?

[kuisathaverat]

What's the reason for using two new MBPs rather than the traditional MBP with the same pipeline for branches and PRs?

We need a pipeline job per branch and PR so MBP fit the use case. Also, the MBP creates the jobs on GitHub events and registers the triggers, with regular pipelines in case of reprovision of the instance we will lose the triggers and we have to manually trigger the jobs. It is not the real reason why @pazone did it, but it is a good reason to do it for jobs that have triggers.

[v1v]

We need a pipeline job per branch and PR so MBP fit the use case.

I don't understand the reason yet for two different Multibranch pipelines, one related to the PRs and another one for the branches.

I understand there is no need to use a single pipeline but MBP instead.

As far as I see, the proposal for PRs and branches could be merged in the same multibranch pipeline, aka MBP, as we normally do in other places.

Can you please elaborate what's the difference between the merge.groovy and pull-request.groovy files? And what's the reason they cannot be merged in one groovy` file instead?

[pazone]

We need a pipeline job per branch and PR so MBP fit the use case.

I don't understand the reason yet for two different pipelines one related to the PRs and another one for the branches.

As far as I see, the proposal for PRs and branches could be merged in the same pipeline as we normally do in other places.

Can you please elaborate what's the difference between the merge.groovy and pull-request.groovy files? And what's the reason they cannot be merged in one groovy` file instead?

@v1v pull request flow will be a bit more complicated.
If ansible stuff is changed it will build temporary AMIs, run tests using new VMs and deregister the AMIs
Merge workflow builds and publishes new AMIs marked as ready to use .ci/ansible is changed.
It's made in order to avoid code complexity and make pipelines straightforward.

[v1v]

If ansible stuff is changed it will build temporary AMIs, run tests using new VMs and deregister the AMIs

If no ansible stuff is changed, what's the flow in place for both cases (prs and branches)?

IIUC, those are not independent pipelines but stages, will something like the below pseudo-code could work?

stage('prs') {
...
}
stage('branches') {
...
}

I somehow prefer to use the same pipeline for branches and PRs. Maybe there is a need to:

• One pipeline for ansible stuff, regardless of the type of changes (PRs or branches)
• Another pipeline for non-ansible stuff, regardless of the type of changes (PRs or branches)

And the pipeline ansible stuff will trigger the non-ansible stuff so it can test other things given the AMIs.

I'm just guessing, as I don't know the implementation details about the pipeline specifics.

In any case, I'm not blocking this PR, but commenting about this

[kuisathaverat]

this changes the current behavior on the stable branch without a change in the pipelines, this breaks the CI. the PR will not be processed until all new pipelines are develop.

[kuisathaverat]

A proposal here, do not touch the main pipeline (.ci/jobs/e2e-testing-mbp.yml) until everything is working. So, create a new file like this one with a different job.name, and use it until you have a process refined and working. For that, you will have to limit the head-filter-regex to a specific branch(where you are working) and PR (only your test PRs). then we can see which changes we can incorporate on the main branch without issues.

[kuisathaverat]

WYT @v1v

[pazone]

@kuisathaverat Great solution! Could you please take a look now?

[kuisathaverat]

we still making changes in .ci/jobs/e2e-testing-mbp.yml that is something we do not want

[kuisathaverat]

no here

[kuisathaverat]

create a new job from scratch

[kuisathaverat]

limit the branches to your working branches and PRs

[elasticmachine]

💔 Build Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2022-12-02T11:36:43.696+0000

• Duration: 9 min 51 sec

Pipeline error

This error is likely related to the pipeline itself. Click here
and then you will see the error (either incorrect syntax or an invalid configuration).

❕ Flaky test report

No test was executed to be analysed.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.