Enhance reconcile of taskrun to avoid extra pod creation

[vincent-pli]

Changes

Fix issue: #1976

The root cause for the issue is:
The reconsile of taskrun hit error:

Failed to update taskRun status, the object has been modified; please apply your changes to the latest version and try again

it's a benign error, we should just ignore it, but current reconcile cannot, see the logic:

• Check if taskrun.Status.Podname is nil, if not, get the pod and move on, one follow-up step is add READY annotation to the pod to make pod run container one by one.
• If taskrun.Status.Podname is nil, create a new pod and move on.

The problem is:
If hit Failed to update taskRun status error, the taskrun will stay in workqueue and reconcile again after increased delay, but the Podname is still nil (although it's not in real world, since the pod has been created), so a extra pod will be created. and the previous pod will stick in Running forever since no one add READY annotation.

Submitter Checklist

These are the criteria that every PR should meet, please check them off as you
review them:

• Includes tests (if functionality changed/added)
• Includes docs (if user facing)
• Commit messages follow commit message best practices

See the contribution guide for more details.

Double check this list of stuff that's easy to miss:

• If you are adding a new binary/image to the cmd dir, please update
  the release Task to build and release this image.

Reviewer Notes

If API changes are included, additive changes must be approved by at least two OWNERS and backwards incompatible changes must be approved by more than 50% of the OWNERS, and they must first be added in a backwards compatible way.

Release Notes

Describe any user facing changes here, or delete this block.

Examples of user facing changes:
- API changes
- Bug fixes
- Any changes in behavior

[vincent-pli]

/test pull-tekton-pipeline-integration-tests

[vincent-pli]

/assign @imjasonh

[ghost]

Nice! Could we also change test/retry_test.go from t.Logf to t.Errorf for this now? Or are there still edge cases that result in too many pods?

[ghost]

/approve

[tekton-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: sbwsg

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [sbwsg]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[chmouel]

master has been failing since last night on every runs :

I just tried it with your PR and it worked! Thanks...

/lgtm