[exporter/datadog]: update error formatting for error spans that are not exceptions.

[ericmustin]

Description:

This PR adds updated default values for some Datadog Span error tags in the case the OTLP Span has a status set to error, and doesn't have any exception Span.Events, but has either a Span.Status.Message or http status code attributes.

In this case, which is common with http clients, the exporter now updates default values of Datadog Span tag to:

• error.type defaults to error.

• If the OTLP Span does not have a status.Message, then error.msg defaults to an interpolated string "<AttributeHTTPStatusCode> <AttributeHTTPStatusText>", if those attributes exist on the otlp span.

• In the case that there is no http status information, the error.msg and error.type fields remain unset, and the Datadog UI handles displaying a reasonable type/msg. This matches the default behavior of the datadog agent (see example https://a.cl.ly/Z4uKzKoL).

For context, these tags have special treatment in the Datadog APM UI when displaying error spans.

Previously these were defaulting to ERR_CODE_2, which had been left in for backward compatibility.

note: This would maybe constitute a breaking change (regardless of what's permitted by semver ), but I think it is reasonable to do regardless, as it is a recurring source of confusion for users, and a bug not an intended default.

Link to tracking Issue:

Testing: Added a unit test, updated existing test.

Documentation:

[mx-psi]

You mention this is a breaking change. Do you think it's worth it to first add a warning for a release and then enable this?

[ericmustin]

@mx-psi re: >You mention this is a breaking change. Do you think it's worth it to first add a warning for a release and then enable this?

No i think we should prioritize getting the change out, and make sure it's included in the release notes, which are a better source of truth than warning logs. I believe we have the capability/resources to message potentially impacted end users via email, so we can take that approach if we deem necessary.

[mx-psi]

We can avoid the warning then. Just two small comments since the code is a bit hard to read with 5 levels of nesting and we can reduce the complexity a little.