feat: Conditionally emit metrics based on enablement (backport #19903)

[mergify]

Description

Closes: #10245

• Introduction of Now Function: A new function called Now has been added to the telemetry package. This function returns the current system time if telemetry is enabled, or a zero time if telemetry is not enabled.
• Atomic Global Variable: We introduced an atomic global variable to manage the state of telemetry's enablement.
• Conditional Telemetry Emission: All telemetry functions have been updated to include a check that determines whether telemetry is enabled. If telemetry is disabled, these functions return early, avoiding unnecessary operations and overhead.

---

Author Checklist

All items are required. Please add a note to the item if the item is not applicable and
please add links to any relevant follow up issues.

I have...

• included the correct type prefix in the PR title
• confirmed ! in the type prefix if API or client breaking change
• targeted the correct branch (see PR Targeting)
• provided a link to the relevant issue or specification
• reviewed "Files changed" and left comments if necessary
• included the necessary unit and integration tests
• added a changelog entry to CHANGELOG.md
• updated the relevant documentation or specification, including comments for documenting Go code
• confirmed all CI checks have passed

Reviewers Checklist

All items are required. Please add a note if the item is not applicable and please add
your handle next to the items reviewed if you only reviewed selected items.

I have...

• confirmed the correct type prefix in the PR title
• confirmed all author checklist items have been addressed
• reviewed state machine logic, API design and naming, documentation is accurate, tests and test coverage

Summary by CodeRabbit

• New Features

  • Introduced a new Now function in the telemetry module to conditionally emit metrics, enhancing performance by reducing unnecessary operations.
  • Implemented thread-safe management of telemetry's enablement state using a global variable.
  • Adjusted telemetry measurement timing across various modules (BaseApp, circuit, crisis, distribution, evidence, gov, mint, slashing, staking, upgrade) for more accurate tracking of query processing and module execution times.

• Refactor

  • Replaced usage of the time package with the custom telemetry package for time-related operations in several modules, ensuring consistent and accurate performance metrics.

• Tests

  • Added a test suite TelemetrySuite to validate the functionality of telemetry-related functions, ensuring correct behavior when telemetry is enabled and disabled.

---

This is an automatic backport of pull request #19903 done by [Mergify](https://mergify.com).

[mergify]

Cherry-pick of 2496cfd has failed:

On branch mergify/bp/release/v0.50.x/pr-19903
Your branch is up to date with 'origin/release/v0.50.x'.

You are currently cherry-picking commit 2496cfdf5.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   server/start.go
	modified:   telemetry/metrics.go
	modified:   telemetry/wrapper.go
	new file:   telemetry/wrapper_test.go
	modified:   x/crisis/abci.go
	modified:   x/evidence/keeper/abci.go
	modified:   x/slashing/abci.go

Unmerged paths:
  (use "git add/rm <file>..." as appropriate to mark resolution)
	both modified:   CHANGELOG.md
	both modified:   x/circuit/module.go
	both modified:   x/crisis/module.go
	both modified:   x/distribution/abci.go
	deleted by us:   x/gov/keeper/abci.go
	both modified:   x/mint/abci.go
	both modified:   x/staking/keeper/abci.go
	both modified:   x/upgrade/abci.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

[github-advanced-security]

CodeQL found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.