Upgrading to DD 0.45.0 breaks StatsD metrics

[joshhoegen]

Describe the bug
After upgrading to 0.45.0 with Python 3.11, our metrics stopped reporting. It seems only the combo of Python 3.10 and datadogpy 0.44.0 fixed our issue. This repo's build from 3 weeks ago also has some failures (I'm not sure if this is related: https://dev.azure.com/datadoghq/datadogpy/_build/results?buildId=131198&view=logs&j=abc39910-4c32-5be0-0b39-6d3f303c7218&t=837b8d2b-2a00-5222-b449-dfcd3b59faed&l=825)

To Reproduce
Steps to reproduce the behavior:

1. Upgrade Python to 3.11
2. Upgrade Datadog to 0.45.0
3. In my case I ran a Fast API app with Gunicorn, using uvicorn for workers
4. No errors, stats aren't sent to DD

Expected behavior
Stats are sent to DD and visible in dasboard

Screenshots
If applicable, add screenshots to help explain your problem.

Environment and Versions (please complete the following information):
pyproject.toml (poetry)

[tool.poetry]
name = "article-analyzer-api"
version = "0.1.0"
description = ""
authors = []

[tool.poetry.dependencies]
python = "^3.10"
psycopg2-binary = "*"
boto3 = '*'
fastapi = "*"
uvicorn = "*"
datadog = "0.45.0"
setproctitle = "^1.3.2"

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

[tool.poetry.scripts]
start = "article_analyzer_api.app:start"

Docker image

python:3.11-slim-buster

Usage

from datadog import initialize, statsd

            dd_options = {
                'statsd_port': 8125,
                'statsd_host': 'statsd',
                'statsd_namespace': 'my_namespace'
            }

            initialize(**dd_options)
            self.ddog = statsd
            
            ######
            
            statsd.increment('my_val')

Additional context

• Agent Version: Unknown (can I see it anywhere in https://{my_org}.datadoghq.com/containers?)
• No errors as noted above
• Running a python:3.11-slim-buster Docker instance on ECS

[sgnn7]

@joshhoegen Can you provide some more info on your problem? It's not clear what the problem area may be from the description.

• What Agent version you are sending information to?
• Any log errors in the Agent's log if you're sending this over regular statsd server?
• Can you provide the import and snippet lines of the code that's doing this sending?
• What type of environment is your client running in (containerized/k8s, bare metal, etc)?
• Have you tried using Python 3.10 with 0.45 to eliminate Python version change as the culprit?

[joshhoegen]

@sgnn7, I've updated the issue above.

• What Agent version you are sending information to?
  • I don't have access to the dd-agent, is there somewhere I can find it in https://{my_org}.datadoghq.com/containers? I imagine it's not the latest, but can't confirm right now.
• Any log errors in the Agent's log if you're sending this over regular statsd server?
  • No
• Can you provide the import and snippet lines of the code that's doing this sending?
  • Added above
• What type of environment is your client running in (containerized/k8s, bare metal, etc)?
  • Running a Docker instance on ECS
• Have you tried using Python 3.10 with 0.45 to eliminate Python version change as the culprit?
  • Yes

[sgnn7]

@joshhoegen Thanks for the info! I have a few followups:

I don't have access to the dd-agent, is there somewhere I can find it in

You might be able to find it in the infrastructure list but it depends on your setup topology. The easiest way to find it is in the first log line that the agent emits (and stores in agent.log) when it initializes. It's also available in the agent pod with agent version command.

Have you tried using Python 3.10 with 0.45 to eliminate Python version change as the culprit?

I'm not quite clear if you meant that it was ok or not. From this matrix below, what was tested and what worked / didn't work (I filled in the things that I think you mentioned)?

• Python 3.10 + Datadogpy 0.44 = OK
• Python 3.10 + Datadogpy 0.45 = ?
• Python 3.11 + Datadogpy 0.44 = ?
• Python 3.11 + Datadogpy 0.45 = FAIL

Some other long-shot questions:

• If you turn on debug logging on failing setup, can you attach a sanitized log here?
• Is there any way for you to capture packets sent by your Python client (tcpdump or similar) and see if anything is sent at all?

[sgnn7]

@joshhoegen / @mark-a-dev I did a simple test using the same invocation code (without Poetry) with virtualenv on Ubuntu 22.04 and Agent 7.43.1. All combinations of Python 3.10/3.11 and Datadogpy 0.44.0/0.45.0 seem to send the data fine so we will for sure need more info on this ticket to proceed.

[github-actions]

Thanks for your contribution!

This issue has been automatically marked as stale because it has not had activity in the last 30 days. Note that the issue will not be automatically closed, but this notification will remind us to investigate why there's been inactivity. Thank you for participating in the Datadog open source community.

If you would like this issue to remain open:

1. Verify that you can still reproduce the issue in the latest version of this project.

2. Comment that the issue is still reproducible and include updated details requested in the issue template.

[aexvir]

@sgnn7

@joshhoegen / @mark-a-dev I did a simple test using the same invocation code (without Poetry) with virtualenv on Ubuntu 22.04 and Agent 7.43.1. All combinations of Python 3.10/3.11 and Datadogpy 0.44.0/0.45.0 seem to send the data fine so we will for sure need more info on this ticket to proceed.

we've hit the same issue with a standalone datadog/dogstatsd:7.35.2 deployment
could you maybe test that version? our agent shows a bunch of errors like

2023-05-26 10:29:40 UTC | DSD | ERROR | (pkg/dogstatsd/server.go:592 in errLog) | Dogstatsd: error parsing metric message '"lynx.rule.cache:********@0.25|#state:hit,environment:production,color:purple|c:e98432af157746286894eb9fdc9d6e7294cd68e4413b20a06825d85563948830"': invalid dogstatsd message format

[john-bodley]

We've already run into the same issue, i.e., statsD metrics not being reported in Datadog, using Python 3.9 with datadog version 0.45.0. Downgrading to version 0.44.0 remedies the issue.

[randori-ddejohn]

@john-bodley @sgnn7 @aexvir @joshhoegen

0.45.0 added container IDs to metrics -- look into whatever you're using to consume your metrics as it may not support that new datagram format: https://docs.datadoghq.com/developers/dogstatsd/datagram_shell/?tab=metrics#dogstatsd-protocol-v12

Added in #720

[asottile-sentry]

we hit the same thing -- it's the new container metrics. there's no easy way to turn it off in code only an environment variable

can you add a similar opt-out to the telemetry opt-out?

for instance I can do:

statsd.disable_telemetry()

it would be nice to also be able to

statsd.disable_origin_detection()

especially because it silently breaks metric delivery when enabled