Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Exclude non-eth0 net ifaces from datadog network metrics #197

Open
cirocosta opened this issue Jan 6, 2020 · 2 comments
Open

Exclude non-eth0 net ifaces from datadog network metrics #197

cirocosta opened this issue Jan 6, 2020 · 2 comments

Comments

@cirocosta
Copy link
Member

cirocosta commented Jan 6, 2020

Currently, the network metrics being collected by the datadog agent that we have
configured for all of our machines include not only "real" interfaces (like,
eth0, which we care about), but also the other ones that we don't care about
(bridges and the host-facing side of the virtual eth pairs).

Given that for each container that we have, we end up with at least 2 of those
(a bridge and a host-side veth), and that for each unique resource config, one
of those get created every hour, that's a lot of distinct timeseries.

Getting rid of these is important because otherwise we'll be paying the price of
something we don't use (similar to #195).
The configuration for this can be done in a similar way too:

  • the network core collector contains a field that lets us specify a regex for
    to see whether to include or not a gven interface:
## @param excluded_interface_re - string - optional
## Completely ignore any network interface matching the given regex.
#
# excluded_interface_re: <NETWORK_INTERFACE_NAME>.*

https://github.com/DataDog/integrations-core/blob/86c5e1caabaed489b19dc5cb172545fda9b99ee7/network/datadog_checks/network/data/conf.yaml.default#L25-L28

leveraging that, we can ignore anything that's not an eth0.

for instance, looking at one of the machine's ifaces, we can come up with the
following:

import re

text = ['eth0', 'wlkcnldkuudo-0', 'wbrdg-0afe0014']

p = re.compile('(?!eth0)')

for t in text:
    if p.match(t):
        print("match: ", t)
match:  wlkcnldkuudo-0
match:  wbrdg-0afe0014

i.e., it'd exclude the ifaces we don't care about.

thanks!

@cirocosta
Copy link
Member Author

I just realized that although we can get rid of the automatic network config generation that the BOSH release does

  dd.generate_network_config:
    default: yes
    description: Automatically generate network monitoring integration, network.yaml

https://github.com/DataDog/datadog-agent-boshrelease/blob/66cf5946b9d7b9fa6358c6267856c9cc79a09cdd/jobs/dd-agent/spec#L182-L184

it seems like there's no way of supplying a network config lol

# Network
<% if p('dd.generate_network_config') == true || p('dd.generate_network_config') =~ (/(true|t|yes|y|1)$/i) %>
mkdir -p ${CONFD_DIR}/network.d/
cat <<EOF > "${CONFD_DIR}/network.d/network.yaml"
---
init_config:
instances:
  - collect_connection_state: <%= p("dd.generate_network_config_connection_state", "yes") %>
    excluded_interfaces:
<% p("dd.generate_network_config_excluded_interfaces", []).each do |inet| %>
      - <%= inet %>
<% end %>
EOF
<% end %>

which would mean that we'd probably need make a PR first to either supply the full config (like it's possible with disk), or get a field for supplying excluded_interface_re (like it's currently possible for the non-regexp based matching).

@cirocosta cirocosta changed the title Exclude bridges and virtual interfaces from datadog network metrics Exclude non-eth0 net ifaces from datadog network metrics Jan 6, 2020
@cirocosta
Copy link
Member Author

I just created an issue on the datadog BOSH release: https://github.com/DataDog/datadog-agent-boshrelease/issues/106

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant