Envoy does duplicate DNS resolutions when there are strict DNS clusters, with the same endpoints

[nirvanagit]

Title: Envoy does duplicate DNS resolutions when there are two strict DNS clusters, and both have the same endpoints

Description:

Describe the desired behavior, what scenario it enables and how it would be used.

Envoy bombards the DNS server, with duplicate DNS resolution calls when multiple STRICT DNS clusters have the same endpoint.

In our setup, we have around 5-7k STRICT DNS clusters, but there are only 340 unique cluster endpoints. At startup, envoy tries to do DNS resolutions for all 5-7k STRICT DNS clusters, and fails for different reasons.

This has resulted in envoy throwing UH errors for a short window just after it marks itself READY.

If Envoy's DNS resolution mechanism can be optimized such that it doesn't start duplicate DNS resolutions, then it would benefit in several ways, for instance:

1. Envoy would not throw UH errors during startup because of failed DNS resolutions. These failed DNS resolutions have happened due to a variety of reasons. Read this thread for more information.
2. Envoy would bootup faster, as it reduces the number of DNS resolutions it would need to do
3. Envoy would not bombard the DNS servers. For example, Amazon DNS server has a limitation of 1024 packets per seconds.

My proposal is to do the following:

1. Envoy does not do duplicate DNS resolutions
2. Envoy batches DNS resolutions and doesn't bombard the DNS server with all DNS resolutions at the same time
3. Envoy should not mark itself ready even if 1 DNS resolution failed. This is critical for systems like API Gateway, where even a single endpoint without DNS resolution can impact traffic.

Relevant Links

Any extra documentation required to understand the issue.

• Issue explaining how large number of STRICT DNS clusters is causing UH / no healthy upstream - DNS resolution failure results in UH / no healthy upstream #31992

[nirvanagit]

Tagging @yuval-k for visibility. Thanks!

[nirvanagit]

@lizan - thanks for looking at this issue, wanted to know if there is anything I can help with to add some traction to this ticket?

This is causing issues due to our scale.

[github-actions]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.

[github-actions]

This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as "help wanted" or "no stalebot". Thank you for your contributions.

[nirvanagit]

Can this be reopened.