Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

No connection re-use on HTTP executor #904

Open
spy16 opened this issue Feb 22, 2021 · 2 comments
Open

No connection re-use on HTTP executor #904

spy16 opened this issue Feb 22, 2021 · 2 comments

Comments

@spy16
Copy link

spy16 commented Feb 22, 2021

Is your feature request related to a problem? Please describe.
The HTTP executor seems to be creating new http.Client for every job execution. This causes 2 types of problems:

  1. (Minor) Poor response times or job execution times because new connections are created every time instead of utilising connection pool maintained by http.Client
  2. (Major) If the server supports Keep Alive connections, repeated job executions end up creating multiple persistent idle connections to the target host eventually causing file-descriptor limit breach on the server side. (If the server handles the timeout on idle connections properly or sends a Connection: close on every response, this is handled. But it would still be ideal to handle this properly from the client)

Describe the solution you'd like

Re-use the http.Client instance in the HTTP executor and allow configuring MaxIdleConns and MaxIdleConnsPerHost configs of the http client. (Handling TLS configs would be slightly tricky I think -- But may be it would at least help if the client instances can be cached against Job ID)

Describe alternatives you've considered
N/A

Additional context
N/A

@ncsibra
Copy link

ncsibra commented Aug 11, 2021

I agree with this issue, we had a goroutine/memory leak, because the TCP connections were kept alive, but the client was not reused, so the resources weren't freed on the server side.
A Connection: close header solves this, but it shouldn't be necessary if dkron reuses the client.

@vcastellm
Copy link
Member

I will leave this open to tackle the connection re-use at some point.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants