Kibana server fails to handle long query string parameters

[azasypkin]

Kibana version: 7.5+ (haven't checked on earlier versions)

Browser version: any browser

Describe the bug: when opening Kibana with a very long query string parameters Kibana returns 400 and logs server error [11:40:44.623] [error][client][connection] Error: Parse Error in the terminal. It's a problem for the SAML authentication when we try to grab Kibana URL fragment and pass it to the server within a query string parameter.

Steps to reproduce:

1. Open this link (e.g. 'http://localhost:5601/app/kibana?query=' + 'kibana'.repeat(1500)) on your local Kibana instance
2. Observe a white screen in browser
3. Observe error in the terminal

Expected behavior: Either such long URL should be properly processed and forwarded to the handler if any or error should be clear

Errors in browser console (if relevant): no errors visible, only 400 in the browser network tab

Provide logs and/or server output (if relevant): server error [11:40:44.623] [error][client][connection] Error: Parse Error

Any additional context: localhost_Archive [19-12-18 11-45-35].zip

/cc @elastic/kibana-security @restrry

[elasticmachine]

Pinging @elastic/kibana-platform (Team:Platform)

[pgayvallet]

The actual error:

{ [Error: Parse Error]
  bytesParsed: 8192,
  code: 'HPE_HEADER_OVERFLOW',

obtained by adding debug logging from

kibana/src/core/server/http/http_tools.ts

Lines 109 to 113 in 1c415e0

	server.listener.on('clientError', (err, socket) => {
	if (socket.writable) {
	socket.end(Buffer.from('HTTP/1.1 400 Bad Request\r\n\r\n', 'ascii'));
	} else {
	socket.destroy(err);

No associated stack trace, unfortunately.

From https://stackoverflow.com/questions/35328725/hpe-header-overflow-exception-when-make-http-request this is directly related to node parser.

Starting with node --max-old-space-size=8000 --max-http-header-size=80000 scripts/kibana.js --no-base-path --dev fixes the issue.

This is related to operations, not platform. Not sure we should do anything though. Very long urls are a bad practice, can be problematic for some proxies/middlewares (such as IIS), and that's basically what payloads are for.

[elasticmachine]

Pinging @elastic/kibana-operations (Team:Operations)

[mshustov]

@pgayvallet Shouldn't we log clientError at least? Otherwise, an exception is swallowed and users have no idea how to solve their problem.

[pgayvallet]

We should probably do so.

I couldn't find where (in the legacy I guess?) the trapped error log
(server error [11:40:44.623] [error][client][connection] Error: Parse Error)
is coming from. We should probably remove it and handle low-level server errors in NP instead.

[azasypkin]

Thanks for investigation! Since it's a natural limit of NodeJS we'll see how we can handle our specific SAML issue without hitting this limit instead in #53478. Looks like this issue isn't really actionable apart from adding a better logging.

[jbudz]

We can bump the limit if needed - we have done once before. The only thing we're looking for on the ops side is to avoid infinite headers. It;s set in bin/kibana

[azasypkin]

We can bump the limit if needed - we have done once before. The only thing we're looking for on the ops side is to avoid infinite headers. It;s set in bin/kibana

Ah, that's interesting, thanks! I didn't realize that we already bumped max-http-header-size to 64k in prod, that sounds like more than enough. Let me check with the reporter what URL they used to trigger that issue.

[tylersmalley]

If the limit is changed, we should coordinate with Cloud to ensure the increase is supported in their routing layer.

[jordansissel]

Proposal: Instead of capturing the url prior to SAML redirect into the URL, can we capture the URL into localStorage? This would remove it from the URL and Kibana's web javascript side would just look at the localStorage for a place to jump to after landing back at the SAML auth redirect.

Impact:

1. Kibana URLs can stay long
2. Proxies and load balancers don't need to be modified at all to support this

[azasypkin]

Proposal: Instead of capturing the url prior to SAML redirect into the URL, can we capture the URL into localStorage? This would remove it from the URL and Kibana's web javascript side would just look at the localStorage for a place to jump to after landing back at the SAML auth redirect.

Impact:
Kibana URLs can stay long
Proxies and load balancers don't need to be modified at all to support this

Well, it's not that simple as it may seem since we need to consider a number of things 😉 If you're curious what options we considered and why we come up with what we have now you can follow the discussion here and here.

Having said that we have a plan on how to solve it nicely and completely, but we're still missing some pieces from the platform (proper custom standalone HTML views) and security (server-side sessions). It just a matter of time and priority, but we'll eventually get there (luckily only SAML is suffering from this, OpenID Connect and other SSO mechanisms we supoort don't have such issues).

[jordansissel]

it's not that simple

I mainly suggested this because I use cookies/localStorage for this same purpose with OAuth2 on some internal apps.

Thanks for the links to the background! I enjoy reading these kinds of things <3

[azasypkin]

I'm closing this issue since with server-side session we've introduced in 7.10 long URLs is no longer a problem for Kibana's SSO providers (i.e. example from #53464 (comment) works well on 7.10) and --max-http-header-size=65536 we use by default is already a reasonably large value.

Thanks everyone!