[resolved] Courier Fetch Error: unhandled courier request error: Authorization Exception in Chrome/Safari on Kibana 4.5.0

[jtryon]

Edit: This issue can be fixed with only configuration changes. See this comment from further down this thread for more details.

---

Description of the problem including expected versus actual behavior:
This issue appeared after upgrading from 4.3.2 to 4.5.0 for me. I'm running debian 64bit with Elasticsearch 2.3.

UI does not load in Chrome or Safari. It does in Firefox.

Steps to reproduce:
1.Launch Browser
2.Open Kibana

Provide logs (if relevant):
`Oops!
Looks like something went wrong. Refreshing may do the trick.

Go back or clear your session

Fatal Error
Courier Fetch Error: unhandled courier request error: Authorization Exception
Version: 4.5.0
Build: 9889
Error: unhandled courier request error: Authorization Exception
at handleError (http://elkarti.blank.com/bundles/kibana.bundle.js?v=9889:78902:23)
at DocRequest.AbstractReqProvider.AbstractReq.handleFailure (http://elkarti.blank.com/bundles/kibana.bundle.js?v=9889:78822:15)
at http://elkarti.blank.com/bundles/kibana.bundle.js?v=9889:78716:18
at Array.forEach (native)
at http://elkarti.blank.com/bundles/kibana.bundle.js?v=9889:78714:19
at processQueue (http://elkarti.blank.com/bundles/commons.bundle.js?v=9889:42357:29)
at http://elkarti.blank.com/bundles/commons.bundle.js?v=9889:42373:28
at Scope.$eval (http://elkarti.blank.com/bundles/commons.bundle.js?v=9889:43601:29)
at Scope.$digest (http://elkarti.blank.com/bundles/commons.bundle.js?v=9889:43412:32)
at Scope.$apply (http://elkarti.blank.com/bundles/commons.bundle.js?v=9889:43709:25)
Fatal Error
Courier Fetch: unhandled courier request error: Authorization Exception
Version: 4.5.0
Build: 9889
Error: unhandled courier request error: Authorization Exception
at handleError (http://elkarti.blank.com/bundles/kibana.bundle.js?v=9889:78902:23)
at DocRequest.AbstractReqProvider.AbstractReq.handleFailure (http://elkarti.blank.com/bundles/kibana.bundle.js?v=9889:78822:15)
at http://elkarti.blank.com/bundles/kibana.bundle.js?v=9889:78716:18
at Array.forEach (native)
at http://elkarti.blank.com/bundles/kibana.bundle.js?v=9889:78714:19
at processQueue (http://elkarti.blank.com/bundles/commons.bundle.js?v=9889:42357:29)
at http://elkarti.blank.com/bundles/commons.bundle.js?v=9889:42373:28
at Scope.$eval (http://elkarti.blank.com/bundles/commons.bundle.js?v=9889:43601:29)
at Scope.$digest (http://elkarti.blank.com/bundles/commons.bundle.js?v=9889:43412:32)
at Scope.$apply (http://elkarti.blank.com/bundles/commons.bundle.js?v=9889:43709:25)
`

Request and response codes:
`POST /elasticsearch/_mget?timeout=0&ignore_unavailable=true&preference=1459445925363 HTTP/1.1
Host: elkarti.blank.com.ca:5601
Connection: keep-alive
Content-Length: 62
Accept: application/json, text/plain, /
Origin: http://elkarti.blank.com.ca:5601
kbn-version: 4.5.0
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.87 Safari/537.36
Content-Type: application/json;charset=UTF-8
Referer: http://elkarti.blank.com.ca:5601/app/kibana
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.8,da;q=0.6

HTTP/1.1 403 Forbidden, kbn-name: kibana, kbn-version: 4.5.0
cache-control: no-cache
Date: Thu, 31 Mar 2016 17:38:39 GMT
Connection: keep-alive
Transfer-Encoding: chunked`

[nokiomanz]

Hi,

Same thing here. But upgrading from kibana 4.4 to 4.5. Elasticisearch 2.3, freshly upgraded.
I am using amazonlinux(centos like)
The installation was done using the yum repo.
I also did try the x64 tar. I have the exact same error log that was provided by jtryon.
I did try using kibana directly from port 5601 as well as what I do in my setup through apache.

Internet explorer and firefox work
Safari and Chrome does not

[epixa]

This is apparently happening with ES 2.3.0 regardless of Kibana version. @nokiomanz was able to reproduce the error using 4.4.2 and 2.3.0.

[epixa]

I can't reproduce the issue on 32 bit linux, so it must be specific to 64 bit.

[epixa]

From @nokiomanz in IRC: "I grabbed the tar of both kibana 4.5 and ES 2.3. Un tar them and ./kibana ./elasticsearch Reached kibana locally on my ubuntu 64bits desktop on port 5601. And it work."

[jtryon]

Reverted my kibana version to 4.3.1 for testing and I see the same issue.
Leads me to think it's related to ES2.3 also.

[PSiAU]

Interesting, i'm running Kibana 4.5, ES 2.3, using Safari and i'm not seeing this issue.
This is on a 64-bit Debian OS.

[nokiomanz]

PSiAU, Is this using a fresh install or an upgraded one?

[nokiomanz]

This is what I have in my kibana log
http://pastebin.com/rsKcHhwa

[PSiAU]

@nokiomanz An upgraded install.

[Tak-MK]

Running Kibana 4.4.2 with ES2.3 and fails.
I updated to Kibana 4.5, but the error won't go away.

[epixa]

@Tak-MK Does it fail with the same error as above? What OS do you have Kibana and ES installed on? What browsers does it fail in?

[Tak-MK]

@epixa I have it installed on an Amazon Linux x64, and it works using Firefox, but fails with Opera/Chrome.
And yes, it's the same "Courier Fetch Error"

[epixa]

Can anyone that's affected by this let me know exactly what version of java they're running for elasticsearch? Ideally vendor (e.g. Oracle), type (e.g. JDK), and version.

[nokiomanz]

$ java -version
openjdk version "1.8.0_77"
OpenJDK Runtime Environment (build 1.8.0_77-b03)
OpenJDK 64-Bit Server VM (build 25.77-b03, mixed mode)
$ rpm -qa |grep java
java-1.8.0-openjdk-headless-1.8.0.77-0.b03.9.amzn1.x86_64
java-1.8.0-openjdk-1.8.0.77-0.b03.9.amzn1.x86_64

[Tak-MK]

$ java -version
java version "1.8.0_65"
Java(TM) SE Runtime Environment (build 1.8.0_65-b17)
$ rpm -qa |grep java
java-1.7.0-openjdk-1.7.0.99-2.6.5.0.66.amzn1.x86_64
tzdata-java-2016a-1.36.amzn1.noarch
javapackages-tools-0.9.1-1.5.amzn1.noarch

[epixa]

For those affected, exactly how did you install both kibana and elasticsearch? I'm unable to reproduce this using 64bit ubuntu 14.04, openjdk 8, and the .tar downloads from the site.

[Tak-MK]

I've installed them by using elastic.co repos.

[nokiomanz]

I Have been using the repo since the start of my cluster about a year and a half ago. I have been udating my cluster every month.

Yum install elasticsearch at first and then yum update every month.
For kibana, I used the tar at first and since the introduction of the repo I have been using the repo

[epixa]

I'm pretty sure I've narrowed this down to the http.cors.enabled configuration on elasticsearch. Kibana 3 required this, but Kibana 4 no longer uses it. Removing that configuration from your <elasticsearch dir>config/elasticsearch.yml should fix this issue.

[nokiomanz]

Can confirm that on my side removing http.cors.enabled from my elasticsearch client config did the trick

[Tak-MK]

On my side too, thanks!

[edeak]

@nokiomanz I can confirm too.

[NeckBeardPrince]

+1 Same here

[epixa]

I'm going to leave this open for a bit since I suspect more people are going to be searching for this issue.

The fix:

If you have http.cors.enabled set in your elasticsearch configuration for Kibana's sake, remove it. This configuration is no longer necessary in Kibana 4.

If you require http.cors.enabled in your elasticsearch config for some other reason, then as of ES 2.3.0, you must also set http.cors.allow-origin: https://www.elastic.co/guide/en/elasticsearch/reference/2.3/breaking_20_setting_changes.html#_cors_allowed_origins

Note: You must make these changes in elasticsearch.yml rather than the cluster update settings api. See this comment by @czerasz for links to details.

[alexforster]

Also experienced this issue starting about two days ago, and also resolved the issue by removing http.cors.enabled.

[epixa]

Thanks @czerasz. I've added that as a note in my "the fix" comment.

[pickypg]

I believe it's related to the allow-headers that I have defined.

I think that I was wrong. I wrote a small proxy and got in-between Kibana and Elasticsearch.

var proxy = require('express-http-proxy');
var app = require('express')();

app.use('/', proxy('localhost:9200', {
  forwardPath: function(req, res) {
    return require('url').parse(req.url).path;
  },
  decorateRequest: function(req) {
    console.log("HEADERS: ", req.headers);
  }
}));

app.listen(3000);

Run it with:

$ npm install express-http-proxy
$ npm install express
$ node proxy.js

Modified my Kibana config:

elasticsearch.url: "http://localhost:3000"

Finally, I waited for a call to go out (in this case it was via the Marvel plugin). I caught two separate requests, which seem to highlight the issue:

{ 'x-forwarded-for': '127.0.0.1',
  'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.110 Safari/537.36',
  referer: 'http://localhost:5601/app/marvel',
  'kbn-version': '4.5.0',
  'accept-language': 'en-US,en;q=0.8',
  'x-forwarded-port': '57663',
  cookie: '_ga=GA1.1.1243869342.1410753201',
  accept: 'application/json, text/plain, */*',
  'x-forwarded-proto': 'http',
  'accept-encoding': 'gzip, deflate, sdch',
  connection: 'close' }

and

{ 'x-forwarded-for': '127.0.0.1',
  'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.110 Safari/537.36',
  referer: 'http://localhost:5601/app/marvel',
  'kbn-version': '4.5.0',
  'accept-language': 'en-US,en;q=0.8',
  'content-type': 'application/json;charset=UTF-8',
  'x-forwarded-port': '57663',
  origin: 'http://localhost:5601',
  cookie: '_ga=GA1.1.1243869342.1410753201',
  accept: 'application/json, text/plain, */*',
  'x-forwarded-proto': 'http',
  'accept-encoding': 'gzip, deflate',
  connection: 'close' }

Setting a much broader allow-origin (specifically * or /.*/) works, but it does defeat the purpose of it. #6484 should remove the origin header though, which should resolve this issue.

It would probably be good to drop a lot of the other headers going to ES as well. There's no need to send most of these to ES (e.g., referer, user-agent, x-forwarded-*).

[jpgil]

I've seen the same problem unanswered in other issues (1, 2) and forums when upgrading to Kibana 4.5 and ES 2.3, with this particular flavour:

Error: unhandled courier request error: socket hang up
    at handleError (http://10.100.3.89:5601/bundles/kibana.bundle.js?v=9889:89249:23)
    at DocRequest.AbstractReqProvider.AbstractReq.handleFailure (http://10.100.3.89:5601/bundles/kibana.bundle.js?v=9889:89169:15)
    at http://10.100.3.89:5601/bundles/kibana.bundle.js?v=9889:89063:18
    at Array.forEach (native)
    at http://10.100.3.89:5601/bundles/kibana.bundle.js?v=9889:89061:19
    at processQueue (http://10.100.3.89:5601/bundles/commons.bundle.js?v=9889:41836:29)
    at http://10.100.3.89:5601/bundles/commons.bundle.js?v=9889:41852:28
    at Scope.$eval (http://10.100.3.89:5601/bundles/commons.bundle.js?v=9889:43080:29)
    at Scope.$digest (http://10.100.3.89:5601/bundles/commons.bundle.js?v=9889:42891:32)
    at Scope.$apply (http://10.100.3.89:5601/bundles/commons.bundle.js?v=9889:43188:25) 

In my cases the "socket hang up" problem was shown because I had http.compression: true in my elasticsearch.yml. I tried with Kibana 4.4.1 and the problem was the same. Using Chrome I saw HTTP responses and I found a "502 bad gateway" when connecting to ES. Then I went to elasticsearch.yml and removed http.compression. I got a new error message: "Courier Fetch Error: unhandled courier request error: Authorization Exception". That lead me to this page. Then I removed http.cors.enabled and I was able to get Kibana 4.5 (and 4.4.1) working.

[epixa]

Thanks for the detailed post, @jpgil. Hopefully that will help someone in a similar situation.

[nielsbasjes]

Ok, disabling http.cors works for me too.
Yet I consider disabling a security feature in general a "Bad idea"
So what is the correct config for http.cors that will make Kibana work?

[epixa]

@nielsbasjes That's a great attitude to have, and the most secure thing you can do with regard to CORS in Elasticsearch is to not enable it in the first place.

CORS is a browser feature that servers cannot strictly disable, and the default behavior when a server is not configured for CORS is actually the most secure behavior - no destructive (e.g. POST, PUT, DELETE) cross-origin requests are allowed.

Part of CORS is to allow servers to broadcast which domains they would accept cross-origin requests from. In the past, simply enabling CORS on Elasticsearch actually resulted in the most generous behavior possible - cross-origin requests were allowed from any domain.

In 2.3, Elasticsearch changed this behavior so that enabling CORS support is a two step process that requires not only enabling CORS but also explicitly defining which domains you wanted to allow cross-origin requests from.

So you see, you're not disabling a security feature in by not enabling CORS support in Elasticsearch. If anything, you're making your cluster slightly more secure.

[nielsbasjes]

So what I now understand is that the default is "no cross-origin requests" and by setting this parameter you can configure other urls from which those "cross-origin requests" should be allowed.

Then I find it strange that if I enable this feature with the "additional" (?) host 'localhost' (as shown in the documentation) that somehow things that work with the default setting (no cross-origin requests) get blocked when I add an allowed host(pattern).

Is this perhaps a bug?

[epixa]

I'm not entirely following you, but it's probably worth mentioning at this point that the fact that Kibana was even triggering CORS behaviors in Elasticsearch is a bug in Kibana that has been fixed in version 5.0. CORS isn't relevant for the communications between Kibana and Elasticsearch because the requests are browser -> server (kibana) -> server (elasticsearch) rather than browser -> server (elasticsearch).

Kibana itself does not allow cross origin requests by design, and there's no way to configure it to do so.

Elasticsearch does still have CORS support on the off-chance that you built some other browser-based app that needs to query ES directly, but it's not there for Kibana's sake.

[sebastiaopf]

Just a comment that might help others. I was getting this error a lot in my setup (kibana 4.5), even after removing the http.cors.enabled setting. Turns out that I was using apache as a reverse proxy to serve kibana, and after i removed it from the way and let the clients connect directly to kibana the problems seems to have stopped.
Didn't have the time to find out what apache was doing that caused the issue, but i hope this can point someone on the way to finding it.

[JathinSanghvi]

@sebastiaopf -- i have a similar problem.. but for me i need the apache reverse proxy.. i am using that for configuring ssl.. did you find a way to get this fix with apache reverse proxy?

[sebastiaopf]

@JathinSanghvi nope, I was using apache as a proxy just because i wanted to route everything through port 80. Ended up opening kibana port directly at least temporarily.
I'll try with nginx as a proxy one of these days and see if it changes anything.
If someone has a clue about this i'd like to know too.
BTW why was this marked as closed? Was it decided that the culprit is indeed apache and kibana is doing nothing wrong?

[epixa]

@sebastiaopf Sorry, the close was not in response to your earlier comment. Purely a coincidence. We had kept the issue open for so long only to help people find it so they could see the necessary configuration change, and headers are no longer being sent through to Elasticsearch by default in 5.0.

If you're still having an issue with apache and you think the root cause is Kibana, then it might be manifesting in the same way, but it's almost certainly a different bug. In that case, I'd urge you to open a new ticket.

[JathinSanghvi]

@sebastiaopf :: got a fix.. when i was looking at the elastic search logs i found this stack..

org.jboss.netty.handler.codec.frame.TooLongFrameException: HTTP header is larger than 8192 bytes. at org.jboss.netty.handler.codec.http.HttpMessageDecoder.readHeader(HttpMessageDecoder.java:624) at org.jboss.netty.handler.codec.http.HttpMessageDecoder.readHeaders(HttpMessageDecoder.java:531) at org.jboss.netty.handler.codec.http.HttpMessageDecoder.decode(HttpMessageDecoder.java:195) at org.jboss.netty.handler.codec.http.HttpMessageDecoder.decode(HttpMessageDecoder.java:102) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500)

And then i found this

When i added the below line in elasticsearch.yml. Kibana errors disappeared.

http.max_header_size: 16kb

[sumit-gupta-sgt]

I have still same issue will ES 2.4.2 and Kibana 4.6.3..Anybody facing same issue..Please let me know.

Thanks,
Sumit

[epixa]

@sumit-gupta-sgt Version 4.6 would still be impacted by the original issue. The solution is identified in this comment: #6719 (comment)

[deanso]

In case somebody else still having issues with this. I had the same issue and the mentioned http.cors.enabled setting is not there in elasticsearch.yml.
I'm using an apache reverse proxy with a connection to active directory for authentication in front of kibana 5.1.

In my case the solution was to disable the authentication headers in the forwarded request to kibana in the apache proxy config (location section):

RequestHeader unset Authorization

This will prevent the Authorization header from reaching the Kibana backend.

[sebastiaopf]

@JathinSanghvi thanks for the information. I'll try that on my setup. Some time after that problem i started using Grafana for user accessible dashboards, and limited Kibana to development. This might enable me to try and switch back to Kibana.

[paddymahadeva]

Hi,
Having the same issue in AWS Elasticsearch instance. Building a proxy for the same. I cannot change the elasticsearch.yml file as I have no control over it. Any help?