After Volume Increase, large number of “java.nio.file.FileSystemException” and “TranslogException” Logs

[benlavender]

Here is the URL to my issues at ES discussion: https://discuss.elastic.co/t/indices-created-in-new-node-after-disk-extend/43794

Since we had a disk volume deplete it’s space the ES went down until we extended it, since then ES has been stopping every day and a large amount of the below logs have been created:

Caused by: java.nio.file.FileSystemException: /var/lib/elasticsearch/elasticsearch/nodes/0/indices
/devicename-2016.02.21/4/index: Too many open files

TranslogException[failed to create new translog file]; nested: FileAlreadyExistsException[/var/lib/elasticsearch/elasticsearch/nodes/0/indices/devicename-2016.03.21/1/translog/translog-17.tlog]
I also noted that since the increase it’s switch to /node1 from /node0.

I’m aware of a bug in V2 but we’re running 2.1.1 right now.

[s1monw]

I think this is fixed by #15788 any chance you can upgrade to 2.1.2 also can you provide the full stacktrace?

[benlavender]

I upgraded to 2.3.0 via the .DEB package, I still see the below type errors when starting ES:

Caused by: [syslog-2016.03.24][[syslog-2016.03.24][1]] EngineCreationFailureException[failed to create engine]; nested: NoSuchFileException[/var/lib/elasticsearch/elasticsearch/nodes/0/indices/syslog-2016.03.24/1/translog/translog-2.ckp];

Caused by: java.nio.file.FileSystemException: /var/lib/elasticsearch/elasticsearch/nodes/0/indices/syslog-2016.03.29/4/translog/translog-21.tlog: Too many open files

The instance will keep logging this until it peaks it's CPU %

[clintongormley]

Too many open files

You need to increase your file handle limit

[benlavender]

Please can someone provide the method of doing this? Thanks

[clintongormley]

@benlavender https://www.elastic.co/guide/en/elasticsearch/reference/2.3/setup-configuration.html#file-descriptors

[s1monw]

closing no feedback