Add HDFS searchable snapshot integration #66185
Conversation
jbaiera
added
>feature
:Distributed/Snapshot/Restore
|
Pinging @elastic/es-distributed (Team:Distributed) |
|
Currently debugging an intermittent SocketPermission exception but the code should be in a place where it's ok for review. |
|
@elasticmachine update branch |
|
Update: I've been repeating the tests that were failing intermittently with SocketPermissions through the afternoon. After pushing 6d86e22 they have not resurfaced. We should be good on that front now! |
There was a problem hiding this comment.
Looks good @jbaiera just one question I think from a quick read through this.
| @@ -50,7 +50,7 @@ | |||
| static { | |||
| // We can do FS ops with only a few elevated permissions: | |||
| SIMPLE_AUTH_PERMISSIONS = new Permission[]{ | |||
| new SocketPermission("*", "connect"), | |||
| new SocketPermission("*", "connect,resolve"), | |||
There was a problem hiding this comment.
Why do we need resolve now, it seems the only new thing we do is a .seek call?
There was a problem hiding this comment.
likely not. I added this to try to appease the permission errors. Given the problem was something else, it's unlikely that we'll need this.
There was a problem hiding this comment.
Other than the resolve permission question that @original-brownbear has, LGTM.
Adds a bounded read implementation on the HDFS blob store as well as integration tests to the searchable snapshot project that ensures functionality on both kerberos and simple authentication HDFS.
* elastic/master: (33 commits) Add searchable snapshot cache folder to NodeEnvironment (elastic#66297) [DOCS] Add dynamic runtime fields to docs (elastic#66194) Add HDFS searchable snapshot integration (elastic#66185) Support canceling cross-clusters search requests (elastic#66206) Mute testCacheSurviveRestart (elastic#66289) Fix cat tasks api params in spec and handler (elastic#66272) Snapshot of a searchable snapshot should be empty (elastic#66162) [ML] DFA _explain API should not fail when none field is included (elastic#66281) Add action to decommission legacy monitoring cluster alerts (elastic#64373) move rollup_index param out of RollupActionConfig (elastic#66139) Improve FieldFetcher retrieval of fields (elastic#66160) Remove unsed fields in `RestAnalyzeAction` (elastic#66215) Simplify searchable snapshot CacheKey (elastic#66263) Autoscaling remove feature flags (elastic#65973) Improve searchable snapshot mount time (elastic#66198) [ML] Report cause when datafeed extraction encounters error (elastic#66167) Remove suggest reference in some API specs (elastic#66180) Fix warning when installing a plugin for different ESversion (elastic#66146) [ML] make `xpack.ml.max_ml_node_size` and `xpack.ml.use_auto_machine_memory_percent` dynamically settable (elastic#66132) [DOCS] Add `require_alias` to Bulk API (elastic#66259) ...
Adds a bounded read implementation on the HDFS blob store as well as integration tests to the searchable snapshot project that ensures functionality on both kerberos and simple authentication HDFS.