Reloadable secure settings for plugins

[albertzaharovits]

EDITED:
The reload-secure-store-action branch tracks the progress of making some of the plugins' secure settings reloadable.

Passwords and access keys of the clients used internally for remote calls are stored as secure settings inside the elasticsearch.keystore file. The contents of this file are retrievable only after decryption, using a password. Currently, the password is only available during node initialization. This effectively renders these settings immutable, as they are read only once, during initialization.

In the new proposed approach, plugins that are using internal clients (AWS, Azure, etc.) should not cache client instances but instead should require a client from a "Registry Service". There is one such service for each plugin. The client instance should be requested on each use (call to external system). Generally, it should be inexpensive for the "Registry Service" to service such demands. The "Registry Service", acting as a client cache, should allow for the clients' credentials to be changed. After the credentials change, subsequent client requests will return a new client constructed with the new credentials. Client instances that have been used before any credentials change will continue to be usable. The present implementation is of a best effort type. Currently, there is no way to wait for plugins to ditch the old settings and use only the new ones. In essence, this is because there is no strict rule of what reload means or what reload is complete means. It might be possible (high hanging fruit) for plugins to signal their completed reload.

The credentials change when the node rereads and decrypts the local keystore. For decryption it requires the password which is broadcasted by a TransportNodesAction action. The password should not be cached anywhere. When the TransportNodesAction action has returned, the keystore is no longer readable.

Too much noise already.
Plugins wishing to become "reloadable" should implement the ReloadablePlugin interface. They have to implement void reload(Settings settings) throws Exception. The settings parameter contains retrievable SecureSettings, but only if accessed under the reload call stack, i.e. don't pass the settings object to an async task and expect for SecureSettings to be further retrievable. Once the reload method has returned the keystore is sealed again.

Tasks:

• broadcast the password to decrypt settings
• make repository-s3-plugin client settings updatable (merged)
• make discovery-ec2-plugin client settings updatable (merged)
• make repository-azure-plugin client settings updatable (merged)
• make repository-gcs credentials file setting updatable (merged)
• factor out common code for the client cache/registry/dictionary (merged)

Secure settings are also used at these other places:

• SSL Configuration
• watcher notification integrations (hipchat, jira, pagerduty, slack)
• LDAP and AD Realm bind password

They might also benefit of this "reloadable" capability.

[elasticmachine]

Pinging @elastic/es-core-infra