need kibana internal user to have read/write access to .kibana* ILM resources

[pmuellr]

Currently, the internal kibana user does not have access to create or get ILM resources at all, as near as I can tell. For the "Make it Action" project, we're adding a new Kibana-specific index (with the default kibana prefix name .kibana*) and associated ILM policies, templates, etc. Currently creating templates and other resources works fine, but when trying to do a GET or PUT on an ILM policy with the .kibana* prefix, we're seeing 403 errors. Not specific to that particular prefix, but if we end up allowing ILM GET/PUT on just indices with that prefix, WORKSFORME.

Via off-line chat, @legrego believes the "answer" is to add more bits here:

elasticsearch/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/authz/store/ReservedRolesStore.java

Line 113 in 212e3e4

"monitor", "manage_index_templates", MonitoringBulkAction.NAME, "manage_saml", "manage_token", "manage_oidc",

I'm unfamiliar with these bits of the Elastic Stack, so will defer to Larry, but am happy to provide more context if required.

[legrego]

@pmuellr, does this set of privileges sound correct to you based on what you're trying to accomplish?

We need the internal kibana user (kibana_system role) to do the following:

1. Read ILM Policies (cluster:admin/ilm/get)
2. Create/Edit ILM Policies (cluster:/admin/ilm/put)
3. Add/Remove/Manage ILM Policies for indices matching the .kibana* pattern (indices:/admin/ilm/*)

And we do not need privileges to do the following:

1. Delete ILM Policies (cluster:admin/ilm/delete)
2. Start/Stop ILM for the cluster as a whole (cluster:admin/ilm/start, cluster:admin/ilm/stop)
3. Read the ILM Status (cluster:admin/ilm/operation_mode/get)

The kibana_system user will be able to read and write any/all ILM policies (not just the policies matching a specific pattern). The kibana_system user will then be able to add/remove any ILM policy to any index matching the .kibana* pattern.

[elasticmachine]

Pinging @elastic/es-security

[albertzaharovits]

Since we currently don't offer permissions for just a namespace of ILM policies, is it possible to hardcode those policies? It's not entirely clear how we can leverage this, but I thought I might ask.
Otherwise, Larry's right. We need to grant privileges to create and edit ILM policies in addition to the index manage.

[legrego]

Since we currently don't offer permissions for just a namespace of ILM policies, is it possible to hardcode those policies? It's not entirely clear how we can leverage this, but I thought I might ask.

This is an interesting idea -- are you thinking about a "reserved" policy that would live inside ES, similar to how we have reserved users and roles defined? If ES defines a .kibana-alerting ILM policy, then all the kibana_system user would need is the ability to associate it to the kibana alerting index. kibana_system already has all against the .kibana* pattern of indices, so that part could work.

I also have 0 experience with IM development, so I don't know what the LoE is to introduce this concept to ES and the ILM UI itself.

If we take this approach, are reserved ILM policies something that other stack components/solutions would want to take advantage of? Alerting probably won't be the last offering to want to leverage ILM "automatically", so maybe this is an idea worth exploring further. We likely won't want all of these service accounts to have the ability to create/edit these policies, especially since they aren't namespaced on their own.

[pmuellr]

@legrego: does this set of privileges sound correct to you based on what you're trying to accomplish?

Ya, sounds right. Currently, we check if an ILM policy with the calculated name exists, and if it doesn't, we create it. We'll also then reference this in the index settings (I think that's where the association is made). We'll provide some default policy, but otherwise won't be updating it, or deleting it.

[legrego]

@pmuellr before I open the PR, can you test this locally to make sure it suites your needs?

1) Create custom_kibana_system role:

PUT _security/role/custom_kibana_system
{
  "cluster" : [
      "monitor",
      "manage_index_templates",
      "cluster:admin/xpack/monitoring/bulk",
      "manage_saml",
      "manage_token",
      "manage_oidc",
      "cluster:admin/xpack/security/privilege/builtin/get",
      "delegate_pki",
      "cluster:admin/ilm/get",
      "cluster:/admin/ilm/put"
    ],
    "global" : {
      "application" : {
        "manage" : {
          "applications" : [
            "kibana-*"
          ]
        }
      }
    },
    "indices" : [
      {
        "names" : [
          ".kibana*",
          ".reporting-*"
        ],
        "privileges" : [
          "all"
        ],
        "allow_restricted_indices" : false
      },
      {
        "names" : [
          ".monitoring-*"
        ],
        "privileges" : [
          "read",
          "read_cross_cluster"
        ],
        "allow_restricted_indices" : false
      },
      {
        "names" : [
          ".management-beats"
        ],
        "privileges" : [
          "create_index",
          "read",
          "write"
        ],
        "allow_restricted_indices" : false
      },
      {
        "names" : [
          ".code-*",
          ".code_internal-*"
        ],
        "privileges" : [
          "all"
        ],
        "allow_restricted_indices" : false
      },
      {
        "names" : [
          ".apm-agent-configuration"
        ],
        "privileges" : [
          "all"
        ],
        "allow_restricted_indices" : false
      }
    ],
    "applications" : [ ],
    "run_as" : [ ],
    "metadata" : {
    },
    "transient_metadata" : {
      "enabled" : true
    }
}

2) Create custom_kibana_system user with this role

PUT _security/user/custom_kibana_system
{
    "roles" : [
      "custom_kibana_system"
    ],
    "full_name" : null,
    "email" : null,
    "metadata" : {
    },
    "enabled" : true,
    "password": "changeme"
}

3) Edit your kibana.yml to use this account:

elasticsearch.username: custom_kibana_system
elasticsearch.password: changeme

4) Do your IL stuff with this internal system user

[pmuellr]

Getting closer. I make two ILM-related calls, currently. A GET policy, and a PUT policy if the GET returned not found. Without the role/user above (just normal kibana run), I'd get a 403 on both. With the role/user above, now getting a 404 on the GET (yay, it works now!), but still get a 403 on the PUT.

Here are the requests / responses:

GET _ilm/policy/.kibana-event-log-policy

{
  "message": "[resource_not_found_exception] Lifecycle policy not found: .kibana-event-log-policy",
  "statusCode": 404
}


PUT _ilm/policy/.kibana-event-log-policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_size": "5GB",
            "max_age": "30d"
          }
        }
      }
    }
  }
}

{
  "message": "[security_exception] action [cluster:admin/ilm/put] is unauthorized for user [custom_kibana_system]",
  "statusCode": 403
}

[pmuellr]

For completeness, I'm making these calls with:

clusterClient.callAsInternalUser(operation, body);

where clusterClient is obtained from CoreSetup::elasticsearch.adminClient$ (aliased as this.clusterClient$ below) via:

const clusterClient = await this.clusterClient$.pipe(first()).toPromise();

[legrego]

@pmuellr thanks for the quick feedback. Looks like I had a typo in my privilege set. This should do the trick:

PUT _security/role/custom_kibana_system
{
  "cluster" : [
      "monitor",
      "manage_index_templates",
      "cluster:admin/xpack/monitoring/bulk",
      "manage_saml",
      "manage_token",
      "manage_oidc",
      "cluster:admin/xpack/security/privilege/builtin/get",
      "delegate_pki",
      "cluster:admin/ilm/get",
      "cluster:admin/ilm/put"
    ],
    "global" : {
      "application" : {
        "manage" : {
          "applications" : [
            "kibana-*"
          ]
        }
      }
    },
    "indices" : [
      {
        "names" : [
          ".kibana*",
          ".reporting-*"
        ],
        "privileges" : [
          "all"
        ],
        "allow_restricted_indices" : false
      },
      {
        "names" : [
          ".monitoring-*"
        ],
        "privileges" : [
          "read",
          "read_cross_cluster"
        ],
        "allow_restricted_indices" : false
      },
      {
        "names" : [
          ".management-beats"
        ],
        "privileges" : [
          "create_index",
          "read",
          "write"
        ],
        "allow_restricted_indices" : false
      },
      {
        "names" : [
          ".code-*",
          ".code_internal-*"
        ],
        "privileges" : [
          "all"
        ],
        "allow_restricted_indices" : false
      },
      {
        "names" : [
          ".apm-agent-configuration"
        ],
        "privileges" : [
          "all"
        ],
        "allow_restricted_indices" : false
      }
    ],
    "applications" : [ ],
    "run_as" : [ ],
    "metadata" : {
    },
    "transient_metadata" : {
      "enabled" : true
    }
}

[pmuellr]

Ya, that worked!

GET _ilm/policy/.kibana-event-log-policy
{
  "message": "[resource_not_found_exception] Lifecycle policy not found: .kibana-event-log-policy",
  "statusCode": 404
}


PUT _ilm/policy/.kibana-event-log-policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_size": "5GB",
            "max_age": "30d"
          }
        }
      }
    }
  }
}

{"acknowledged":true}

[legrego]

Opened #49451

[pmuellr]

Sorry, somehow completely missed this comment: #46894 (comment)

Since we currently don't offer permissions for just a namespace of ILM policies, is it possible to hardcode those policies?

We are currently providing a very basic ILM policy, with the assumption a customer may want to customize it further. So "hard-coding" the policy values won't work.

[albertzaharovits]

@pmuellr Even if we hard-coded an initial policy , we could document a way for the administrator to change it. For these uncommon changes we could require creating roles, etc...

Under the present model, Larry's solution is the only option, but the new system role has potential for user data loss, if it can add the delete action to any existing ILM policy. I'll bring this issue up to the team and report back.

[pmuellr]

Perhaps there's a better way to deal with this, compared to what I'm currently doing, this seemed like the most straight-forward approach. Open to other ideas.

I'd imagine we might see more usage like this - Kibana subsystem wants to create some kind of new "log", that would be appropriate to apply ILM to, so set it up with a default ILM policy, and allow the customer to customize it. In fact, I've chatted with folks about a new log they might be adding in the future, that would probably have some of the same ILM type of characteristics.

Seems like ultimately we want to have some kind of RBAC where we could give the kibana system user r/w access to just ILM policies with names that start with ${kibana-index-name}. Kinda thing.