Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add 'enrich' operator for setting up enrich indices #1796

Open
craigtaverner opened this issue Nov 2, 2023 · 0 comments
Open

Add 'enrich' operator for setting up enrich indices #1796

craigtaverner opened this issue Nov 2, 2023 · 0 comments
Labels
enhancement Improves the status quo good first issue Small, contained changes that are good for newcomers help wanted We'd be happy about a community contribution :Track Management New operations, changes in the track format, track download changes and the like :Usability Makes Rally easier to use

Comments

@craigtaverner
Copy link

craigtaverner commented Nov 2, 2023

The recent work to benchmark ES|QL has benefitted greatly from the addition of the esql operator in #1791. However, the benchmarks that make use of the enrich command need enrich indices set up, and this is currently achieved using a complex set of multiple raw operations:

  • Create enrich source index and ingest data
  • Delete previous enrich policy
  • Create new enrich policy
  • Execute the enrich policy
  • Use the enrich policy within an ingestion pipeline

The three steps for delete/create/execute of the enrich policy can be combined into a single enrich policy operation. This operation should combine the following raw steps:

Delete

    {
      "name": "delete-enrich-nyc_vendors",
      "operation-type": "raw-request",
      "ignore": [404],
      "include-in-reporting": false,
      "method": "DELETE",
      "path": "/_enrich/policy/nyc_vendors"
    }

Create

    {
      "name": "create-enrich-nyc_vendors",
      "include-in-reporting": false,
      "operation-type": "raw-request",
      "method": "PUT",
      "path": "/_enrich/policy/nyc_vendors",
      "body": {
        "match": {
          "indices": "nyc_vendors",
          "match_field": "id",
          "enrich_fields": [ "name" ]
        }
      }
    }

Execute

    {
      "name": "execute-enrich-nyc_vendors",
      "operation-type": "raw-request",
      "method": "POST",
      "path": "/_enrich/policy/nyc_vendors/_execute"
    }

Combined

The above operations could be combined into one:

    {
      "name": "setup-enrich-nyc_vendors",
      "operation-type": "enrich_policy",
      "policy_name": "nyc_vendors",
      "delete": true,                              // default to true
      "enrich_type": "match",             // could be default to 'match'
      "indices": "nyc_vendors",          // could be default to value of 'policy_name'
      "match_field": "id",
      "enrich_fields": [ "name" ]
    }

Multiple policies

An interesting alternative might be to allow the definition of multiple policies with a more flexible syntax:

    {
      "name": "setup-enrich-policies",
      "operation-type": "enrich_policy",
      "delete": true,
      "policies": {
        "nyc_vendors": {
            "match": {
              "indices": "nyc_vendors",
              "match_field": "id",
              "enrich_fields": [ "name" ]
          }
        },
        "nyc_payment_types_fares": {
          "match": {
            "indices": "nyc_payment_types",
            "match_field": "type",
            "enrich_fields": [ "name", "fare" ]
          }
        }
      }
    }

One advantage of this syntax is the body of the enrich policy matches exactly the body of the create part of the enrich policy REST request, so we could just pass through that part of the json object.

@b-deam b-deam added enhancement Improves the status quo :Track Management New operations, changes in the track format, track download changes and the like good first issue Small, contained changes that are good for newcomers help wanted We'd be happy about a community contribution :Usability Makes Rally easier to use labels Nov 10, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Improves the status quo good first issue Small, contained changes that are good for newcomers help wanted We'd be happy about a community contribution :Track Management New operations, changes in the track format, track download changes and the like :Usability Makes Rally easier to use
Projects
None yet
Development

No branches or pull requests

2 participants