[Breaking][data.indexPatterns] Remove expression-based scripted fields from Kibana.

[lukeelmers]

Summary

There are a few remaining traces of support for scripted fields using Lucene's expression language floating around Kibana. Since we have already deprecated expression-based scripted fields in 7.0, we should clean up the remaining traces in 8.0 and create a clear migration path for users.

A History Lesson

In 7.0, we removed support for expression-based scripted fields after Elasticsearch stopped supporting expression scripts in filters.

The changes made at the time essentially did the following:

• The UI no longer allows people to create or edit scripted fields using expression language
  • The only option available in the field editor language dropdown is painless
  • If you try to edit an existing expression scripted field, it forces you to change to painless
  • You can't run a "preview" of an expression scripted field as you will get errors back from ES
  • You can't apply a filter on an expression scripted field as you will get errors back from ES
• A deprecation notice is displayed in the field editor if the field is using any language other than painless.
  • It mentions that "support will be removed in the next major version of Kibana"

However, these changes did not entirely remove expression scripts:

• If you had an expression script field in 6.x, your index pattern stays the same in 7.0
• You can still use expression script fields outside of a filter context (e.g. they work in Discover)
• Kibana still has an /api/kibana/scripts/languages endpoint that always returns ['painless', 'expression'], while we have hardcoded painless as the only "supported" language in index patterns UI.
  • This is stuff that formerly lived in the legacy ui/scripting_languages directory.

Tasks

In 8.0 we should remove the remaining pieces supporting expression-based scripted fields, which should help simplify some code and will prevent confusion among users by providing a more consistent experience.

Currently there are only two ways you would know that you are using a deprecated expression scripted field:

1. If you had visualizations / dashboards relying on filters on those fields, they'd already be broken.
2. If you navigate to the field editor for that specific field, you'd see a warning.

As a result, it's possible we have users who still have expression script fields from 6.x and haven't noticed any changes in 7.x. We need to make sure they are aware that these will break starting in 8.0 before they upgrade.

There are a few things we can do to make this transition as smooth as possible for users.

1. Write a migration that throws an error (cc @elastic/kibana-platform)

We can't write a migration to update these fields because each script's source will need to be rewritten in painless. However, we can still write a migration that simply checks each index pattern and throws an error if it detects an expression scripted field. This will fail the migration, and require users to take action to address the problematic index patterns.

2. Add a check to the Upgrade Assistant (cc @elastic/es-ui)

This way users can be made aware if they have any index patterns using expression script fields before attempting an upgrade. It will also provide a better experience for Cloud users, since Cloud checks the assistant before allowing users to perform an upgrade. This item would ideally be done as early as possible so that folks can already be made aware so they have more time to make updates.

3. ...then remove what's left

• Remove the /api/kibana/scripts/languages route
• Remove the logic in index_pattern_management/public/scripting_languages that performs checks on available/supported languages
• Remove the deprecation notice that appears in the field editor UI
• Remove the "language" selection dropdown in the field editor UI
• Update index patterns mocks & fixtures to remove instances of lang: 'expression'

[elasticmachine]

Pinging @elastic/kibana-app-arch (Team:AppArch)

[alisonelizabeth]

Hi @lukeelmers! Would you mind including some steps on how we can test this?

[lukeelmers]

Sure @alisonelizabeth!

To manually test:

1. In 6.8.x create a scripted field on an index pattern with expression selected as the scripted field language.
2. Export index pattern saved object and import into any 7.x deployment.
3. The index pattern should still work but you should get a deprecation notice on the scripted field and you'll be unable to edit it would changing to painless.

To programmatically determine whether this is an issue for users, you'd need to retrieve all index patterns, filter down to scripted fields, and check for any scripted fields with lang: "expression".

Pseudocode for doing this with the index patterns service would look something like this:

const { indexPatterns } = data;
const ids = await indexPatterns.getIds();

const hasDeprecatedScriptField = async (id: string): boolean => {
  const scriptedFields = await indexPatterns.get(id).getScriptedFields();
  return scriptedFields.some(f => f.lang === 'expression');
};

// untested, but you get the idea
const results = await Promise.all(ids.map(async (id) => await hasDeprecatedScriptField(id)));

[kobelb]

@lukeelmers do we know how many users will be affected by this breaking change? If we don't have data to support this, a guess is better than nothing :)

[mattkime]

@kobelb We don't have numbers. In my nearly two years of supporting index patterns I've gotten zero questions about expression based scripted fields.

[lukeelmers]

I've gotten zero questions about expression based scripted fields.

++ there's no data on this, but anecdotally, I never once had this question come up either.

Considering that we've had the deprecation notice since the beginning of 7.x, and have disabled the ability to create new expression scripted fields or edit existing ones, this would only affect users who:

1. Had an expression script field in 6.x
2. Upgraded to 7.x and never had a need to touch that field
3. Are preparing to upgrade to 8.x (again, having not changed the field)

The users meeting the above criteria are the only ones who should be affected by this change. For them, it's admittedly annoying because they need to rewrite those scripts.

However, anyone else who had an expression field would have already been forced to convert to painless the first time they tried to edit it (assuming they did so via the UI and not by mucking around in the saved objects)

[elasticmachine]

Pinging @elastic/kibana-app-services (Team:AppServices)

[alisonelizabeth]

I'm going to remove the Elasticsearch UI team label. This deprecation should be registered by the plugin owner via the core deprecations service (#94845). All registered deprecations will be displayed in the Upgrade Assistant (to be implemented via #97159). Feel free to reach out to myself or the core team with any questions!

[mattkime]

Closing this as scripted fields are deprecated and will eventually be removed all together. It no longer makes sense to have a separate deprecation process for expression scripted fields.