Deprecate and remove freeze and unfreeze REST endpoints

[zuketo]

With the addition of a new Frozen Tier in Elasticsearch, and recent improvements in heap utilization, “frozen indices” (_freeze / _unfreeze endpoints) provide little end-user benefit.

Elasticsearch should deprecate, and then remove, _freeze and _unfreeze endpoints. This work should occur in a way that minimally impacts users, with a large amount of notice, and escape hatches along the way.

7.x:

• Deprecate freeze index endpoint (Deprecate freeze index API #72618)
• Deprecate the ILM freeze action (Adding deprecation logging and deprecation info API for ILM freeze ation #77969)
  - [ ] Warn on startup if frozen indices exist
• Add deprecation issue to deprecation info api that checks whether frozen indices exist in a cluster. (Add frozen index deprecation check. #78915)
• Warn when searching frozen indices? (Warn when searching a frozen index. #78184)
• Warn on use of ignore_throttled parameter? (Deprecate ignore_throttled parameter #77479)

8.0:

• Remove _freeze endpoint (no new frozen indices can be created) (Remove endpoint for freezing indices #78918)
• Make the ILM freeze action a no-op in 8.0 (Make the ILM freeze action a no-op #77158)
• Continue to support the _unfreeze endpoint throughout 8.x (give users an escape hatch for unfreezing existing frozen indices in their cluster)

9.0:

• Remove the _unfreeze endpoint in 9.0
• Automatically _unfreeze any frozen indices upgrading to 9.0
• Continue to retain the lazy engine internally for shared_cache searchable, for the Frozen Tier
• Add migration assistant support for inspecting for frozen indices and freeze actions in ILM policies

CC @jasontedor @ywelsch @dakrone

[elasticmachine]

Pinging @elastic/es-core-features (Team:Core/Features)

[elasticmachine]

Pinging @elastic/es-core-features (Team:Core/Features)

[jakelandis]

We met today to discuss how this removal intersects with REST API compatibility. In summary there is not really any way to apply compatibility here. The closest analog is searchable shapshots, but that has significantly different requirements and setup. The plan is to use REST API compatibility to help provide more meaningful errors. For example, rather then just a 404 returned, REST API compatibility can respond with some nice error message stating that freeze is no longer supported. This applies to both the freeze API's and the add/update ILM policies if they include a freeze action.

[cjcenizal]

@danhermann @jakelandis @sajjadwahmed Do we have telemetry on how many users depend on the ILM freeze action in their ILM policies? If this is a significant amount, then I'm concerned that users who have invested heavily in using the freeze action will see its removal as a major obstacle to upgrading to 8.0.

If this is the case then they'll need clear and detailed guidance on how to decide whether the heap memory usage improvements are sufficient for their needs or if they need to use the frozen tier and, for the latter, how to migrate from the freeze action to the frozen tier.

Additionally, I think we need to increase the level of the deprecation message (#77969) to critical when users have ILM policies with this action defined. We want users to address this before upgrading, otherwise they're at risk for their data management system to stop behaving as expected.

[danhermann]

@cjcenizal, we've already made the ILM freeze action a no-op in 8.0 (#77158), so I don't think it's necessary to flag it as a critical deprecation issue since their existing ILM policies will still function.

[cjcenizal]

@danhermann Thanks, I understand the policies will technically function, but a user's expectations will no longer be met. Imagine this scenario:

• User has invested heavily in using freeze actions in ILM to manage their data and is used to seeing frozen indices be created by their policies.
• User upgrades to 8.0.
• Surprise! Policies stop behaving as expected -- frozen indices stop being created.
• User is suddenly under intense time pressure. They assume they did something wrong. They scramble to figure out why this is happening and what to do about it.
• Eventually the user discovers this is a deliberate breaking change in ES. How frustrating!
• They try to find some docs to guide them through the process of using the replacement to frozen indices. Can't find any. At this point, they're really stressed and feeling set up to fail by our products.

This is the user experience I'm concerned about. If we're making a breaking change to how our products behave, I think we need to really make this clear to the user. Upgrade Assistant makes critical issues prominent and warning issues subdued, and I worry that if this issue is warning it will escape the user's attention. Of course, if telemetry indicates nobody is using the freeze action any more, it's not really a problem.

[dakrone]

• Surprise! Policies stop behaving as expected -- frozen indices stop being created.
• User is suddenly under intense time pressure. They assume they did something wrong. They scramble to figure out why this is happening and what to do about it.

@cjcenizal can you explain how you get to the intense time pressure here? I don't follow your line of reasoning for this.

And also this:

They try to find some docs to guide them through the process of using the replacement to frozen indices. Can't find any. At this point, they're really stressed and feeling set up to fail by our products.

Our docs are pretty clear that frozen indices are being removed because they are no longer necessary. I'm not sure how this would set them up for stress or failure?

[cjcenizal]

can you explain how you get to the intense time pressure here

@dakrone Sure. Whenever I make a mistake that's public and affects others, I feel intense pressure to fix it. Even something as simple as force-pushing to a shared branch and causing a code review to be lost causes me anxiety. I want to fix it as soon as possible. If I were responsible for upgrading a production cluster and/or managing the data lifecycle of this cluster, and I messed that up, then I'd feel pretty bad. Folks who depend on me wouldn't be happy, maybe even my boss would be displeased, and I'd feel pressure to fix things so I can feel good about the job I'm doing. This was my thought process when trying to build empathy for these folks.

Our docs are pretty clear that frozen indices are being removed because they are no longer necessary. I'm not sure how this would set them up for stress or failure?

Can you link me to these docs? The ones I found weren't very helpful.

[VimCommando]

This is because frozen indices provide no benefit given improvements in heap memory utilization.

From: https://www.elastic.co/guide/en/elasticsearch/reference/7.16/migrating-7.16.html#breaking_716_ilm_changes

I take issue with this guidance because _freeze does more than just reduce heap usage. Other benefits include:

1. Searches count against a separate search_throttled thread pool, limiting the number of active queries
2. Searches do not include frozen indices by default, limiting the number of shards in a query

So this is more than just segment memory, it also helps control heap used to satisfy queries.

Not to mention the actual heap usage reduction is not specified in the docs at all. While overall Elasticsearch is more memory efficient, is that true for all data structures and use-cases? Is the heap reduction the same if you have logging data vs. geographic data?

Our docs are pretty clear that frozen indices are being removed because they are no longer necessary. I'm not sure how this would set them up for stress or failure?

Lets not forget about users going from 6.8 => 7.16 => 8.0. Even if we've deprecated a feature for multiple minor versions, this will be the first time many users have ever seen this change and it will require a significant re-evaluation of the data lifecycle.

Using data_frozen instead of data with a cold node attribute is not trivial; this required task isn't even mentioned:

https://www.elastic.co/guide/en/elasticsearch/reference/7.16/migrate-index-allocation-filters.html

If the expectation is the new data_frozen will replace this, that level of searchable snapshots requires and Enterprise license, where _freeze is available even under Basic. So maybe the "replacement" isn't even an option.

I strongly second this being a critical deprecation.

[dakrone]

Of course, if telemetry indicates nobody is using the freeze action any more, it's not really a problem.

I checked the telemetry for this, and 0.5% of clusters that use ILM use the freeze action.

a mistake that's public and affects others, I feel intense pressure to fix it

Okay, I think this treats the deprecation and noop behavior as a bug, which I'm not sure I agree that it is a bug?

Not to mention the actual heap usage reduction is not specified in the docs at all.

It is; in the documentation for both the freeze and unfreeze endpoint (https://www.elastic.co/guide/en/elasticsearch/reference/7.15/frozen-indices.html) the deprecation notice links to https://www.elastic.co/blog/significantly-decrease-your-elasticsearch-heap-memory-usage which shows the actual heap reduction.

Using data_frozen instead of data with a cold node attribute is not trivial; this required task isn't even mentioned:

I think there may be confusion here, frozen indices have nothing to do with the data_frozen tier at all. There is no requirement for use of that node role with regard to frozen indices.

If the expectation is the new data_frozen will replace this, that level of searchable snapshots requires and Enterprise license, where _freeze is available even under Basic.

That is not the expectation, the expectation is that frozen indices were a workaround that allowed a user to have higher density of data at the cost of throttling searches and potentially much slower response times. The throttling of searches was less of a feature and more of a way for frozen indices not to blow up the heap due to being able to load their search components serially rather than all at once (and potentially using too much memory). Using them as a visibility differentiator was never their primary goal, that was there because it had to be, not because we wanted it to be.

[cjcenizal]

0.5% of clusters that use ILM use the freeze action.

Thanks for checking this. I think that's low enough to render my concerns moot!

[VimCommando]

I think there may be confusion here, frozen indices have nothing to do with the data_frozen tier at all. There is no requirement for use of that node role with regard to frozen indices.

Yes, there definitely is confusion here, but it is largely due to the name collision. I've had to explain the difference between "frozen indices" and "frozen data tier" more than once. While the approach in significantly different, they do overlap in solving a similar problem: allowing a data node to search across more stored data in the same heap space, at the tradeoff of speed. ⚖️

The Elasticsearch 7.7 blog post is a great one, I've referenced it before. Although, I think we can do better in our deprecation notice than just linking to it. I'll give that some thought and make a docs update suggestion 🤔

[droberts195]

We have noticed a downside to warning about searching frozen indices on every search, i.e. the part that was done in #78184.

Because searching lots of frozen indices is so catastrophically slow most of the out-of-the-box ML jobs that are installed by the security and observability solutions contain ignore_throttled in their datafeed configs. This means that if a user happens to have a frozen index that matches the index pattern then we don't saturate their frozen index threadpool. The majority of users won't have any frozen indices, and in the past having ignore_throttled in the prebuilt ML jobs shipped by security and observability was a no-op for these users. But now all users of security and observability are going to see deprecation warnings relating to ignore_throttled, even the ones who don't have frozen indices.

Would it be possible to reconsider issuing a deprecation warning for every search that specifies ignore_throttled? For example, one of the following:

1. Never do it, and warn that frozen indices exist through some other mechanism
2. Only issue a deprecation warning if ignore_throttled is specified on a search and frozen indices actually exist in the cluster
3. Only deprecate ignore_throttled from the time when frozen indices are removed - it becomes a no-op at that time and is removed 18 months after frozen indices
4. Offer a way to avoid the deprecation warning, for example a special header that can be sent with the request to say that a deprecation warning should not be issued, which can then be sent with internal searches that use ignore_throttled "just in case"

[dakrone]

@droberts195 we do have throttling/caching of deprecation messages, based on the key of the message and the request opaque-id. This would only emit the message if it weren't already in the in-memory cache (however, there is a size to the cache, so it can end up evicted). Would that lower the amount of deprecation warning messages enough for this case?

[droberts195]

Would that lower the amount of deprecation warning messages enough for this case?

That depends on our policy for deprecation messages generated by Elastic code. Do we strive for Elastic's own code to not generate deprecation messages at all? Or are we happy for a user to start trialling the Elastic Stack and see a deprecation warning that's unrelated to anything they explicitly created themselves?

This same question applies generally - it's not specific to ignore_throttled. I'll raise it elsewhere.

One thing that's interesting is that Elasticsearch's own policy is to treat unexpected deprecation warnings as errors in integration tests. So it's not considered acceptable for Elasticsearch to generate deprecation warnings from its own internal operations. But then it seems inconsistent if it is acceptable for the overall Elastic codebase to generate deprecation warnings from its own internal operations. An end user of the Elastic Stack doesn't want to worry about the distinction between code that different sub-teams within the Elastic dev organisation write.

[tomcallahan]

We want to avoid throwing user-visible deprecations for Elastic code whenever possible. It's just not a good UX. Will follow up out of this issue

[asazallesmilner]

Just wanted to comment that after upgrading to 7.16.1 searches come up with deprecation notice even though as far as I know we have no code that uses frozen indices. Trying to find a way to disable that notice before update goes live to my community after updating to 7.16.1. ...

[Luegg]

I've just raised #81939 which covers the deprecation of frozen indices in SQL. Would it make sense to add the deprecation and removal in SQL to the schedule in the OP?

[jamshid]

It's frustrating and confusing to see deprecation warnings pop up about things the client is not even doing/using, and it can't be turned off, all when using an official elasticsearch sdk (supposed to be forward compatible) to talk to an official elasticsearch server.

Seeing this warning when using the ES 6 Java sdk to talk to ES 7. Seems that should be supported so that an application can e.g. support rolling reboots from ES 6 to 7?
The client should always be updated last, once all of the nodes in the cluster have been upgraded to the new major version
Anyway this warning started after upgrading server from 7.10.2 to 7.16.1. My ask is:
Can this deprecation warning be changed to ignore the parameter when the server detects the request is from an ES 6.8.x SDK, since it seems to be an SDK default parameter?

2021-12-23T02:59:21,021 WARN [qtp1400142922-82] RestClient: request [POST http://172.21.0.11:9200/myndex/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=true&ignore_throttled=true&search_type=query_then_fetch&batched_reduce_size=512&ccs_minimize_roundtrips=true] returned 1 warnings: [299 Elasticsearch-7.16.1-5b38441b16b1ebb16a27c107a4c3865776e20c53 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]

I guess elasticsearch 6.8.x is EOL in a couple of months so time to upgrade to SDK 7.x but that leaves existing 6.8 users unsupported, who are in process of upgrading. Maybe it can be disabled with a code change in the application, will see if that parameter can be explicitly set false.

[joschi]

We've upgraded to Elasticsearch 7.16.x recently and also saw the deprecation warning about the ignore_throttled index option.

The annoying thing is that the deprecation warning is triggered by default when using the Java High Level REST client and the default IndicesOptions for SearchRequest and MultiSearchRequest. This is really frustrating because the only option we have (except for upgrading to ES 8.x and the new Java client), is explicitly setting indices options for every single SearchRequest or MultiSearchRequest in our complete code base.

elasticsearch/server/src/main/java/org/elasticsearch/action/search/SearchRequest.java

Line 102 in 4e6e4ea

public static final IndicesOptions DEFAULT_INDICES_OPTIONS = IndicesOptions.strictExpandOpenAndForbidClosedIgnoreThrottled();

elasticsearch/server/src/main/java/org/elasticsearch/action/support/IndicesOptions.java

Lines 155 to 158 in 4e6e4ea

	public static final IndicesOptions STRICT_EXPAND_OPEN_FORBID_CLOSED_IGNORE_THROTTLED = new IndicesOptions(
	EnumSet.of(Option.ALLOW_NO_INDICES, Option.FORBID_CLOSED_INDICES, Option.IGNORE_THROTTLED),
	EnumSet.of(WildcardStates.OPEN)
	);

elasticsearch/server/src/main/java/org/elasticsearch/action/support/IndicesOptions.java

Lines 397 to 399 in 4e6e4ea

	if (request.hasParam("ignore_throttled")) {
	DEPRECATION_LOGGER.critical(DeprecationCategory.API, "ignore_throttled_param", IGNORE_THROTTLED_DEPRECATION_MESSAGE);
	}

I've written up some details
https://discuss.elastic.co/t/default-indicesoptions-in-searchrequest-and-disable-specific-deprecation-warnings/299242 and a colleague has opened #84827 for addressing the sub-optimal (?) default values.