[Transform] add transform upgrade endpoint #77566
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
hendrikmuhs
force-pushed
the
transform-upgrade
branch
5 times, most recently
from
8e97475
to
a93c2de
Compare
hendrikmuhs
force-pushed
the
transform-upgrade
branch
5 times, most recently
from
fd0aeac
to
ed42c75
Compare
hendrikmuhs
force-pushed
the
transform-upgrade
branch
3 times, most recently
from
2ec4e19
to
eea9cc0
Compare
hendrikmuhs
force-pushed
the
transform-upgrade
branch
3 times, most recently
from
b8e5005
to
bf07e9f
Compare
|
Pinging @elastic/ml-core (Team:ML) |
hendrikmuhs
force-pushed
the
transform-upgrade
branch
from
e3698f7
to
de98187
Compare
...el/src/test/java/org/elasticsearch/client/transform/hlrc/UpgradeTransformsResponseTests.java
Show resolved
Hide resolved
rest-api-spec/src/main/resources/rest-api-spec/api/transform.upgrade_transforms.json
Outdated
Show resolved
Hide resolved
| "Test upgrade transform": | ||
| - do: | ||
| transform.upgrade_transforms: | ||
| dry_run: false |
There was a problem hiding this comment.
Could you also add a test case with dry_run==true?
There was a problem hiding this comment.
👍 Note, I have rolling upgrade test in the works (this one will be more meaningful), but I can't check that in as part of this PR, as I 1st need the backport.
| DeleteIndexRequest deleteRequest = new DeleteIndexRequest( | ||
| TransformInternalIndexConstants.INDEX_NAME_PATTERN, | ||
| TransformInternalIndexConstants.INDEX_NAME_PATTERN_DEPRECATED, | ||
| "-" + TransformInternalIndexConstants.LATEST_INDEX_VERSIONED_NAME |
There was a problem hiding this comment.
Does the minus ("-") sign mean to exclude an index from being deleted?
There was a problem hiding this comment.
yes, with - one can exclude, see Multi-target syntax
...ain/java/org/elasticsearch/xpack/transform/persistence/IndexBasedTransformConfigManager.java
Outdated
Show resolved
Hide resolved
...ain/java/org/elasticsearch/xpack/transform/persistence/IndexBasedTransformConfigManager.java
Outdated
Show resolved
Hide resolved
...ain/java/org/elasticsearch/xpack/transform/persistence/IndexBasedTransformConfigManager.java
Outdated
Show resolved
Hide resolved
.../test/java/org/elasticsearch/xpack/transform/persistence/InMemoryTransformConfigManager.java
Outdated
Show resolved
Hide resolved
.../test/java/org/elasticsearch/xpack/transform/persistence/InMemoryTransformConfigManager.java
Outdated
Show resolved
Hide resolved
hendrikmuhs
force-pushed
the
transform-upgrade
branch
from
844ffd9
to
10c4a27
Compare
added 9 commits
October 12, 2021 09:21
hendrikmuhs
force-pushed
the
transform-upgrade
branch
from
8c93ab8
to
21675ce
Compare
benwtrent
reviewed
Oct 12, 2021
There was a problem hiding this comment.
My two major concerns:
- How do we recover from the old docs not being deleted after an upgrade?
- How does this behave in multi-node. We recursively upgrade and that could be on a network thread. I don't think we lock anything, and pushing/popping the listeners from that threadpool is probably just fine. Just paranoid.
Overall, this is some good stuff
...ore/src/main/java/org/elasticsearch/xpack/core/transform/action/UpgradeTransformsAction.java
Outdated
Show resolved
Hide resolved
...ore/src/main/java/org/elasticsearch/xpack/core/transform/action/UpgradeTransformsAction.java
Outdated
Show resolved
Hide resolved
...src/main/java/org/elasticsearch/xpack/transform/action/TransportUpgradeTransformsAction.java
Outdated
Show resolved
Hide resolved
hendrikmuhs
commented
Oct 12, 2021
...src/main/java/org/elasticsearch/xpack/transform/action/TransportUpgradeTransformsAction.java
Outdated
Show resolved
Hide resolved
…transform/action/UpgradeTransformsAction.java Co-authored-by: Benjamin Trent <ben.w.trent@gmail.com>
…rch into transform-upgrade
benwtrent
approved these changes
Oct 12, 2021
|
run elasticsearch-ci/part-2 |
hendrikmuhs
added
the
auto-backport
💔 Backport failed
You can use sqren/backport to manually backport by running |
hendrikmuhs
pushed a commit
to hendrikmuhs/elasticsearch
that referenced
this pull request
Oct 13, 2021
Add an _upgrade endpoint to bulk upgrade transforms. _upgrade rewrites all transforms and its artifacts into the latest format to the latest storage(index). If all transforms are upgraded old indices and outdated documents get deleted. Using the dry_run option it is possible to check if upgrades are necessary without applying changes.
elasticsearchmachine
pushed a commit
that referenced
this pull request
Oct 14, 2021
* [Transform] add transform upgrade endpoint (#77566) Add an _upgrade endpoint to bulk upgrade transforms. _upgrade rewrites all transforms and its artifacts into the latest format to the latest storage(index). If all transforms are upgraded old indices and outdated documents get deleted. Using the dry_run option it is possible to check if upgrades are necessary without applying changes. * fix merge conflicts * 7.x requires a different license check
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add an _upgrade endpoint to bulk upgrade transforms.
_upgraderewrites all transforms and its artifacts into the latest format to the latest storage(index). If all transforms are upgraded old indices and outdated documents get deleted.Todo: