-
Notifications
You must be signed in to change notification settings - Fork 24.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Enable deprecation log indexing by default #78319
Enable deprecation log indexing by default #78319
Conversation
Pinging @elastic/es-core-infra (Team:Core/Infra) |
.../src/main/java/org/elasticsearch/xpack/deprecation/logging/DeprecationIndexingComponent.java
Outdated
Show resolved
Hide resolved
…n_indexing_by_default
…n_indexing_by_default
…n_indexing_by_default
…n_indexing_by_default
…n_indexing_by_default
…n_indexing_by_default
@@ -234,6 +234,17 @@ The user ID is included in the `X-Opaque-ID` field in deprecation JSON logs. | |||
--------------------------- | |||
// NOTCONSOLE | |||
|
|||
Deprecation logs are indexed into `.logs-deprecation.elasticsearch-default` data stream by default. | |||
You can disable deprecation log indexing by changing a |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You can disable deprecation log indexing by changing a | |
You can disable deprecation log indexing by changing the |
|
||
==== Deprecation logs throttling | ||
Deprecation logs are deduplicated based on a deprecated feature key | ||
and x-opaque-id so that if a feature is repeatedly used, it will not overload the deprecation logs. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we link x-opaque-id
to a description somewhere?
@@ -782,7 +782,7 @@ protected static void wipeAllIndices() throws IOException { | |||
boolean includeHidden = minimumNodeVersion().onOrAfter(Version.V_7_7_0); | |||
try { | |||
//remove all indices except ilm history which can pop up after deleting all data streams but shouldn't interfere | |||
final Request deleteRequest = new Request("DELETE", "*,-.ds-ilm-history-*"); | |||
final Request deleteRequest = new Request("DELETE", "*,-.ds-ilm-history-*,-.ds-.logs-deprecation.elasticsearch-default-*"); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Did you ask the Data Management team about this?
|
||
ObjectMapper mapper = new ObjectMapper(); | ||
final JsonNode jsonNode = mapper.readTree(response.getEntity().getContent()); | ||
logger.warn(documents); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think we want this linen now.
@@ -121,6 +125,7 @@ | |||
/** | |||
* Base class of ML integration tests that use a native autodetect process | |||
*/ | |||
//@ESIntegTestCase.ClusterScope(scope = TEST, numClientNodes = 0, maxNumDataNodes = 1) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Leftover line?
).actionGet(); | ||
assertAcked(response); | ||
} | ||
|
||
@Override | ||
protected void beforeIndexDeletion() throws Exception { | ||
super.beforeIndexDeletion(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this is problematic, because if we want to remove indices on cleanup we have to remember to disable deprecation log indexing. Otherwise it might create a deprecation and recreate datastream and its indices.
closing in favour of #78991 |
Deprecation logs indexing is enabled by default. As before XOpaqueId is used for logs throttling, but it can be turned off and be ignored which can be useful for smaller clusters.
closes #76292
closes #77936