-
Notifications
You must be signed in to change notification settings - Fork 24.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[TSDB] Enablement of TSDB fails for package with fields having flattened type #94113
Comments
Pinging @elastic/es-analytics-geo (Team:Analytics) |
The above mentioned problem appeared when TSDB enablement was attempted in MySQL's performance datastream . Component template is as below. You can identify a field having If the
|
My initial estimate is below mentioned Package datastreams would be impacted
|
@agithomas Thanks for the list. The number is pretty low. We can wait with migrating these datasets until the bug is resolved. |
Packages with (type: flattened) in fields.yml
|
The list of package above include both logs and metric data streams. Current there are two related? behaviours:
@martijnvg, Let's know if we need to track the Synthetic source issue separately, or this issue will track both? |
This should be tracked separately. The cause of this error is different than when dimension fields are part of flattened fields. |
@martijnvg , is there a timeline you can give on this issue fix? |
@agithomas We're working on it. We hope to resolve these two issues before 8.8 FF. |
With PR #94842 we are ready to support synthetic source for flattened fields. This is required to be able to use flattened fields as dimensions in TSDB indices. I am going to work on that after clarifying a few things. Right now dimension fields are identified by a boolean parameter My idea was to use a list, part of the flattened field mapping like in the following example:
(I used made-up fields because have no idea of what sorts of data is stored in this "digest" flattened field) If that is ok for you I would ask the following questions:
|
@salvatore-campagna , related to fields having flattened type, there are two issues
The issue no - 1, is currently the blocker for me . Can can prioritise resolving this error? This git issue is created to report the issue no -1 . I understand that your comment is more around issue no -2 - how to make flattened type as a dimension field? I managed to find a work-around a solution for the second problem which is extract the required field from flattened type in the ingest pipeline, annotate the extracted field as dimension filed. In the usecase that i have, i have only one field in the flattened type and hence the options you proposed, including regular expression matching works for my usecase. @ruflin / @lalit-satapathy , considering the current usecases and potential usecases that may arise in future, do you have any recommendations ? |
@agithomas This should be fixed via #94842 |
@agithomas, thanks for your answer. Among all the options/questions I have above there is only one thing that I will advise to avoid, unless there is really a reason behind it, which is having wildcards and the reason is that wildcards might cause issues when we have too many dimensions. You are aware that we have a limit on the number of dimension fields right now (because of a Lucene limitation on the maximum size of the tsid). Using wildcards it means we don't know exactly the number of fields that will be included in the tsid, if a flattened field is used (all matching fields will be used), and that indexing might fail because we exceed that limit. Not using wildcards and specifying the fields in a list including full names makes it easier to see if we reached the limit or not. For this reason I would advise not to support wildcards unless there is a strong reason to require that. The easiest thing would be just to have a list with full names. Would this be enough? |
For the usecases that i am aware of, this would suffice - to have a list with full names. |
Closing the issue . Test results after testing using 8.8-SNAPSHOT (image pull date: today) |
If there exist a field of type
flattened
in the field mapping of a datastream, on TSDB enablement of the package in Kibana, the below error is displayed in kibana.In this case, none of the
flattened
field type is annotated withdimension: true
.Impact:
Many integration packages have
flattened
field type , TSDB enablement will fail unless this issue is fixed.The text was updated successfully, but these errors were encountered: