A framework for flexible field transformations

[rashidkpc]

We need a way to transform the view of a field's value in a flexible way. Right now we have things like noXml, urlLink and stringify that do some minor transformations to make fields display in a way that is acceptable to the user. However there's currently no way to perform optional transformations, for example, transforming dates to local time, or prettying fields that contain JSON.

I'm opening this ticket for discussion on how to accomplish that. We need to discuss:

• scope
  • How far do we go? The easiest way to restrict this to the table panel, and this may be the only way, since to transform anything facet related would require the transformation be done on the elasticsearch side
• code structure
  • Where does this functionality go? How does the user implement new transformations?
• UI
  • What does the UI look like for configuring this? How do you display that a field is being transformed and how its happening?

[grubino]

I need something similar to that, I think. In a "terms" panel, it would be really nice if I could transform "legend" labels before displaying them. My use case is that the values that I'm counting in my terms panel are integer references. The references themselves are not human-readable and right now they require a manual lookup to translate into human readable form. I believe that kibana could do that lookup automatically without much trouble. The reason I don't just perform the lookup at load time is because it takes O(100ms) and I don't necessarily want to do it for every record (of which there are O(100000) per hour).

Unfortunately, I'm not much of a JS programmer, so I don't even know where to begin.

[dhawani]

As far as transformation of date & time is concerned, I think not only allow transformation to local time but also to other timezone should be allowed. For instance, data center may be in US serving African countries and user (kibana viewer) local time zone may be India. But as the service is for Africa, user might want to see time transformed to African timezone.

Saying so, i would expect all date/time related panels to show me same information. Hence, if i am viewing in Africa/Nairobi timezone, i would expect the selection of time from timepicker is in same zone. Also, interaction with histogram panel should be on same zone and even displaying in table should show in same zone.

Of-course doing this is a challenge as all zones may be different. However, some standards can be assumed to simplify, like storing all time in UTC at server, standard format of date/time, and kibana can (on client side) transform these time/date to appropriate zone as applicable.

[avleen]

This would be really cool.
Something simple in the table, could be just to apply a regex with grouping, then print the groups.
I could really make use of this - it would be even more awesome if that transformed data could them be manipulated further, or used to generate more queries but that's a but farther down the line :)

[rashidkpc]

The issue is much more about how to implement it than which specific transformations should be implemented.

And the problem with grouping goes back to that, how flexible should the transformations really be? Obviously transforming on a timestamp is easy, no matter how you adjust it, it will always sort the same, filters will always apply to it as normal. But say you synthesized new fields using a regex with grouping. You can't really do anything with them, sorting would still occur on the original value, filters would still only apply to the original value.

[t0mas]

I have the exact same use-case as @grubino so I've created a simple option in the "terms" panel to fetch a translator document (basically a mapping from "term" to "label") from Elasticsearch and translate the IDs to readable labels. This keeps the original term in a new field "term" to allow for filtering etc and changes only the "label" used for displaying.

It's in this commit: t0mas@eea0035

If this could be a first step towards a field-transformations framework I could polish it a little and create a pull-request. Feedback on where/how you would want to have this in a more generalized way is welcome :-)

[spalger]

This is on the roadmap now. Track #1543 for updates