[O11y][Kafka] Migration for visualizations to lens in kibana 8.8.0 #5594

harnish-elastic · 2023-03-20T17:58:13Z

Enhancement

What does this PR do?

Migrate visualizations to lens for Kafka integration package
[Kafka Metrics] visualization contains 4 multiple metrics which are not supported in Lens so we manually migrated it into 4 separate legacy metrics and also changed the size for the legacy metric to make other visualizations best fit in the dashboard.
Statistics for Kafka Lens migration:

Kafka Dashboards	Before Migration			After Migration
	Search	Lens	Visualization	Search	Lens	Visualization
[Logs Kafka] Overview	2	0	2	2	2	0
[Metrics Kafka] Overview	0	0	9	0	13	0
	2	0	11	2	15	0

Checklist

I have added an entry to my package's changelog.yml file.
I have verified that panels are populated with data.
I have verified that panels are not distorted after being migrated to lens.
I have updated screenshots of dashboard.
I have verified that data count are matching and panel aggregations are same as before.

Author's checklist

Migrated panels should be removed from visualization folder.
Migrated visualizations are populating in current Kibana version 8.8.0 itself.

Related issues

Relates Lens Migration of OBS integrations #5221
Closes [O11y][Kafka] Update layout of dashboards #7148
Closes [O11y][Kafka] Lens Migration #5591

Note: - Please refer the following before migrated screenshot for [Logs Kafka] Overview dashboard, because the current screenshot is not updated.

elasticmachine · 2023-03-20T18:13:56Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-07-27T09:48:15.136+0000
Duration: 15 min 40 sec

Test stats 🧪

Test	Results
Failed	0
Passed	20
Skipped	0
Total	20

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

elasticmachine · 2023-03-20T18:14:08Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`1/1`)	💚
Files	100.0% (`1/1`)	💚
Classes	100.0% (`1/1`)	💚
Methods	72.222% (`13/18`)	❕
Lines	77.778% (`56/72`)	❕
Conditionals	100.0% (`0/0`)	💚

agithomas

LGTM!.

@lalit-satapathy , please note this is a steep jump in the version. We had considered this in this past. However do you want to reassess?

kush-elastic

LGTM!

agithomas · 2023-03-21T10:41:36Z

"Support multiple metrics in one chart using lens". Is the panel migrated to Lens or retained with old Viz panel type?

harnish-elastic · 2023-03-21T10:45:24Z

"Support multiple metrics in one chart using lens". Is the panel migrated to Lens or retained with old Viz panel type?

It's migrated into lens. We have created 4 individual lens charts.

agithomas · 2023-03-27T04:21:03Z

In the description, i read Due to not having last bucket value support in lens, we are not able to migrate into lens

The two vis that are not migrated are because of this limitation? Please confirm

lalit-satapathy · 2023-03-27T04:25:51Z

LGTM!.

@lalit-satapathy , please note this is a steep jump in the version. We had considered this in this past. However do you want to reassess?

Why kafka dashboard requires a higher kibana version?

harnish-elastic · 2023-03-27T05:24:37Z

In the description, i read Due to not having last bucket value support in lens, we are not able to migrate into lens

The two vis that are not migrated are because of this limitation? Please confirm

Yes, two vis are not migrated because of Due to not having last bucket value support in lens, we are not able to migrate into lens

harnish-elastic · 2023-03-27T05:58:44Z

LGTM!.
@lalit-satapathy , please note this is a steep jump in the version. We had considered this in this past. However do you want to reassess?

Why kafka dashboard requires a higher kibana version?

@lalit-satapathy For the Consumer Partition Reassignments, visualization contains bucket script so we need to manually migrate it into Lens. In which there are if else conditions which we are able to migrate to kibana version 8.6.0. Please refer the below screenshots for more information.

Before migration:

After migration:

…into lens_migration-kafka-8.6.0

harnish-elastic · 2023-04-04T05:26:25Z

@agithomas Updated the PR with requested changes. Thanks!

agithomas · 2023-04-06T05:58:43Z

@lalit-satapathy , are you good to approve the PR?

agithomas · 2023-04-10T16:22:51Z

@lalit-satapathy , are you good to approve the PR?

@lalit-satapathy , can you please confirm?

botelastic · 2023-05-10T16:58:40Z

Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

botelastic · 2023-06-10T06:53:23Z

Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

botelastic · 2023-07-10T06:53:30Z

Hi! This PR has been stale for a while and we're going to close it as part of our cleanup procedure. We appreciate your contribution and would like to apologize if we have not been able to review it, due to the current heavy load of the team. Feel free to re-open this PR if you think it should stay open and is worth rebasing. Thank you for your contribution!

lalit-satapathy · 2023-07-17T11:29:14Z

@harnish-elastic,

With Kafka package version now upgraded to 8.8.0, what needs to be done here?

harnish-elastic · 2023-07-17T11:49:42Z

@harnish-elastic,

With Kafka package version now upgraded to 8.8.0, what needs to be done here?

We can start taking this up since it is upgraded to 8.8.0

…into lens_migration-kafka-8.6.0 Conflicts: packages/kafka/changelog.yml packages/kafka/manifest.yml

harnish-elastic · 2023-07-25T09:17:03Z

@lalit-satapathy @agithomas Migrated the Kafka dashboard and exported to Kibana 8.7.0

agithomas · 2023-07-25T11:43:58Z

packages/kafka/kibana/dashboard/kafka-943caca0-87ee-11e7-ad9c-db80de0bf8d3.json

In the overview dashboard please have the below changes made

Single stat representing the brokers must be the first one in the first row

Have consumer partition reassignments and Kafka broker table in the second row

Have kafka topic chart and table in the thrid row

Have consumer group chart and table in the fourth row

@agithomas, We have created an enhancement issue for this task and updated the changes in this PR. Thanks!

agithomas · 2023-07-25T11:46:46Z

packages/kafka/img/filebeat-kafka-logs-overview.png

In the logs overview dashboard include the below changes

Have the legend for the Logs over time added to the bottom of the visualisation instead of right side

Reduce the space allocated for log.level and more space added for the message

@agithomas, We have created an enhancement issue for this task and updated the changes in this PR. Thanks!

agithomas · 2023-07-25T11:47:44Z

Please change the title of this PR. It is no longer migration to 8.6.0

agithomas

LGTM!

elasticmachine · 2023-07-28T09:58:53Z

Package kafka - 1.9.1 containing this change is available at https://epr.elastic.co/search?package=kafka

…5594) * Migrate kafka visualizations to lens in Kibana 8.6.0 * update changelog.yml * migrate pending visualizations to lens * export dashboards to 8.7.0 * address review comments

Migrate kafka visualizations to lens in Kibana 8.6.0

32ea877

update changelog.yml

7388f08

harnish-elastic mentioned this pull request Mar 20, 2023

[O11y][Kafka] Lens Migration #5591

Closed

8 tasks

harnish-elastic marked this pull request as ready for review March 20, 2023 18:44

harnish-elastic requested a review from a team as a code owner March 20, 2023 18:44

kush-elastic requested review from lalit-satapathy, kush-elastic and agithomas March 20, 2023 18:45

kush-elastic assigned harnish-elastic Mar 20, 2023

kush-elastic added enhancement New feature or request Integration:kafka Kafka v8.6.0 Team:Service-Integrations Label for the Service Integrations team labels Mar 20, 2023

agithomas reviewed Mar 21, 2023

View reviewed changes

kush-elastic approved these changes Mar 21, 2023

View reviewed changes

harnish-elastic added 2 commits April 3, 2023 21:13

Merge branch 'main' of https://github.com/harnish-elastic/integrations …

3d09a13

…into lens_migration-kafka-8.6.0

migrate pending visualizations to lens

b6fd0a9

harnish-elastic requested a review from agithomas April 3, 2023 15:59

agithomas mentioned this pull request Apr 25, 2023

Lens Migration of OBS integrations #5221

Closed

36 tasks

botelastic bot added the Stalled label May 10, 2023

kush-elastic removed the Stalled label May 11, 2023

botelastic bot added the Stalled label Jun 10, 2023

botelastic bot closed this Jul 10, 2023

rajvi-patel-22 reopened this Jul 10, 2023

rajvi-patel-22 requested a review from a team as a code owner July 10, 2023 06:54

botelastic bot removed the Stalled label Jul 10, 2023

harnish-elastic added 2 commits July 25, 2023 11:05

Merge branch 'main' of https://github.com/harnish-elastic/integrations …

9372945

…into lens_migration-kafka-8.6.0 Conflicts: packages/kafka/changelog.yml packages/kafka/manifest.yml

export dashboards to 8.7.0

9b3b3ee

harnish-elastic requested a review from milan-elastic July 25, 2023 09:15

agithomas reviewed Jul 25, 2023

View reviewed changes

harnish-elastic changed the title ~~[O11y][Kafka] Lens migration for visualizations to kibana version 8.6.0~~ [O11y][Kafka] Migration for visualizations to lens in kibana 8.8.0 Jul 25, 2023

address review comments

af34876

harnish-elastic requested a review from agithomas July 27, 2023 09:56

harnish-elastic mentioned this pull request Jul 27, 2023

[O11y][Kafka] Update layout of dashboards #7148

Closed

agithomas approved these changes Jul 28, 2023

View reviewed changes

harnish-elastic merged commit 4e64865 into elastic:main Jul 28, 2023
1 check passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[O11y][Kafka] Migration for visualizations to lens in kibana 8.8.0 #5594

[O11y][Kafka] Migration for visualizations to lens in kibana 8.8.0 #5594

harnish-elastic commented Mar 20, 2023 •

edited

Loading

elasticmachine commented Mar 20, 2023 •

edited

Loading

Build stats

Test stats 🧪

elasticmachine commented Mar 20, 2023 •

edited

Loading

agithomas left a comment

kush-elastic left a comment

agithomas commented Mar 21, 2023

harnish-elastic commented Mar 21, 2023

agithomas commented Mar 27, 2023

lalit-satapathy commented Mar 27, 2023

harnish-elastic commented Mar 27, 2023

harnish-elastic commented Mar 27, 2023

harnish-elastic commented Apr 4, 2023

agithomas commented Apr 6, 2023

agithomas commented Apr 10, 2023

botelastic bot commented May 10, 2023

botelastic bot commented Jun 10, 2023

botelastic bot commented Jul 10, 2023

lalit-satapathy commented Jul 17, 2023

harnish-elastic commented Jul 17, 2023

harnish-elastic commented Jul 25, 2023

agithomas Jul 25, 2023

harnish-elastic Jul 27, 2023

agithomas Jul 25, 2023

harnish-elastic Jul 27, 2023

agithomas commented Jul 25, 2023

agithomas left a comment

elasticmachine commented Jul 28, 2023

[O11y][Kafka] Migration for visualizations to lens in kibana 8.8.0 #5594

[O11y][Kafka] Migration for visualizations to lens in kibana 8.8.0 #5594

Conversation

harnish-elastic commented Mar 20, 2023 • edited Loading

What does this PR do?

Checklist

Author's checklist

Related issues

elasticmachine commented Mar 20, 2023 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

elasticmachine commented Mar 20, 2023 • edited Loading

🌐 Coverage report

agithomas left a comment

Choose a reason for hiding this comment

kush-elastic left a comment

Choose a reason for hiding this comment

agithomas commented Mar 21, 2023

harnish-elastic commented Mar 21, 2023

agithomas commented Mar 27, 2023

lalit-satapathy commented Mar 27, 2023

harnish-elastic commented Mar 27, 2023

harnish-elastic commented Mar 27, 2023

harnish-elastic commented Apr 4, 2023

agithomas commented Apr 6, 2023

agithomas commented Apr 10, 2023

botelastic bot commented May 10, 2023

botelastic bot commented Jun 10, 2023

botelastic bot commented Jul 10, 2023

lalit-satapathy commented Jul 17, 2023

harnish-elastic commented Jul 17, 2023

harnish-elastic commented Jul 25, 2023

agithomas Jul 25, 2023

Choose a reason for hiding this comment

harnish-elastic Jul 27, 2023

Choose a reason for hiding this comment

agithomas Jul 25, 2023

Choose a reason for hiding this comment

harnish-elastic Jul 27, 2023

Choose a reason for hiding this comment

agithomas commented Jul 25, 2023

agithomas left a comment

Choose a reason for hiding this comment

elasticmachine commented Jul 28, 2023

harnish-elastic commented Mar 20, 2023 •

edited

Loading

elasticmachine commented Mar 20, 2023 •

edited

Loading

elasticmachine commented Mar 20, 2023 •

edited

Loading