[SDK] Support More MCKind in tune

[Electronic-Waste]

What you would like to be added?

Support specifying more types of metrics collector in the tune function.

katib/sdk/python/v1beta1/kubeflow/katib/api/katib_client.py

Line 189 in 51b246f

metrics_collector_config: Dict[str, Any] = {"kind": "StdOut"},

Why is this needed?

Currently, we only support kind param in metrics_collector_config, which means only StdOut and Push collector can be specified in tune function.

katib/sdk/python/v1beta1/kubeflow/katib/api/katib_client.py

Lines 387 to 391 in 51b246f

	# Add metrics collector to the Katib Experiment.
	# Up to now, We only support parameter `kind`, of which default value is `StdOut`, to specify the kind of metrics collector.
	experiment.spec.metrics_collector = models.V1beta1MetricsCollectorSpec(
	collector=models.V1beta1CollectorSpec(kind=metrics_collector_config["kind"])
	)

However, there are many other collector types, such as File, Prometheus, Custom, TensorFlowEvent collector. They are important too. The tune function would be more flexible and powerful if users can specify all these kind of collectors.

Currently we support:

• Push Collector: [GSoC] Add New Parameter in tune #2369
• StdOut Collector: [GSoC] Add New Parameter in tune #2369
• File Collector
• Prometheus Collector
• Custom Collector
• TensorFlowEvent Collector

Love this feature?

Give it a 👍 We prioritize the features with most 👍

[Electronic-Waste]

/remove-label lifecycle/needs-triage

/area sdk

[google-oss-prow]

@Electronic-Waste: The label(s) `/remove-label lifecycle/needs-triage

cannot be applied. These labels are supported:tide/merge-method-merge, tide/merge-method-rebase, tide/merge-method-squash. Is this label configured under labels -> additional_labelsorlabels -> restricted_labelsinplugin.yaml`?

In response to this:

/remove-label lifecycle/needs-triage

/area sdk

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[Electronic-Waste]

/good-first-issue

[google-oss-prow]

@Electronic-Waste:
This request has been marked as suitable for new contributors.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-good-first-issue command.

In response to this:

/good-first-issue

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[prakhar479]

hey @Electronic-Waste I wanted to work on this issue and have some doubts regarding the same. Are the only modifications required in tune function to allow other function signature or does V1beta1CollectorSpec method from models class would also need some changes to handle these different collector types. Also, what is the specific representation (string key) for these other collector types (any example would be helpful). Thanks!

[Electronic-Waste]

Hi @prakhar479 , thank you for your passion for this issue!

I think V1beta1CollectorSpec is already up-to-date to handle different types of MC. So probably you only need to add some fields to metrics_collector_config and modify these lines of code in tune function.

katib/sdk/python/v1beta1/kubeflow/katib/api/katib_client.py

Lines 387 to 391 in b6f7cfd

	# Add metrics collector to the Katib Experiment.
	# Up to now, We only support parameter `kind`, of which default value is `StdOut`, to specify the kind of metrics collector.
	experiment.spec.metrics_collector_spec = models.V1beta1MetricsCollectorSpec(
	collector=models.V1beta1CollectorSpec(kind=metrics_collector_config["kind"])
	)

Also, you may find these examples helpful to you :)

https://github.com/kubeflow/katib/blob/master/examples/v1beta1/metrics-collector/custom-metrics-collector.yaml
https://github.com/kubeflow/katib/blob/master/examples/v1beta1/metrics-collector/file-metrics-collector-with-json-format.yaml
https://github.com/kubeflow/katib/blob/master/examples/v1beta1/metrics-collector/file-metrics-collector.yaml

cc👀 @kubeflow/wg-automl-leads

[Electronic-Waste]

/assign @prakhar479

[prakhar479]

thanks for the helpful insight @Electronic-Waste. I wanted to confirm one doubt I encountered while going through the code.
V1beta1CollectorSpec method has a custom_collector: Any | None = None field which is not being used currently in the initialization. can we set up extra fields in the metrics_collector_config for other types of collectors and then pass them in custom_collector field in line 390

[prakhar479]

Also, can someone lead me to a guide to set up and run tests for the repo. I searched around and was able to get few things going but couldn't run e2e tests due to incomplete set up on my end. Thanks!

[Electronic-Waste]

can we set up extra fields in the metrics_collector_config for other types of collectors and then pass them in custom_collector field in line 390

@prakhar479 Yes, of course. You can add any fields you want in metrics_collector_config.

Also, can someone lead me to a guide to set up and run tests for the repo. I searched around and was able to get few things going but couldn't run e2e tests due to incomplete set up on my end. Thanks!

In fact, I run the e2e tests by submitting PRs to this repo. E2e tests will be triggered automatically in CI/CD. You can also consult @andreyvelich @tenzen-y @johnugeorge for more suggestions.

[Electronic-Waste]

/remove-label lifecycle/needs-triage