[MetaSchedule][Refactor] Introduce TuneConfig

[junrushao]

This PR unifies the existing EvolutionarySearchConfig, ReplayFuncConfig and ReplayTraceConfig into TuneConfig, and refactored the logic in meta_schedule/tune.py

[junrushao]

@masahi @zxybazh @michalpiszczek @mbrookhart would love to ask you guys to review this PR given it's going to be breaking the existing tune_relay interface

[junrushao]

@masahi @zxybazh comments addressed :-) To be extremely careful, please verify every test in tests/python/unittest/test_meta_schedule_tune_relay/tir/te.py before approving this PR. Thanks a lot!

@michalpiszczek This API breaking change precisely follows our offload discussion on Tuesday. Having left it open for a few days, and will continue to keep you updated 👍

[michalpiszczek]

Thanks @junrushao1994 , this LGTM to me!

[masahi]

In general, do you recommend setting max_trials_per_task to be the same value as max_trials_global, or some arbitrary large number? This change makes me think that the value of max_trials_per_task doesn't matter much as long as it is sufficiently big.

[masahi]

I see now that the value of max_trials_per_task is optional:

max_trials_per_task: Optional[int] = None

How about removing the explicit specification of max_trials_per_task here and elsewhere in tests, if that's not important?

[zxybazh]

In this case, I think max_trials_global is the default for max_trials_per_task, except when we want to limit the trials for each task. In these cases, if the search space is very small, the evolutionary search strategy would do an early stopping; when we only want to do test, we can limit the max_trials_global to be a small number close to num_trials_per_iter*task_num. Therefore, I think it could be a good move to use max_trials_global as default to max_trials_per_task unless specified.

[masahi]

yeah I mean, if this value is optional, at least one test should exercise that scenario / code path.

[junrushao]

Hi Masa, this is actually a great point! Generally speaking, I would recommend max_trials_per_task == max_trials_global by default, and leave the TaskSchedule to decide how to allocate those trials

[junrushao]

Let me improve this API by marking max_trials_per_task as optional

[junrushao]

oh actually I just found myself implemented the logic in create_strategy already...Let me update the docs to reflect this

[zxybazh]

IMHO we should still set max_trials_global as something like 2000 here and add comment suggested 20k or 1k * num of subgraphs beside because this is still a functionality test. I will send a PR for meta schedule tutorial as Masa suggested later this week and that would be a better example to show case our interface usage.

[junrushao]

Well, yes or no. There is no "definitely correct" suggestion in terms of how many trials should be allocated globally or locally...

[junrushao]

Thanks @masahi @zxybazh @michalpiszczek for the careful review and discussion!