Azureml groups optimization

[miguelgfierro]

Description

Optimizing the AzureML groups to maximize compute usage.

Could you please review @pradnyeshjoshi

Related Issues

Checklist:

• I have followed the contribution guidelines and code style for this project.
• I have added tests covering my contributions.
• I have updated the documentation accordingly.
• This PR is being made to staging branch and not to main branch.

[miguelgfierro]

@pradnyeshjoshi there is a failure:

"error": ***
        "code": "UserError",
        "message": "User program failed with KeyError: 'group_notebooks_cpu_002'",
        "messageParameters": ***,
        "detailsUri": "https://aka.ms/azureml-run-troubleshooting",
        "details": []
    ***,

It seems the system is trying to find a group that doesn't exist, could you please take a look? if we are looping through a hardcoded list of groups, can we change that to a dynamic list? so we can add any new group and the system will catch it?

[pradnyeshjoshi]

@pradnyeshjoshi there is a failure:

"error": ***
        "code": "UserError",
        "message": "User program failed with KeyError: 'group_notebooks_cpu_002'",
        "messageParameters": ***,
        "detailsUri": "https://aka.ms/azureml-run-troubleshooting",
        "details": []
    ***,

It seems the system is trying to find a group that doesn't exist, could you please take a look? if we are looping through a hardcoded list of groups, can we change that to a dynamic list? so we can add any new group and the system will catch it?

@miguelgfierro the workflows now dynamically retrieve the group names from test_groups.py. Could you please review?
I will optimize the groups further in another PR.

[miguelgfierro]

Looks good to me, I can't approve because I started the PR