Allow authentication using AAD. Priority is to support ServicePrincipal, but others should also be supported.

[paselem]

This will require reformatting the secrets.yaml file into per-auth type sections. Backwards compatibility with old secrets.yaml files is required.

Good resources for reference:

http://msrestazure.readthedocs.io/en/latest/msrestazure.html#module-msrestazure.azure_active_directory.

http://azure-sdk-for-python.readthedocs.io/en/latest/resourcemanagementbatch.html

[paselem]

Proposals for the body of the secrets.yaml file:

Option 1

Break out each authentication type into it's own section, and for any form of AAD auth use a Resource ID to define the batch account and storage account

pros The simplified resource ID is a simple copy/paste from the portal
cons This requires the user to have previously created the Batch and Storage accounts in the portal

# For instructions on creating a Batch and Storage account, see
# Getting Started (https://github.com/Azure/aztk/blob/master/docs/00-getting-started.md)
# NOTE - YAML requires a space after the colon. Ex: "batchaccountname: mybatchaccount"

# Use Shared Key authentication
sharedKey:
    batch:
        batchaccountname: batch_account_name
        batchaccountkey: batch_account_key
        batchserviceurl: batch_account_url

    storage:
        storageaccountname: storage_account_name
        storageaccountkey: storage_account_key
        storageaccountsuffix: core.windows.net

# Use Service Principal Authentication
servicePrincipal:
    tenantid: my_tenant_id
    clientid: my_client_id
    credential: my_crenential

    batchaccountresourceid: my_batch_account_resource_id
    storageaccountresourceid: my_storage_account_resource_id
    

# Configuration for private docker repositories. If using public containers you do not need to provide authentification
docker:
    # username:
    # password:
    # endpoint:

default:
    # SSH keys used to create a user and connect to a server.
    # The public key can either be the public key itself (ssh-rsa ...) or the path to the ssh key.
    # ssh_pub_key: ~/.ssh/id_rsa.pub

Option 2

Break out each authentication type into it's own section, and for any form of AAD auth use specific variables for Subscription ID, Resouce Group and Resource Name to define the batch account and storage account

pros If the resources don't exist yet, we can create them for the user
cons Requires a lot more copy/pasting from the portal to the configuration file. Also, limits the storage account and batch account to be in the same subscription and resource group.
note In most cases, my assumption is that users will have already created the batch and storage accounts, so I don't think creating on-the fly in this case is going to be very common

# For instructions on creating a Batch and Storage account, see
# Getting Started (https://github.com/Azure/aztk/blob/master/docs/00-getting-started.md)
# NOTE - YAML requires a space after the colon. Ex: "batchaccountname: mybatchaccount"

# Use Shared Key authentication
sharedKey:
    batch:
        batchaccountname: batch_account_name
        batchaccountkey: batch_account_key
        batchserviceurl: batch_account_url

    storage:
        storageaccountname: storage_account_name
        storageaccountkey: storage_account_key
        storageaccountsuffix: core.windows.net

# Use Service Principal Authentication
servicePrincipal:
    tenantid: my_tenant_id
    clientid: my_client_id
    credential: my_crenential

    subscription: my_subscription
    resourceGroup: my_resource_group
    batchAccountName: my_batch_account_name
    storageAccountName: my_storage_account_name

# Configuration for private docker repositories. If using public containers you do not need to provide authentification
docker:
    # username:
    # password:
    # endpoint:

default:
    # SSH keys used to create a user and connect to a server.
    # The public key can either be the public key itself (ssh-rsa ...) or the path to the ssh key.
    # ssh_pub_key: ~/.ssh/id_rsa.pub

Option 3

Very similar to option 2 but allows the storage and batch accounts to be in different subscriptions and resource groups

pros Offers more flexibility
cons Lots more configuration and can let the users create batch and storage accounts in different regions which is a bad idea

Diff from Option 2

# Use Service Principal Authentication
servicePrincipal:
    tenantid: my_tenant_id
    clientid: my_client_id
    credential: my_crenential

    batchAccountSubscription: my_subscription
    batchAccountResourceGroup: my_resource_group
    batchAccountName: my_batch_account_name
    storageAccountSubscription: my_subscription
    storageAccountResourceGroup: my_resource_group
    storageAccountName: my_storage_account_name

Summary

In any case, when using Service Principal we would have to make multiple calls to Azure to get properties and get keys required to create regular BatchClient and StorageClient objects.

[jiata]

my vote is option 1 👍

OR maybe something like this

# Use Shared Key authentication
sharedKey:
  batchaccountname: batch_account_name
  batchaccountkey: batch_account_key
  batchserviceurl: batch_account_url
  storageaccountname: storage_account_name
  storageaccountkey: storage_account_key
  storageaccountsuffix: core.windows.net

[paselem]

@jiata that looks good to me too. Maybe worth keeping them split to help keep backwards compat? Either way works for me though

@emlyn you have already started work on this in the VNET branch. What do you think of this proposal?

[emlyn]

Any of these seem reasonable to me, probably leaning towards something like option 1.
For backwards compatibility I suppose it would it accept the batch and storage keys either inside sharedKey or at the top level?
EDIT: one general point about all the options: the batchserviceurl is included under sharedKey, but I believe it is required in addition to the credentials, whether shared key or service principal (here). Should it be moved somewhere outside of any credential section?

[paselem]

@emlyn, once you have the shared principal I believe you can figure out the batch url by using the BatchManagementClient, so it should be programmatically available.

W.r.t the final design for the structure of the secrets file, lets go with option 1, but then use JS's suggestion to collapse them down so that both sharedKey and servicePrincipal have a single 'glob' instead of separate sections for Storage and Batch.

# Use Shared Key authentication
sharedKey:
  batchaccountname: batch_account_name
  batchaccountkey: batch_account_key
  batchserviceurl: batch_account_url
  storageaccountname: storage_account_name
  storageaccountkey: storage_account_key
  storageaccountsuffix: core.windows.net

Finally, for back-compat I think we would either check for batch && storage at the root level OR check for sharedKey.

@emlyn, is this something you care to work on?

[emlyn]

@paselem sounds good, I'll have a go at implementing that in the vnet branch.

[emlyn]

@paselem I've had a look at the BatchManagementClient, but it seems that it needs the subscription_id, which as far as I can tell is not currently available unless we were to add it to secrets.yaml.
Once I have a BatchManagementClient I can call batch_client.batch_account.get(...), which would give me the account_endpoint (which I assume is the same as the base_url needed for the Batch client), but in addition to the account_name/resource_id, it also needs the resource_group_name, which as far as I can tell is not available either. I suppose I could loop over batch_client.batch_account.list() for an entry with the right name but then we'd have to deal with the possibility of multiple matches from different resource groups.
Any ideas on the best way to proceed?

[paselem]

@emlyn based on the design for option 1 (below)

# Use Service Principal Authentication
servicePrincipal:
    tenantid: my_tenant_id
    clientid: my_client_id
    credential: my_crenential

    batchaccountresourceid: my_batch_account_resource_id
    storageaccountresourceid: my_storage_account_resource_id

when using the service principal we should have everything we need from the resource_id. The resource id is composed of the sub id, resource group name and resource name (i.e. the batch account name).

One example is:

/subscriptions/00000000-0000-0000-0000-00000000/resourceGroups/myResourceGroup/providers/Microsoft.Batch/batchAccounts/mybatchaccount

I think it should be trivial to simply parse that string and pull out the pieces we need. This string can be found on the properties page of a batch account or storage account in the portal.

When using sharedKeys, we can simply skip this step and create Batch and Storage clients directly.

[emlyn]

@paselem thanks, for some reason I was thinking this was just the account name, but this makes more sense! I don't have the full resource ID string on the properties page (or anywhere else that I can find), but the individual parts are there so it's not hard to construct.

[paselem]

@emlyn you're absolutely right! They are not there for the Batch UI but they are there for the Storage UI. I will make sure they are added to the Batch UI very soon so we can write some useful docs around them.