Cannot login to Databricks - auth endpoint cannot be reached.

[marrobi]

When trying to log into Databricks from a VM over private endpoint, I get the following:

Looks like a needed rule might be blocked.

cc @guybartal

[macborowy]

I currently worked on this in my private project. You need to create two Private Endpoints - one for private connectivity to Databricks UI, second for OAuth redirect URL.

This article might be useful: Enable Azure Private Link as a standard deployment and Step 4: Create a private endpoint to support SSO (required for UI access)

[guybartal]

@macborowy , there are two private endpoints, in fact three (the third is for the DBFS),
see here

resource "azurerm_private_endpoint" "databricks_control_plane_private_endpoint" {
  name                = "pe-adb-cp-${local.service_resource_name_suffix}"
  location            = data.azurerm_resource_group.ws.location
  resource_group_name = data.azurerm_resource_group.ws.name
  subnet_id           = data.azurerm_subnet.services.id
  tags                = local.tre_workspace_service_tags

  lifecycle { ignore_changes = [tags] }

  private_service_connection {
    name                           = "private-service-connection-databricks-control-plane-${local.service_resource_name_suffix}"
    private_connection_resource_id = azurerm_databricks_workspace.databricks.id
    is_manual_connection           = false
    subresource_names              = ["databricks_ui_api"]
  }

  private_dns_zone_group {
    name                 = "private-dns-zone-group-databricks-control-plane-${local.service_resource_name_suffix}"
    private_dns_zone_ids = [data.azurerm_private_dns_zone.databricks.id]
  }
}

resource "azurerm_private_endpoint" "databricks_auth_private_endpoint" {
  name                = "pe-adb-auth-${local.service_resource_name_suffix}"
  location            = data.azurerm_resource_group.ws.location
  resource_group_name = data.azurerm_resource_group.ws.name
  subnet_id           = data.azurerm_subnet.services.id
  tags                = local.tre_workspace_service_tags

  lifecycle { ignore_changes = [tags] }

  private_service_connection {
    name                           = "private-service-connection-databricks-auth-${local.service_resource_name_suffix}"
    private_connection_resource_id = azurerm_databricks_workspace.databricks.id
    is_manual_connection           = false
    subresource_names              = ["browser_authentication"]
  }

  private_dns_zone_group {
    name                 = "private-dns-zone-group-databricks-auth-${local.service_resource_name_suffix}"
    private_dns_zone_ids = [data.azurerm_private_dns_zone.databricks.id]
  }
}

[guybartal]

I think the problem is that we have more than one auth private link per region.

[guybartal]

There are two options to handle this issue:

1. Provide functionality to existing Databricks workspace service bundle to deploy it with auth only or without, and make sure it gets deployed only once per region (because TRE supports single region currently, we can say per TRE Env).
2. Deploy Databricks auth service as shared service.

Option 1 is simpler because Databricks needs two subnets (host and container) and can have it from workspace mechanism instead of asking it from core, but we need to make sure it gets deployed only once.

[guybartal]

Summarizing today's discussion:
option 1 would not work because of cross workspace communication is blocked.
option 2 will be implemented with isolated vnet from tre network.

Tasks:

Databricks ws service:
remove auth private endpoint from databricks ws service

Databrick shared service:

• create new shared service
  • vnet (outside tre network)
  • 2 subnets (host,container)
  • databricks workspace (allow public network access = disabled, requierd nsg rules=no azure databricks rules)
  • private endpoint to privatelink.azuredatabricks.net dns zone inside SharedSubnet insde core

Test on external subscription
* 2 db private + vms
* shared service adb