Databricks Private Endpoint APIM Guide

VDAB needs to call the Databricks REST APIs on behalf of your users. These API calls allow VDAB to browse Databricks resources, trigger jobs, query SQL warehouses, read metadata, and power Databricks-backed app components.

Many enterprise Databricks workspaces are configured with an Azure Private Endpoint. This keeps the Databricks workspace off the public internet and makes it reachable only from inside your Azure Virtual Network. This is a strong enterprise security pattern, but it also means VDAB cannot call the Databricks APIs directly unless you provide a secure network path.

The recommended solution is to use Azure API Management, or APIM, as a customer-owned gateway. APIM can reach your Databricks private endpoint from inside your Azure environment while exposing a public HTTPS endpoint that VDAB can call with an API subscription key.

1. Create Azure API Management

Create or use an Azure API Management instance in your Azure environment.

Configure APIM so it can reach the VNet where the Databricks private endpoint is available.

APIM should expose a public HTTPS endpoint for VDAB while routing backend traffic to Databricks privately through your Azure network.

2. Configure APIM to Forward to Databricks

Now, create an APIM API that represents your Databricks workspace.

An example APIM endpoint is:

https://customer-vdab-gateway.azure-api.net/databricks

Configure the APIM backend to forward requests to your private Databricks workspace URL, such as:

https://adb-0000000000000000.00.azuredatabricks.net

APIM should preserve the request method, path, query string, request body, and Databricks Authorization header. It should return the Databricks response status code and response body back to VDAB.

At minimum, allow the Databricks API paths required by the VDAB features you plan to use. Common path prefixes include:

• /api/2.0/ for core Databricks REST APIs.
• /api/2.1/ for jobs and other versioned Databricks APIs when used.
• /oidc/ for Databricks OAuth authorize and token endpoints.
• /apps/ for Databricks Apps links and app-related routes when needed.

Keep the path list narrow. Add optional paths only after confirming that a VDAB feature needs them.

3. Protect the APIM Endpoint

APIM should require an API subscription key for VDAB calls. This gives your organization a credential that can be rotated or revoked independently from Databricks credentials.

In APIM, create a product or subscription for VDAB and generate an API subscription key. Keep this key available for the VDAB connection step.

You can also add other APIM controls based on your organization’s policies:

• HTTPS only.
• Rate limits appropriate for your Databricks workspace.
• Request size limits appropriate for expected Databricks API usage.
• Audit logging that does not include secrets.
• IP restrictions to VDAB’s published static egress IP addresses if your team wants an additional network control.

4. Configure Databricks Authorization

VDAB still needs Databricks authorization values so users can connect to the workspace and VDAB can call the Databricks APIs. APIM only provides the network path to the private workspace.

In Databricks, open or create the App Connection used by VDAB.

1. Register the VDAB callback URL, such as https://app.vdab.ai/callback.
2. Grant the API access required for the Databricks features you plan to use in VDAB.
3. Copy the Client ID.
4. Copy the Client Secret.
5. Note the Access Token TTL and Refresh Token Lifespan.

Keep these values available for the VDAB connection step.

5. Add the Private Databricks Workspace in VDAB

Now the private Databricks workspace can be added to VDAB through your APIM URL.

In VDAB, open Databricks Connections and add or edit the workspace connection.

1. Enter a Workspace Display Name, such as Production Workspace via APIM.
2. Enter the APIM endpoint as the Workspace URL, such as https://customer-vdab-gateway.azure-api.net/databricks.
3. Enter the APIM API Subscription Key.
4. Enter the Databricks App Connection Client ID.
5. Enter the Databricks App Connection Client Secret.
6. Enter the Access Token TTL from Databricks.
7. Enter the Refresh Token Lifespan from Databricks.
8. Save the connection.

After saving, select Authorize on the workspace card and complete the Databricks authorization flow.

Security Checklist

Before using the connection for production VDAB apps, confirm the following items:

• Databricks public network access remains disabled.
• APIM can reach Databricks through the private endpoint and private DNS.
• APIM requires an API subscription key for VDAB calls.
• APIM only allows the Databricks paths required by VDAB.
• APIM logs do not contain tokens, secrets, full authorization headers, API subscription keys, or sensitive response bodies.
• The Databricks App Connection Client Secret is stored only in VDAB and rotated according to your organization’s policy.
• Your team has a process for rotating the APIM subscription key or disabling APIM access if needed.

Troubleshooting

If VDAB receives a network timeout, confirm that APIM can reach the private Databricks endpoint. Check VNet integration, private DNS, route tables, network security groups, and Databricks private endpoint health.

If VDAB receives 401 or invalid_client during authorization, confirm that the Client ID and Client Secret in VDAB match the current values from the Databricks App Connection.

If VDAB receives 403 from APIM, confirm that the API subscription key entered in VDAB is correct and that the APIM product or subscription is active.

If APIM returns 404 for a Databricks feature, confirm whether the required Databricks path prefix is allowed by APIM. Only add new path prefixes after validating that VDAB needs them.