second run of `bundle deploy` breaks app permissions to run job/pipeline

[na-wapf]

Describe the issue

The first time I run bundle deploy everything works (see databricks.yaml extract below):

• Observing the job config in the databricks UI confirms that my_job has my_app's uuid listed with CAN_MANAGE_RUN
• Running the app, it can trigger my_job no problem

The second time I run bundle deploy the CAN_MANAGE_RUN permission disappears form the job.

• Observing databricks UI for my_job no longer lists my app in the permissions
• App can no longer trigger the job (see error message below)

Workaround

One way to fix this is bundle destroy followed by bundle deploy; this is unacceptable workaround as it changes the URL for the app.

The only other way is to remember to open the databricks UI manually click "disconnect resource" and "edit permissions" every time 😢

Configuration

Please provide a minimal reproducible configuration for the issue

# extract from databricks.yml
resources:
  apps:
    my_app:
      name: portal
      source_code_path: "../dist_app"
      resources:
        - name: "my-job-id"
          job:
            id: ${resources.jobs.my_job.id}
            permission: CAN_MANAGE_RUN
  jobs:
    my_job:
      name: My Job
      tasks:
        - ...
      queue:
        enabled: true
      environments:
        - environment_key: Default
          spec:
            environment_version: "4"
      performance_target: PERFORMANCE_OPTIMIZED

Steps to reproduce the behavior

Please list the steps required to reproduce the issue, for example:

1. Run databricks bundle deploy ...
2. Run databricks bundle run ...
3. See error

Expected Behavior

bundle deploy should be idempotent and not break permissions when it is run a second time.

Actual Behavior

Error message from app logz

databricks.sdk.errors.platform.PermissionDenied: User [[my_app service principal uuid]] does not have Manage Run or Owner or Admin permissions on job [my_job id number]. Config: host=[[databricks host url]], client_id=[[app client id]], client_REDACTED_SECRET auth_type=oauth-m2m. Env: DATABRICKS_HOST, DATABRICKS_CLIENT_ID, DATABRICKS_CLIENT_SECRET

OS and CLI version

Linux mcr.microsoft.com/devcontainers/typescript-node:1-22-bookworm
Databricks CLI v0.281.0

Edit: I just tested this with Databricks CLI v0.283.0 and the problem persists

Is this a regression?

Unknown

Debug Logs

mmm if you can tell me which bit you need to see it would be easier?

Do you want to see the contents of 04:00:39 Debug: POST /api/2.0/workspace-files/import-file/Workspace/Bundles/my_bundle/state/terraform.tfstate?overwrite=true the body of this request is the only place I see "job": {"id": ...,"permission": "CAN_MANAGE_RUN"},

[na-wapf]

For those finding this before there is a fix, I addressed this bug by adding the following to my post-deploy python script

from databricks.sdk.service.jobs import JobAccessControlRequest, JobPermissionLevel
from databricks.sdk import WorkspaceClient

wc = WorkspaceClient(...)

# retrieve job id from the deployed app resources since job.settings.name may not be unique;
job_id = next(r.job for r in wc.apps.get("my_app").resources if r.name=="my-job-id").id
assert job_id is not None

# get app service principal name
service_principal_id = wc.apps.get("my_app").service_principal_client_id
assert service_principal_id is not None

wc.jobs.update_permissions(
    job_id=job_id,
    access_control_list=[
        JobAccessControlRequest(
            service_principal_name=service_principal_id,
            permission_level=JobPermissionLevel.CAN_MANAGE_RUN
        )
    ]
)

[ncf5031]

@na-wapf I'm seeing the exact same thing and it's causing me some heartache right now. Thanks for the workaround, hoping this is updated in the next version as I also need to use valueFrom in app.yaml to pass target variables through.

[denik]

@na-wapf thanks for the report! Do you have permissions set on bundle level? Could you share output of "databricks bundle validate -t your_target -o json"? (You can send to dabs-feedback@databricks.com if don't want to post publicly).