Azure Databricks

Expert knowledge for Azure Databricks development including troubleshooting, best practices, deci…

by microsoftdocs · MIT

The skill

Expert knowledge for Azure Databricks development including troubleshooting, best practices, decision making, architecture & design patterns, limits & quotas, security, configuration, integrations & coding patterns, and deployment. Use when using Unity Catalog, Lakehouse/Lakeflow pipelines, AI Runtime/MLflow, model serving, or agents on Databricks, and other Azure Databricks related development tasks. Not for Azure Synapse Analytics (use azure-synapse-analytics), Azure HDInsight (use azure-hdinsight), Azure Machine Learning (use azure-machine-learning), Azure Data Factory (use azure-data-factory).

Azure Databricks Skill

This skill provides expert guidance for Azure Databricks. Covers troubleshooting, best practices, decision making, architecture & design patterns, limits & quotas, security, configuration, integrations & coding patterns, and deployment. It combines local quick-reference content with remote documentation fetching capabilities.

How to Use This Skill

IMPORTANT for Agent: Use the Category Index below to locate relevant sections. For categories with line ranges (e.g., L35-L120), use read_file with the specified lines. For categories with file links (e.g., [security.md](security.md)), use read_file on the linked reference file

IMPORTANT for Agent: If metadata.generated_at is more than 3 months old, suggest the user pull the latest version from the repository. If mcp_microsoftdocs tools are not available, suggest the user install it: Installation Guide

This skill requires network access to fetch documentation content:

Preferred: Use mcp_microsoftdocs:microsoft_docs_fetch with query string from=learn-agent-skill. Returns Markdown.
Fallback: Use fetch_webpage with query string from=learn-agent-skill&accept=text/markdown. Returns Markdown.

Category Index

Category	Location	Description
Troubleshooting	L37-L182	Diagnosing and fixing Azure Databricks issues: clusters, jobs, SQL errors, connectors/Lakeflow pipelines, model serving, agents, Git/IDE/CLI, and performance or reliability problems.
Best Practices	L183-L400	End-to-end Databricks best practices: cost control, governance, security, performance, streaming, RAG/AI, Lakehouse data engineering, BI, Lakeflow, Model Serving, and troubleshooting.
Decision Making	decision-making.md	Guides for architectural and cost decisions: choosing Databricks runtimes, compute, AI/ML/agent options, Unity Catalog, Lakeflow, connectors, and planning/migrating workloads and governance.
Architecture & Design Patterns	architecture-patterns.md	Architectural blueprints and patterns for Databricks and Lakebase: DR/HA, networking, storage, medallion and CDC pipelines, streaming, SQL scaling, feature stores, agents, and secure, well-architected platforms.
Limits & Quotas	limits-quotas.md	Limits, quotas, and constraints for Databricks compute, AI/GenAI, connectors, Lakeflow/Lakehouse features, Unity Catalog, and platform resources, plus related rate limits, regions, and usage caps.
Security	security.md	Identity, access control, encryption, networking, compliance, and governance for Azure Databricks and Unity Catalog, including ABAC/RBAC, tokens, secrets, CI/CD auth, and secure external connections.
Configuration	configuration.md	Configuring every aspect of Azure Databricks: accounts, workspaces, networking, security, storage, compute, AI/ML/agents, Lakeflow, Unity Catalog, SQL, streaming, and automation tooling.
Integrations & Coding Patterns	integrations.md	Patterns and APIs for integrating Databricks with agents, apps, BI/ETL tools, external DBs/services, Lakehouse Federation, Lakeflow, AI Runtime, MLflow, and SQL/PySpark UDFs and functions.
Deployment	deployment.md	Deploying and managing Azure Databricks workspaces, apps, agents, Lakeflow/Lakebase, and models using ARM/Bicep/Terraform/Bundles, plus CI/CD, DR, autoscaling, and infrastructure-as-code.

Troubleshooting

Topic	URL
Use Databricks identity management readiness report	https://learn.microsoft.com/en-us/azure/databricks/admin/users-groups/automatic-identity-management/readiness-report
Handle long-running tasks with Supervisor Agent continuation	https://learn.microsoft.com/en-us/azure/databricks/agents/agent-bricks/multi-agent-supervisor-long-running-tasks
Agent Bricks FAQ for building and running agents	https://learn.microsoft.com/en-us/azure/databricks/agents/concepts/faq
Debug custom code agents on Azure Databricks	https://learn.microsoft.com/en-us/azure/databricks/agents/custom-agents/debug-agent
Detect and clean up unused AI Search endpoints	https://learn.microsoft.com/en-us/azure/databricks/ai-search/unused-endpoints
Troubleshoot Databricks Artifact Registry issues	https://learn.microsoft.com/en-us/azure/databricks/artifact-registry/troubleshooting
Troubleshoot Azure Databricks cluster termination errors	https://learn.microsoft.com/en-us/azure/databricks/compute/troubleshooting/cluster-error-codes
Debug Spark applications using Databricks Spark UI	https://learn.microsoft.com/en-us/azure/databricks/compute/troubleshooting/debugging-spark-ui
Monitor Databricks dashboard usage with audit logs	https://learn.microsoft.com/en-us/azure/databricks/dashboards/monitor-usage
Troubleshoot common Databricks CLI issues	https://learn.microsoft.com/en-us/azure/databricks/dev-tools/cli/troubleshooting
Use Databricks app details for monitoring and troubleshooting	https://learn.microsoft.com/en-us/azure/databricks/dev-tools/databricks-apps/view-app-details
Troubleshoot Databricks Connect for Python issues	https://learn.microsoft.com/en-us/azure/databricks/dev-tools/databricks-connect/python/troubleshooting
Troubleshoot Databricks Connect for Scala issues	https://learn.microsoft.com/en-us/azure/databricks/dev-tools/databricks-connect/scala/troubleshooting
Troubleshoot Databricks Terraform provider issues	https://learn.microsoft.com/en-us/azure/databricks/dev-tools/terraform/troubleshoot
Resolve common issues with Databricks VS Code extension	https://learn.microsoft.com/en-us/azure/databricks/dev-tools/vscode-ext/faqs
Troubleshoot issues with the Databricks IDE extension	https://learn.microsoft.com/en-us/azure/databricks/dev-tools/vscode-ext/troubleshooting
Handle ARITHMETIC_OVERFLOW errors in Databricks	https://learn.microsoft.com/en-us/azure/databricks/error-messages/arithmetic-overflow-error-class
Resolve CAST_INVALID_INPUT errors in Databricks	https://learn.microsoft.com/en-us/azure/databricks/error-messages/cast-invalid-input-error-class
Diagnose DC_GA4_RAW_DATA_ERROR in Databricks	https://learn.microsoft.com/en-us/azure/databricks/error-messages/dc-ga4-raw-data-error-error-class
Understand DC_SFDC_API_ERROR in Databricks connectors	https://learn.microsoft.com/en-us/azure/databricks/error-messages/dc-sfdc-api-error-error-class
Diagnose DC_SQLSERVER_ERROR in Databricks connectors	https://learn.microsoft.com/en-us/azure/databricks/error-messages/dc-sqlserver-error-error-class
Handle DELTA_ICEBERG_COMPAT_V1_VIOLATION errors	https://learn.microsoft.com/en-us/azure/databricks/error-messages/delta-iceberg-compat-v1-violation-error-class
Resolve DIVIDE_BY_ZERO error in Azure Databricks SQL	https://learn.microsoft.com/en-us/azure/databricks/error-messages/divide-by-zero-error-class
Handle Azure Databricks error condition strings	https://learn.microsoft.com/en-us/azure/databricks/error-messages/error-classes
Troubleshoot EWKB_PARSE_ERROR in Databricks	https://learn.microsoft.com/en-us/azure/databricks/error-messages/ewkb-parse-error-error-class
Troubleshoot EWKT_PARSE_ERROR in Databricks	https://learn.microsoft.com/en-us/azure/databricks/error-messages/ewkt-parse-error-error-class
Troubleshoot GEOJSON_PARSE_ERROR in Databricks	https://learn.microsoft.com/en-us/azure/databricks/error-messages/geojson-parse-error-error-class
Resolve GROUP_BY_AGGREGATE errors in Databricks SQL	https://learn.microsoft.com/en-us/azure/databricks/error-messages/group-by-aggregate-error-class
Handle H3_INVALID_CELL_ID errors in Databricks	https://learn.microsoft.com/en-us/azure/databricks/error-messages/h3-invalid-cell-id-error-class
Fix H3_INVALID_GRID_DISTANCE_VALUE errors	https://learn.microsoft.com/en-us/azure/databricks/error-messages/h3-invalid-grid-distance-value-error-class
Fix H3_INVALID_RESOLUTION_VALUE errors	https://learn.microsoft.com/en-us/azure/databricks/error-messages/h3-invalid-resolution-value-error-class
Resolve H3_NOT_ENABLED errors in Databricks	https://learn.microsoft.com/en-us/azure/databricks/error-messages/h3-not-enabled-error-class
Handle INSUFFICIENT_TABLE_PROPERTY errors	https://learn.microsoft.com/en-us/azure/databricks/error-messages/insufficient-table-property-error-class
Resolve INVALID_ARRAY_INDEX errors in Databricks	https://learn.microsoft.com/en-us/azure/databricks/error-messages/invalid-array-index-error-class
Resolve INVALID_ARRAY_INDEX_IN_ELEMENT_AT errors	https://learn.microsoft.com/en-us/azure/databricks/error-messages/invalid-array-index-in-element-at-error-class
Fix MISSING_AGGREGATION errors in GROUP BY	https://learn.microsoft.com/en-us/azure/databricks/error-messages/missing-aggregation-error-class
Troubleshoot ROW_COLUMN_ACCESS errors in Databricks	https://learn.microsoft.com/en-us/azure/databricks/error-messages/row-column-access-error-class
Interpret Azure Databricks SQLSTATE error codes	https://learn.microsoft.com/en-us/azure/databricks/error-messages/sqlstates
Resolve TABLE_OR_VIEW_NOT_FOUND errors in Databricks	https://learn.microsoft.com/en-us/azure/databricks/error-messages/table-or-view-not-found-error-class
Fix UNRESOLVED_ROUTINE errors in Databricks SQL	https://learn.microsoft.com/en-us/azure/databricks/error-messages/unresolved-routine-error-class
Handle UNSUPPORTED_TABLE_OPERATION errors	https://learn.microsoft.com/en-us/azure/databricks/error-messages/unsupported-table-operation-error-class
Handle UNSUPPORTED_VIEW_OPERATION errors	https://learn.microsoft.com/en-us/azure/databricks/error-messages/unsupported-view-operation-error-class
Troubleshoot WKB_PARSE_ERROR in Databricks	https://learn.microsoft.com/en-us/azure/databricks/error-messages/wkb-parse-error-error-class
Troubleshoot WKT_PARSE_ERROR in Databricks	https://learn.microsoft.com/en-us/azure/databricks/error-messages/wkt-parse-error-error-class
Troubleshoot common Genie Agent query issues	https://learn.microsoft.com/en-us/azure/databricks/genie-agents/troubleshooting
Auto Loader FAQ and operational guidance	https://learn.microsoft.com/en-us/azure/databricks/ingestion/cloud-object-storage/auto-loader/faq
Monitor and troubleshoot Auto Loader pipelines	https://learn.microsoft.com/en-us/azure/databricks/ingestion/cloud-object-storage/auto-loader/observability
Troubleshoot 1Password Event Logs connector errors	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/1password-event-logs-troubleshoot
Troubleshoot Aha! managed connector issues	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/aha-troubleshoot
Troubleshoot Akamai WAF connector authentication and errors	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/akamai-waf-troubleshoot
Troubleshoot Databricks Amplitude connector issues	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/amplitude-troubleshoot
Troubleshoot Databricks Anaplan connector errors	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/anaplan-troubleshoot
Troubleshoot Anthropic connector ingestion issues	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/anthropic-troubleshoot
Troubleshoot Anysphere Audit Logs connector issues	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/anysphere-audit-logs-troubleshoot
Troubleshoot Anysphere Organization connector problems	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/anysphere-organization-troubleshoot
Troubleshoot Atlassian audit logs connector errors	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/atlassian-audit-logs-troubleshoot
Troubleshoot Databricks Celigo managed connector errors	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/celigo-troubleshoot
Troubleshoot Databricks Confluence ingestion errors	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/confluence-troubleshoot
Troubleshoot CrowdStrike Falcon Event Stream connector issues	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/crowdstrike-falcon-event-stream-troubleshoot
Troubleshoot Databricks Dynamics 365 connector issues	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/d365-faq
Troubleshoot Dynamics 365 ingestion via Synapse Link	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/d365-troubleshoot
Troubleshoot full refresh of Lakeflow pipelines	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/full-refresh
Troubleshoot common GitLab connector errors	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/gitlab-troubleshoot
Troubleshoot Glean connector errors in Lakeflow	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/glean-troubleshoot
Troubleshoot Gmail connector ingestion issues	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/gmail-troubleshoot
Troubleshoot Google Ads connector errors	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/google-ads-troubleshoot
Troubleshoot GA4 Raw Data connector issues	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/google-analytics-troubleshoot
Troubleshoot Google Drive ingestion in Lakeflow	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/google-drive-troubleshoot
Troubleshoot Databricks Google Search Console connector	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/google-search-console-troubleshoot
Troubleshoot Databricks Google Workspace connector errors	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/google-workspace-troubleshoot
Troubleshoot HubSpot connector ingestion problems	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/hubspot-troubleshoot
Troubleshoot Jira ingestion issues in Lakeflow	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/jira-troubleshoot
Troubleshoot Databricks Kafka ingestion connector errors	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/kafka-troubleshoot
Troubleshoot LinkedIn Ads connector errors	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/linkedin-ads-troubleshoot
Troubleshoot Marketo connector pipeline errors	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/marketo-troubleshoot
Diagnose and fix Meta Ads Lakeflow Connect ingestion errors	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/meta-ads-troubleshoot
Troubleshoot Microsoft 365 audit connector issues	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/microsoft-365-troubleshoot
Troubleshoot Monday.com Lakeflow connector errors	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/monday-com-troubleshoot
Troubleshoot MySQL ingestion issues in Lakeflow Connect	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/mysql-troubleshoot
Troubleshoot Netskope Logs Lakeflow connector	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/netskope-logs-troubleshoot
Troubleshoot Notion connector authentication and sync issues	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/notion-troubleshoot
Troubleshoot Databricks Okta System Logs connector errors	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/okta-system-logs-troubleshoot
Troubleshoot Databricks OpenAI connector issues	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/openai-troubleshoot
Troubleshoot Oracle integrated CDC ingestion issues	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/oracle-troubleshoot
Troubleshoot Databricks Outlook ingestion connector issues	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/outlook-troubleshoot
Troubleshoot Databricks PagerDuty connector errors	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/pagerduty-troubleshoot
Troubleshoot Databricks Pendo connector issues	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/pendo-troubleshoot
Troubleshoot Databricks PostgreSQL ingestion issues	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/postgresql-troubleshoot
Troubleshoot Databricks Productboard connector errors	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/productboard-troubleshoot
Troubleshoot Databricks query-based connectors	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/query-based-troubleshoot
Troubleshoot Databricks RabbitMQ ingestion connector	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/rabbitmq-troubleshoot
Troubleshoot Databricks Reddit Ads connector issues	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/reddit-ads-troubleshoot
FAQ for Databricks Salesforce ingestion connector	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/salesforce-faq
Troubleshoot Databricks Salesforce ingestion issues	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/salesforce-troubleshoot
Troubleshoot SendGrid connector authentication and rate limits	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/sendgrid-troubleshoot
Troubleshoot ServiceNow connector ingestion problems	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/servicenow-troubleshoot
Troubleshoot Salesforce Marketing Cloud ingestion issues	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/sfmc-troubleshoot
Troubleshoot Databricks SharePoint ingestion errors	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/sharepoint-troubleshoot
Troubleshoot Databricks Shopify connector issues	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/shopify-troubleshoot
Troubleshoot Databricks Smartsheet connector errors	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/smartsheet-troubleshoot
Troubleshoot Databricks SQL Server ingestion issues	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/sql-server-troubleshoot
Troubleshoot Databricks Square connector errors	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/square-troubleshoot
Troubleshoot Databricks Strac connector errors	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/strac-troubleshoot
Troubleshoot Databricks TikTok Ads connector issues	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/tiktok-ads-troubleshoot
Troubleshoot Databricks Lakeflow ingestion pipelines	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/troubleshoot
Resolve UNITY_CATALOG_INITIALIZATION_FAILED in pipelines	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/uc-initialization-troubleshoot
Troubleshoot Databricks Veeva Vault connector errors	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/veeva-vault-troubleshoot
Troubleshoot Databricks Verkada connector issues	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/verkada-troubleshoot
Troubleshoot Wiz Audit Logs connector errors	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/wiz-audit-logs-troubleshoot
Troubleshoot Workday Activity Logging connector issues	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/workday-activity-logging-troubleshoot
Troubleshoot common Workday HCM connector errors	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/workday-hcm-troubleshoot
Troubleshoot Databricks Workday ingestion issues	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/workday-reports-troubleshoot
Troubleshoot common Workiva connector errors	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/workiva-troubleshoot
Troubleshoot Zendesk Support connector issues	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/zendesk-support-troubleshoot
Troubleshoot Databricks Zip connector errors	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/zip-troubleshoot
Troubleshoot Databricks Zoho Books connector issues	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/zoho-books-troubleshoot
Handle Zerobus Ingest error codes and failures	https://learn.microsoft.com/en-us/azure/databricks/ingestion/zerobus-errors
Understand and use Azure Databricks init script logging	https://learn.microsoft.com/en-us/azure/databricks/init-scripts/logs
Diagnose and optimize Lakeflow job performance	https://learn.microsoft.com/en-us/azure/databricks/jobs/diagnose-job-performance
Troubleshoot and repair Lakeflow job failures	https://learn.microsoft.com/en-us/azure/databricks/jobs/repair-job-failures
Resolve high initialization times in pipelines	https://learn.microsoft.com/en-us/azure/databricks/ldp/fix-high-init
Recover Lakeflow pipelines from checkpoint failures	https://learn.microsoft.com/en-us/azure/databricks/ldp/recover-streaming
Track AI Runtime usage and troubleshoot errors	https://learn.microsoft.com/en-us/azure/databricks/machine-learning/ai-runtime/guides/
Debug and explore Databricks Feature Views in Unity Catalog	https://learn.microsoft.com/en-us/azure/databricks/machine-learning/feature-store/explore-feature-views
Troubleshoot Databricks Feature Store and limits	https://learn.microsoft.com/en-us/azure/databricks/machine-learning/feature-store/troubleshooting-and-limitations
Debug Databricks Model Serving endpoint issues	https://learn.microsoft.com/en-us/azure/databricks/machine-learning/model-serving/model-serving-debug
Diagnose Databricks model serving with Genie Code	https://learn.microsoft.com/en-us/azure/databricks/machine-learning/model-serving/model-serving-genie-code
Debug Python code in Databricks notebooks	https://learn.microsoft.com/en-us/azure/databricks/notebooks/debugger
Monitor and troubleshoot Lakebase logical replication	https://learn.microsoft.com/en-us/azure/databricks/oltp/projects/logical-replication-manage
Detect and resolve Lakebase issues with Insights	https://learn.microsoft.com/en-us/azure/databricks/oltp/projects/observability-ai-insights
Troubleshoot Databricks OpenSharing data access errors	https://learn.microsoft.com/en-us/azure/databricks/opensharing/troubleshooting
Troubleshoot failing Spark jobs and removed executors	https://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/failing-spark-jobs
Handle lost spot instances in Databricks clusters	https://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/losing-spot-instances
Troubleshoot Spark executor memory issues in Databricks	https://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/spark-memory-issues
Troubleshoot Databricks Partner Connect issues	https://learn.microsoft.com/en-us/azure/databricks/partner-connect/troubleshoot
Troubleshoot Databricks publishing and connections to Power BI	https://learn.microsoft.com/en-us/azure/databricks/partners/bi/power-bi/troubleshooting
Troubleshoot Azure Databricks Git folder errors	https://learn.microsoft.com/en-us/azure/databricks/repos/errors-troubleshooting
Detect and repair Delta table issues with FSCK REPAIR TABLE	https://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/delta-fsck
Handle INVALID_UTF8_STRING errors in Databricks SQL	https://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/functions/validate_utf8
Monitor and troubleshoot materialized view refreshes	https://learn.microsoft.com/en-us/azure/databricks/sql/mv-st/materialized-monitor
Use query history to troubleshoot Databricks SQL performance	https://learn.microsoft.com/en-us/azure/databricks/sql/user/queries/query-history
Interpret Databricks SQL query profiles for performance troubleshooting	https://learn.microsoft.com/en-us/azure/databricks/sql/user/queries/query-profile

Best Practices

Topic	URL
Tag Databricks resources for cost attribution	https://learn.microsoft.com/en-us/azure/databricks/admin/account-settings/usage-detail-tags
Use default Databricks compute policy families	https://learn.microsoft.com/en-us/azure/databricks/admin/clusters/policy-families
Apply identity best practices in Azure Databricks	https://learn.microsoft.com/en-us/azure/databricks/admin/users-groups/best-practices
Apply best practices for Databricks serverless workspaces	https://learn.microsoft.com/en-us/azure/databricks/admin/workspace/serverless-workspaces-best-practices
Synthetically generate agent evaluation sets	https://learn.microsoft.com/en-us/azure/databricks/agents/agent-evaluation/synthesize-evaluation-set
Load test Databricks Apps agents for QPS capacity	https://learn.microsoft.com/en-us/azure/databricks/agents/custom-agents/load-test-agent-app
Measure RAG performance with retrieval and response metrics	https://learn.microsoft.com/en-us/azure/databricks/agents/tutorials/ai-cookbook/evaluate-assess-performance
Define RAG application quality with evaluation sets	https://learn.microsoft.com/en-us/azure/databricks/agents/tutorials/ai-cookbook/evaluate-define-quality
Evaluate and monitor RAG applications for quality, cost, latency	https://learn.microsoft.com/en-us/azure/databricks/agents/tutorials/ai-cookbook/fundamentals-evaluation-monitoring-rag
Design and optimize RAG inference chains on Databricks	https://learn.microsoft.com/en-us/azure/databricks/agents/tutorials/ai-cookbook/fundamentals-inference-chain-rag
Build Databricks unstructured RAG data pipelines	https://learn.microsoft.com/en-us/azure/databricks/agents/tutorials/ai-cookbook/quality-data-pipeline-rag
Improve RAG application quality via key tuning knobs	https://learn.microsoft.com/en-us/azure/databricks/agents/tutorials/ai-cookbook/quality-overview
Optimize RAG chain components for better responses	https://learn.microsoft.com/en-us/azure/databricks/agents/tutorials/ai-cookbook/quality-rag-chain
Control coding agent costs with Unity Gateway budgets and routing	https://learn.microsoft.com/en-us/azure/databricks/ai-gateway/coding-agent-costs
Apply performance best practices for AI Search	https://learn.microsoft.com/en-us/azure/databricks/ai-search/best-practices
Load test Databricks AI Search endpoints	https://learn.microsoft.com/en-us/azure/databricks/ai-search/endpoint-load-test
Improve Databricks AI Search retrieval quality	https://learn.microsoft.com/en-us/azure/databricks/ai-search/retrieval-quality
Migrate Databricks library installs from init scripts	https://learn.microsoft.com/en-us/azure/databricks/archive/compute/libraries-init-scripts
Apply best practices for Databricks compute policies	https://learn.microsoft.com/en-us/azure/databricks/archive/compute/policies-best-practices
Use DBIO for transactional writes to cloud storage in Databricks	https://learn.microsoft.com/en-us/azure/databricks/archive/legacy/dbio-commit
Optimize skewed joins in Databricks using skew hints	https://learn.microsoft.com/en-us/azure/databricks/archive/legacy/skew-join
Migrate from Databricks Deep Learning Pipelines	https://learn.microsoft.com/en-us/azure/databricks/archive/spark-3.x-migration/deep-learning-pipelines
Apply Azure Databricks platform administration best practices	https://learn.microsoft.com/en-us/azure/databricks/cheat-sheet/administration
Optimize BI serving performance on Azure Databricks	https://learn.microsoft.com/en-us/azure/databricks/cheat-sheet/bi-serving
Apply Databricks BI data preparation best practices	https://learn.microsoft.com/en-us/azure/databricks/cheat-sheet/bi-serving-data-prep
Optimize Azure Databricks SQL warehouses for BI	https://learn.microsoft.com/en-us/azure/databricks/cheat-sheet/bi-serving-sql-serving
Apply Azure Databricks compute creation best practices	https://learn.microsoft.com/en-us/azure/databricks/cheat-sheet/compute
Implement Azure Databricks production job scheduling best practices	https://learn.microsoft.com/en-us/azure/databricks/cheat-sheet/jobs
Apply Databricks-specific Power BI performance best practices	https://learn.microsoft.com/en-us/azure/databricks/cheat-sheet/power-bi
Use AI-generated comments for Unity Catalog documentation	https://learn.microsoft.com/en-us/azure/databricks/comments/ai-comments
Apply best practices for Databricks classic compute	https://learn.microsoft.com/en-us/azure/databricks/compute/cluster-config-best-practices
Improve Databricks compute reliability with flexible node types	https://learn.microsoft.com/en-us/azure/databricks/compute/flexible-node-types
Apply best practices for Databricks pools	https://learn.microsoft.com/en-us/azure/databricks/compute/pool-best-practices
Apply best practices for Databricks serverless compute	https://learn.microsoft.com/en-us/azure/databricks/compute/serverless/best-practices
Use Lakehouse Replay to validate runtime upgrades	https://learn.microsoft.com/en-us/azure/databricks/compute/serverless/lakehouse-replay
Tune SQL warehouse settings for BI workloads	https://learn.microsoft.com/en-us/azure/databricks/compute/sql-warehouse/bi-workload-settings
Control large interactive queries with Query Watchdog	https://learn.microsoft.com/en-us/azure/databricks/compute/troubleshooting/query-watchdog
Configure dataset materialization for Databricks dashboards	https://learn.microsoft.com/en-us/azure/databricks/dashboards/materialization
Apply Azure Databricks data engineering best practices	https://learn.microsoft.com/en-us/azure/databricks/data-engineering/best-practices
Implement observability for Databricks streaming workloads	https://learn.microsoft.com/en-us/azure/databricks/data-engineering/observability-best-practices
Best practices for Unity Catalog ABAC policies	https://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/abac/best-practices
Common ABAC row filtering and masking patterns	https://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/abac/common-patterns
Optimize ABAC row filter and mask performance	https://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/abac/performance
Apply Unity Catalog governance best practices	https://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/best-practices
Service policy examples for AI governance	https://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/service-policies/policy-examples
Work with legacy Hive metastore objects in Databricks	https://learn.microsoft.com/en-us/azure/databricks/database-objects/hive-metastore
Safely use and migrate away from DBFS root	https://learn.microsoft.com/en-us/azure/databricks/dbfs/dbfs-root
Apply best practices for DBFS and Unity Catalog	https://learn.microsoft.com/en-us/azure/databricks/dbfs/unity-catalog
Apply Delta Lake best practices on Databricks	https://learn.microsoft.com/en-us/azure/databricks/delta/best-practices
Handle Delta Lake limitations on S3 safely	https://learn.microsoft.com/en-us/azure/databricks/delta/s3-limitations
Use selective overwrite options with Delta Lake on Databricks	https://learn.microsoft.com/en-us/azure/databricks/delta/selective-overwrite
Apply recommended CI/CD workflows on Databricks	https://learn.microsoft.com/en-us/azure/databricks/dev-tools/ci-cd/flows
View Databricks policy families via CLI	https://learn.microsoft.com/en-us/azure/databricks/dev-tools/cli/reference/policy-families-commands
Develop Databricks Apps using supported frameworks and patterns	https://learn.microsoft.com/en-us/azure/databricks/dev-tools/databricks-apps/app-development
Apply best practices for Databricks Apps	https://learn.microsoft.com/en-us/azure/databricks/dev-tools/databricks-apps/best-practices
Add and use model serving endpoints in Databricks Apps	https://learn.microsoft.com/en-us/azure/databricks/dev-tools/databricks-apps/model-serving
Advanced configuration and usage of Databricks Connect	https://learn.microsoft.com/en-us/azure/databricks/dev-tools/databricks-connect/advanced
Test Databricks Connect Python code with pytest	https://learn.microsoft.com/en-us/azure/databricks/dev-tools/databricks-connect/python/testing
Handle async queries and interruptions with Databricks Connect	https://learn.microsoft.com/en-us/azure/databricks/dev-tools/databricks-connect/queries
Test Databricks Connect Scala code with ScalaTest	https://learn.microsoft.com/en-us/azure/databricks/dev-tools/databricks-connect/scala/testing
Call Databricks REST API with performance best practices	https://learn.microsoft.com/en-us/azure/databricks/dev-tools/rest-api
Apply Databricks developer and CI/CD best practices	https://learn.microsoft.com/en-us/azure/databricks/developers/best-practices
Choose between Unity Catalog volumes and workspace files	https://learn.microsoft.com/en-us/azure/databricks/files/files-recommendations
Store and reference Databricks init scripts in workspace files	https://learn.microsoft.com/en-us/azure/databricks/files/workspace-init-scripts
Curate effective Genie Agents for accurate answers	https://learn.microsoft.com/en-us/azure/databricks/genie-agents/best-practices
Apply prompt and context best practices for Genie Code	https://learn.microsoft.com/en-us/azure/databricks/genie-code/tips
Deep clone managed Iceberg tables in Unity Catalog	https://learn.microsoft.com/en-us/azure/databricks/iceberg/clone
Apply Auto Loader best practices for reliable ingestion	https://learn.microsoft.com/en-us/azure/databricks/ingestion/cloud-object-storage/auto-loader/best-practices
Configure Azure Databricks Auto Loader for production	https://learn.microsoft.com/en-us/azure/databricks/ingestion/cloud-object-storage/auto-loader/production
Use automatic type widening in Databricks Auto Loader	https://learn.microsoft.com/en-us/azure/databricks/ingestion/cloud-object-storage/auto-loader/type-widening
Apply common COPY INTO data loading patterns	https://learn.microsoft.com/en-us/azure/databricks/ingestion/cloud-object-storage/copy-into/examples
Celigo connector FAQ and usage guidance	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/celigo-faq
Apply common patterns to Lakeflow ingestion pipelines	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/common-patterns
Apply Confluence connector behaviors and FAQs	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/confluence-faq
CrowdStrike Falcon Event Stream connector FAQ	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/crowdstrike-falcon-event-stream-faq
Apply Glean connector FAQs and usage guidance	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/glean-faq
Use Marketo connector effectively in Lakeflow Connect	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/marketo-faq
Apply Meta Ads connector usage guidance and FAQs	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/meta-ads-faq
MySQL Lakeflow connector FAQ and usage guidance	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/mysql-faq
Use Oracle integrated CDC connector FAQs and tips	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/oracle-faq
Apply PostgreSQL connector FAQs and guidance	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/postgresql-faq
Maintain and operate PostgreSQL Lakeflow pipelines	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/postgresql-maintenance
Use Productboard connector features and behaviors	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/productboard-faq
Reddit Ads connector usage FAQs and behaviors	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/reddit-ads-faq
Configure incremental ingestion of Salesforce formula fields	https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/salesforce-formula-fields
Use Zerobus acknowledgment callbacks effectively	https://learn.microsoft.com/en-us/azure/databricks/ingestion/zerobus-callbacks
Choose Zerobus blocking methods for durability	https://learn.microsoft.com/en-us/azure/databricks/ingestion/zerobus-message-blocking
Implement resilient Zerobus recovery patterns	https://learn.microsoft.com/en-us/azure/databricks/ingestion/zerobus-recovery
Design Zerobus schemas for evolving data	https://learn.microsoft.com/en-us/azure/databricks/ingestion/zerobus-schema-management
Use and configure Databricks cluster init scripts	https://learn.microsoft.com/en-us/azure/databricks/init-scripts/
Reference external files in Databricks init scripts	https://learn.microsoft.com/en-us/azure/databricks/init-scripts/referencing-files
Test applications using the legacy Simba JDBC Driver	https://learn.microsoft.com/en-us/azure/databricks/integrations/jdbc/testing
Test Databricks ODBC driver connections in code	https://learn.microsoft.com/en-us/azure/databricks/integrations/odbc/testing
Schedule recurring SQL queries with backfill in Jobs	https://learn.microsoft.com/en-us/azure/databricks/jobs/how-to/create-recurring-job
Apply classic compute best practices for Jobs	https://learn.microsoft.com/en-us/azure/databricks/jobs/run-classic-jobs
Apply Databricks cost optimization best practices	https://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/cost-optimization/best-practices
Implement best practices for Databricks data and AI governance	https://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/data-governance/best-practices
Design observability and monitoring strategy for Databricks	https://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/deployment-guide/observability
Apply interoperability and usability best practices on Databricks	https://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/interoperability-and-usability

Use this skill

Connect your agent to the Skill Store MCP server and ask for it by ref: microsoftdocs/azure-databricks-94965928255.

Download SKILL.md · Markdown version

Details

Publisher: microsoftdocs
Category: Data & Analytics › Data pipelines & ETL
License: MIT
Updated: 2026-10-19

References

Related skills

Azure Synapse Analytics Expert knowledge for Azure Synapse Analytics development including troubleshooting, best practice…
Azure Machine Learning Expert knowledge for Azure Machine Learning development including troubleshooting, best practices…
Azure Hdinsight Expert knowledge for Azure HDInsight development including troubleshooting, best practices, decis…
Azure Sql Database Expert knowledge for Azure SQL Database development including troubleshooting, best practices, de…
Azure Data Explorer Expert knowledge for Azure Data Explorer development including troubleshooting, best practices, d…