Job Title: Senior Databricks Data Platform Engineer

Location: Canada, Remote

Seniority on the skill/s required on this requirement: Senior Level

Estimated Duration: 1 year contract with possible extension

Work authorization: Candidate must be authorized to work in Canada.

Job Summary We are looking for a highly experienced Senior Databricks Data Platform Engineer to own the engineering, automation, administration, and optimization of an enterprise Databricks Lakehouse platform. The ideal candidate should have strong hands‑on experience in Databricks platform administration, Unity Catalog, Terraform, CI/CD automation, platform cost optimization (FinOps), Delta Lake, and cloud infrastructure.

This is a platform engineering role rather than a traditional data pipeline development position. The candidate will be responsible for building a secure, scalable, reliable, and cost‑efficient Databricks platform while enabling engineering and data science teams.

Key Responsibilities

Databricks Platform Engineering & Administration

Own Databricks environments across development, testing, and production.

Administer Unity Catalog, including metastores, catalogs, schemas, external locations, and access controls.

Manage compute resources, cluster policies, SQL warehouses, and serverless compute.

Manage Databricks Runtime upgrades and platform configurations.

Develop reusable platform templates and self-service capabilities.

CI/CD & Platform Automation

Build CI/CD pipelines using Databricks Asset Bundles, Git folders, Azure DevOps, or GitHub Actions.

Implement Terraform for Databricks and cloud infrastructure.

Automate testing, security scanning,



and deployment workflows.

Establish release, branching, and rollback standards.

Automate SCIM provisioning, secret management, token rotation, and environment drift detection.

Agentic Workflows & AI-Enabled Operations

Develop AI-driven workflows for job failure analysis, root-cause investigation, and cost anomaly detection.

Build AI agents using Mosaic AI Agent Framework, Model Serving, and Vector Search.

Integrate enterprise systems through Model Context Protocol (MCP).

Implement AI governance, observability, evaluation, and human-in-the-loop controls.

Support GenAI, RAG, and ML workloads.

Platform Cost Optimization (FinOps)

Monitor Databricks usage and costs using system tables.

Optimize cluster sizing, autoscaling, auto-termination, Photon, and serverless compute.

Implement cost governance, budget alerts, and tagging standards.

Develop cost dashboards and anomaly detection alerts.

Support DBU forecasting and capacity planning.

Storage & Performance Optimization

Implement Delta Lake storage optimization standards.

Manage OPTIMIZE, VACUUM, liquid clustering, Z-ordering, and partitioning.

Optimize ADLS Gen2 / S3 storage and retention policies.

Tune Spark and SQL workloads.

Identify and resolve small-file issues, redundant data,



and storage inefficiencies.

Governance, Security & Compliance

Implement secure platform controls and least-privilege access.

Manage Unity Catalog governance, lineage, and audit logging.

Implement data masking, classification, and row/column-level security.

Partner with security and compliance teams.

Reliability & Operational Excellence

Define platform SLOs and monitoring standards.

Lead incident management and root-cause analysis.

Develop operational runbooks and platform documentation.

Mentor engineers and promote Databricks best practices.

Required Qualifications

7+ years of experience in data engineering, data platform, or DevOps roles.

3+ years of enterprise Databricks administration and engineering experience.

Robust expertise in Unity Catalog, workspace administration, cluster policies, and Databricks Workflows.

Hands‑on experience with Databricks Asset Bundles and Terraform.

Experience building CI/CD pipelines using Azure DevOps, GitHub Actions, or similar tools.

Demonstrated experience in Databricks cost optimization and FinOps.

Strong knowledge of Delta Lake internals and storage optimization.

Strong programming skills in Python and SQL, with working knowledge of PySpark.

Experience with Azure, AWS, or GCP.

Experience implementing security and governance for sensitive data.

Preferred Qualifications

Experience with Agentic AI and GenAI frameworks such as Mosaic AI, LangChain, and LangGraph.

Practical knowledge of MCP.

Familiarity with GitHub Copilot and Anthropic models.

Databricks Data Engineer

📌 Databricks Data Platform Engineer (Winnipeg)
🏢 Mindtris
📍 Winnipeg