Job Title: Senior Databricks Data Platform Engineer

Location: Canada, Remote

Seniority on the skill/s required on this requirement: Senior Level

Estimated Duration: 1 year contract with possible extension

Work authorization: Candidate must be authorized to work in Canada.

Job Summary

We are looking for a highly experienced Senior Databricks Data Platform Engineer to own the engineering, automation, administration, and optimization of an enterprise Databricks Lakehouse platform. The ideal candidate should have strong hands‑on experience in Databricks platform administration, Unity Catalog, Terraform, CI/CD automation, platform cost optimization (FinOps), Delta Lake, and cloud infrastructure.

This is a platform engineering role rather than a traditional data pipeline development position. The candidate will be responsible for building a secure, scalable, reliable, and cost‑effective Databricks platform while enabling engineering and data science teams.

Key Responsibilities

- Databricks Platform Engineering & Administration
- Own Databricks environments across development, testing, and production.
- Administer Unity Catalog, including metastores, catalogs, schemas, external locations, and access controls.
- Manage compute resources, cluster policies, SQL warehouses, and serverless compute.
- Manage Databricks Runtime upgrades and platform configurations.
- Develop reusable platform templates and self-service capabilities.
- CI/CD & Platform Automation
- Build CI/CD pipelines using Databricks Asset Bundles, Git folders, Azure DevOps, or GitHub Actions.
- Implement Terraform for Databricks and cloud infrastructure.
- Automate testing, security scanning,



and deployment workflows.
- Establish release, branching, and rollback standards.
- Automate SCIM provisioning, secret management, token rotation, and environment drift detection.
- Agentic Workflows & AI-Enabled Operations
- Develop AI-driven workflows for job failure analysis, root-cause investigation, and cost anomaly detection.
- Build AI agents using Mosaic AI Agent Framework, Model Serving, and Vector Search.
- Integrate enterprise systems through Model Context Protocol (MCP).
- Implement AI governance, observability, evaluation, and human-in-the-loop controls.
- Support GenAI, RAG, and ML workloads.
- Platform Cost Optimization (FinOps)
- Monitor Databricks usage and costs using system tables.
- Optimize cluster sizing, autoscaling, auto-termination, Photon, and serverless compute.
- Implement cost governance, budget alerts, and tagging standards.
- Develop cost dashboards and anomaly detection alerts.
- Support DBU forecasting and capacity planning.
- Storage & Performance Optimization
- Implement Delta Lake storage optimization standards.
- Manage OPTIMIZE, VACUUM, liquid clustering, Z-ordering, and partitioning.
- Optimize ADLS Gen2 / S3 storage and retention policies.
- Tune Spark and SQL workloads.
- Identify and resolve small-file issues,



redundant data, and storage inefficiencies.
- Governance, Security & Compliance
- Implement secure platform controls and least-privilege access.
- Manage Unity Catalog governance, lineage, and audit logging.
- Implement data masking, classification, and row/column-level security.
- Partner with security and compliance teams.
- Reliability & Operational Excellence
- Define platform SLOs and monitoring standards.
- Lead incident management and root-cause analysis.
- Develop operational runbooks and platform documentation.
- Mentor engineers and promote Databricks best practices.

Required Qualifications

- 7+ years of experience in data engineering, data platform, or DevOps roles.
- 3+ years of enterprise Databricks administration and engineering experience.
- Strong expertise in Unity Catalog, workspace administration, cluster policies, and Databricks Workflows.
- Hands‑on experience with Databricks Asset Bundles and Terraform.
- Experience building CI/CD pipelines using Azure DevOps, GitHub Actions, or similar tools.
- Demonstrated experience in Databricks cost optimization and FinOps.
- Strong knowledge of Delta Lake internals and storage optimization.
- Strong programming skills in Python and SQL, with working knowledge of PySpark.
- Experience with Azure, AWS, or GCP.
- Experience implementing security and governance for sensitive data.

Preferred Qualifications

- Experience with Agentic AI and GenAI frameworks such as Mosaic AI, LangChain, and LangGraph.
- Practical knowledge of MCP.
- Familiarity with GitHub Copilot and Anthropic models.

📌 Databricks Data Platform Engineer (Canada)
🏢 Mindtris
📍 Canada