As a Senior Databricks Data Platform Engineer, you will own the engineering, automation, and day-to-day health of our enterprise Databricks Lakehouse platform. This is a platform-first role: rather than building individual business pipelines, you will build the foundation that every data engineer, analyst, and data scientist depends on, including workspaces, compute, Unity Catalog governance, deployment pipelines, and the guardrails that keep the platform secure, reliable, and cost-efficient.

You will treat the platform as a product. That means codifying infrastructure and configuration, delivering changes through fully automated CI/CD, introducing agentic AI workflows that reduce manual operational effort, and continuously driving down compute and storage cost while improving performance. You will combine deep hands-on Databricks expertise with a FinOps and DevOps mindset, and you will measure your success in platform reliability, developer productivity, and dollars saved.

This is a highly collaborative role. You will partner with data engineering, data science, security, cloud infrastructure, and finance teams to set platform standards, guide adoption of new Databricks capabilities, and mentor engineers across the organization.

About the role

Own the architecture, configuration, and lifecycle of client's Databricks environments across development, test, and production, including workspaces, account-level settings, networking, and identity integration

Administer and evolve Unity Catalog, including metastores, catalogs, schemas, external locations, storage credentials, and fine-grained access controls

Design and enforce compute strategy across all-purpose clusters, job clusters, serverless compute, SQL warehouses, instance pools, and cluster policies

Manage platform upgrades, Databricks Runtime adoption, feature previews, and deprecations with minimal disruption to consuming teams

Build reusable platform templates, starter projects, and self-service capabilities that let engineering teams onboard quickly and consistently

CI/CD & Platform Automation

Design, build, and maintain CI/CD pipelines for Databricks code and configuration using Databricks Asset Bundles, Git folders, and Azure DevOps or GitHub Actions

Codify platform infrastructure with Terraform (Databricks and cloud providers), ensuring every workspace, cluster policy, permission, and Unity Catalog object is version-controlled and reproducible

Implement automated testing, code quality gates, security scanning, and promotion workflows from development through production

Establish branching, release, and rollback standards for notebooks, jobs, Delta Live Tables / Lakeflow pipelines, and ML assets

Automate routine administration such as user and group provisioning (SCIM), secret management, token rotation, and environment drift detection

Agentic Workflows & AI-Enabled Operations

Design and deploy agentic workflows that automate platform operations, such as job failure triage, root-cause analysis, cost anomaly investigation, and remediation recommendations

Build AI agents and tools on Databricks (Mosaic AI Agent Framework, Model Serving, Vector Search) and integrate them with enterprise systems through Model Context Protocol (MCP)





Use Copilot and Anthropic models to accelerate infrastructure-as-code, pipeline development, code review, and documentation

Establish guardrails, evaluation, observability, and human-in-the-loop controls so agentic automation is safe, auditable, and trusted

Partner with data science and engineering teams to provide a governed platform foundation for GenAI, RAG, and ML workloads

Platform Cost Optimization (FinOps)

Own Databricks cost visibility and accountability using system tables (billing, usage, compute, and query history), tagging standards, and chargeback or showback reporting

Identify and deliver savings through cluster right-sizing, auto-termination, autoscaling tuning, spot and reserved capacity strategies, Photon and serverless evaluation, and workload consolidation

Enforce cost guardrails through cluster policies, budget alerts, and compute governance that prevent waste without slowing teams down

Build dashboards and automated alerts that surface cost anomalies, idle resources, and optimization opportunities to engineering and finance stakeholders

Partner with finance and procurement on DBU commitments, forecasting, and capacity planning

Storage & Performance Optimization

Define and enforce Delta Lake storage standards, including file sizing, liquid clustering, Z-ordering, partitioning strategy, and deletion vectors

Automate table maintenance using predictive optimization, OPTIMIZE, VACUUM, and retention policies to control storage growth and improve query performance

Manage cloud object storage (ADLS Gen2 / S3) layout, lifecycle and tiering policies, and data retention aligned with business and compliance requirements

Tune Spark and SQL workloads by analyzing query profiles, shuffle, skew, caching, and Photon usage to improve speed and reduce cost

Identify and remediate small-file problems, orphaned data, unused tables, and redundant copies across the Lakehouse

Governance, Security & Compliance

Implement secure-by-default platform controls, including private networking, IP access lists, encryption, secrets management, and least-privilege access

Ensure auditable data access and lineage through Unity Catalog, audit logs, and system tables, with appropriate handling of PII and regulatory requirements

Apply data classification, row- and column-level security, and data masking standards across catalogs

Partner with security and compliance teams on platform reviews, vulnerability remediation, and audit evidence

Reliability & Operational Excellence

Define platform SLOs and build monitoring, alerting, and observability for jobs, clusters, warehouses, and pipelines

Lead incident response, root-cause analysis, and post-incident improvements for platform issues, and participate in on-call rotations

Create clear runbooks, standards, and platform documentation that enable self-service and reduce support load





Continuously evaluate new Databricks and cloud capabilities and lead proofs of concept that improve the platform

Mentor engineers, run enablement sessions, and champion platform best practices across McAfee

About you You are a hands-on platform engineer who thinks in terms of automation, reliability, and unit economics. You enjoy building the systems that make other engineers productive, you are not satisfied until a manual process is codified, and you can explain the cost and performance impact of a platform decision to both engineers and business leaders.

Required Qualifications

7+ years of experience in data engineering, data platform, or DevOps roles, including 3+ years administering and engineering Databricks in an enterprise environment

Deep expertise in Databricks platform administration, including Unity Catalog, workspace and account configuration, cluster policies, SQL warehouses, and Databricks Workflows

Proven experience building CI/CD for data platforms using Databricks Asset Bundles, Terraform, and Azure DevOps, GitHub Actions, or a similar tool

Demonstrated track record of reducing platform cost through compute right-sizing, policy enforcement, and usage analytics, with measurable results

Robust knowledge of Delta Lake internals and storage optimization, including file compaction, clustering, partitioning, and table maintenance

Strong programming skills in Python and SQL, with working knowledge of PySpark and Spark performance tuning

Hands-on experience with at least one major cloud platform (Azure, AWS, or GCP), including networking, identity, and object storage

Experience securing platforms that handle sensitive data, including PII, with attention to compliance and governance

Preferred Qualifications

Experience building agentic or GenAI workflows using Mosaic AI, LangChain, LangGraph, or similar frameworks

Practical knowledge of Model Context Protocol (MCP) to connect enterprise applications and automate platform workflows

Familiarity with Copilot and Anthropic models to enhance development speed, code quality, and documentation

Databricks certifications such as Data Engineer Professional or Platform Administrator

Experience with Databricks system tables, Lakehouse Monitoring, and building FinOps dashboards

Familiarity with Lakeflow / Delta Live Tables, Databricks SQL, Model Serving, and MLOps practices

Knowledge of containerization and orchestration technologies (Docker, Kubernetes)

Experience migrating workloads from legacy platforms (Hadoop, Snowflake, or on-premises warehouses) to Databricks

Personal Attributes

Automation-first mindset, with a habit of eliminating toil and codifying repeatable work

Cost-conscious and data-driven, consistently tying platform decisions to measurable business value

Strong ownership and accountability for platform reliability and the experience of the teams who depend on it

Excellent communication skills, with the ability to influence standards across technical and non-technical stakeholders

Proactive problem-solver with strong analytical and troubleshooting skills

Passionate about emerging technologies, especially in AI, automation, and the Lakehouse ecosystem

Committed to mentoring and knowledge-sharing that elevates the wider engineering team

#J-18808-Ljbffr

📌 Databricks Data Platform Engineer (Winnipeg)
🏢 UnivEdge Consulting
📍 Winnipeg