Looking for: Senior / Lead Data Engineer - Databricks, AI Engineering & Retail & Omnichannel Analytics

Job Type: Contract

Location: Remote (Canada)

Description:

About the Role

We are seeking a Senior / Lead Data Engineer to help build the next generation of our Retail & Omnichannel Analytics platform.

This role combines hands-on engineering, technical leadership, and architecture. You will help define scalable data ingestion patterns, establish Databricks best practices, develop reusable frameworks, enable self-service analytics, and drive AI-enabled engineering practices across the organization.

You will work across the full analytics lifecycle, from data ingestion and transformation through semantic models, reporting, governance, and AI-powered data products. The ideal candidate has successfully implemented enterprise-scale data and analytics platforms using Databricks and can bring proven patterns and best practices from previous initiatives.

Key Responsibilities

- Design and implement scalable data ingestion, transformation, and data product frameworks.
- Define and drive adoption of Databricks best practices, including Medallion Architecture (Bronze/Silver/Gold), performance optimization, governance, and operational excellence.
- Build batch, near real-time, and streaming data pipelines using Databricks and Azure technologies.
- Design and deliver end-to-end data products supporting Retail & Omnichannel Analytics use cases.
- Develop trusted analytical datasets, semantic models, and governed consumption layers for reporting and self-service analytics.




- Enable data democratization through capabilities such as Databricks Genie and reusable business-ready data products.
- Lead proof-of-concepts and evaluate emerging platform capabilities across the Databricks ecosystem.
- Define AI SDLC and AI-assisted development patterns, including the use of AI agents and engineering accelerators.
- Implement data quality, lineage, monitoring, and observability practices.
- Partner with engineering, analytics, product, and business teams to deliver scalable solutions and mentor engineers on best practices.

Required Qualifications

- 7+ years of Data Engineering experience.
- Strong hands-on experience with Databricks in enterprise production environments.

• Advanced expertise in:

- Python
- SQL
- Scala
- Spark / PySpark
- Delta Lake

• Experience building Lakehouse architectures and implementing Medallion Architecture (Bronze/Silver/Gold).
- Strong experience with ETL/ELT design, data integration, and scalable data pipelines.
- Experience creating analytical and dimensional data models.
- Experience delivering end-to-end analytics solutions from ingestion through semantic modeling and reporting.
- Experience implementing CI/CD and Data Engineering SDLC best practices.
- Experience with Azure DevOps, Git, and release management processes.




- Strong understanding of data governance, security, data quality, and performance optimization.
- Solid communication skills and ability to lead technical discussions.

Preferred Qualifications

- Experience with Unity Catalog, LakeFlow, Delta Live Tables, Databricks SQL, and Databricks Genie.
- Experience establishing engineering standards, reusable frameworks, or platform best practices used by multiple teams.
- Experience implementing AI-assisted development workflows and AI engineering practices.
- Experience designing AI agents or automation solutions that improve engineering productivity.
- Experience leading platform modernization initiatives and technical proof-of-concepts.
- Experience in Retail & Omnichannel Analytics, Merchandising, Inventory, Supply Chain, Store Operations, Customer Analytics, or Digital Commerce.

Tech Stack:

Languages

- Python
- SQL
- Scala

Data Engineering

- Spark
- PySpark
- Delta Lake

Data Platform

- Databricks
- Unity Catalog
- LakeFlow
- Delta Live Tables (DLT)
- Databricks SQL
- Databricks Genie
- Databricks Workflows

Data Integration & Orchestration

- Azure Data Factory (ADF)
- Databricks Workflows
- REST APIs
- Batch & Streaming Pipelines

DevOps

- Azure DevOps
- Git
- CI/CD Pipelines

Architecture & Governance

- Lakehouse Architecture
- Medallion Architecture
- Data Products
- Semantic Models
- Data Quality
- Data Lineage
- Observability & Monitoring

AI Engineering

- AI-Assisted Development Tools
- AI Agents
- GenAI-enabled Engineering Workflows

📌 Senior / Lead Data Engineer - Databricks, AI Engineering & Retail & Omnichannel Analytics (Canada)
🏢 Diligente Technologies
📍 Canada