Job Title: Lead Data Engineer

Contract Length: 6-month

Location: Hybrid Vancouver Preference (Mostly remote w/onsite required approx. 1 day/ month). Open to Remote Canada

:

Insight Global is seeking a Lead Data Engineer to join a large retail client based in Vancouver, BC. The data engineering team builds and maintains the data infrastructure that powers analytics, reporting, and data-driven decision making across digital, retail, and guest-facing platforms. This role will be focused on the organization's Guest Data Lake and guest data domain, supporting initiatives involving customer, retail, and PII data.

The ideal candidate will be a hands-on technical leader who can partner closely with Product Managers, Data Engineers, and business stakeholders to drive delivery, refine requirements, groom technical backlogs, and ensure data products align with business needs.

A key focus of this role will be modernizing the data platform through the migration of pipelines from Azure Data Factory to Databricks, the implementation and expansion of Unity Catalog, and the adoption of Medallion Architecture best practices. The successful candidate will bring strong expertise in Databricks, declarative pipelines, data governance, and scalable lakehouse design, with the ability to operate as both a technical contributor and strategic lead.

Updated Day-to-Day Activities

- Serve as a technical lead for the Guest Data Lake, partnering with Product Managers and Engineering teams to define, refine, and groom backlog items and technical requirements
- • Design, build, and optimize scalable data pipelines supporting guest, customer, retail, and PII data domains
- • Provide hands-on engineering leadership, balancing approximately 80% technical execution and 20% leadership, planning, and stakeholder engagement
- • Lead Unity Catalog implementation and expansion efforts to improve governance, security, lineage,



and metadata management
- • Architect and implement Databricks-based data solutions leveraging Medallion Architecture and declarative pipelines
- • Collaborate with cross-functional teams to deliver high-quality data products that support analytics, reporting, and operational use cases
- • Mentor engineers and drive engineering best practices across coding standards, SDLC processes, testing, and deployment strategies
- • Support backlog prioritization, sprint planning, ticket grooming, and technical roadmap development
- • Ensure data quality, governance, security, and compliance standards are met across guest and customer data assets
- • Partner with business and technical stakeholders to understand the guest domain and translate business requirements into scalable technical solutions

Must Haves:

- 8-10+ years of Data Engineering experience, including 3+ years leading complex data initiatives and mentoring Engineers
- • Strong hands-on experience developing production-grade solutions with Azure Databricks, Delta Lake, Spark, Python, PySpark, Spark SQL, and SQL
- • Hands-on Unity Catalog implementation experience, including catalogs, schemas, lineage, permissions, RBAC, and fine-grained data access controls
- • Deep understanding of Medallion Architecture and the design of governed Bronze, Silver, and Gold data layers
- • Experience building declarative pipelines using Delta Live Tables or Lakeflow Declarative Pipelines, as applicable to the client’s environment




- • Experience with Databricks Asset Bundles and automated deployment of Databricks pipelines and resources across environments
- • Strong experience with infrastructure as code and CI/CD using Terraform, Git, and Azure DevOps, GitHub Actions, Jenkins, or comparable tooling
- • Experience building reusable data engineering utilities and frameworks for ingestion, transformation, validation, and data quality
- • Experience ingesting and processing structured, semi-structured, batch, and streaming data, ideally using Auto Loader or comparable ingestion frameworks
- • Strong Spark performance-tuning experience, including partitioning, cluster and job configuration, memory optimization, and query tuning
- • Demonstrated experience supporting Agile delivery through ticket grooming, requirement analysis, technical planning, and partnership with Product Managers and Engineers

Plusses:

- Data Mesh or domain-oriented data product experience

• Experience working with guest, customer, retail, consumer, loyalty, identity, or other PII-intensive data domains
- DBT experience, including reusable models, testing, macros, and integration with Databricks workflows
- Real-time or event-driven data pipeline experience
- Data observability, lineage, and automated data-quality monitoring experience
- MLOps experience with Databricks and MLflow
- Retail, ecommerce, customer identity, loyalty, personalization, or guest analytics experience
- Databricks Data Engineer Skilled or Microsoft Azure Data Engineer certification
- AWS or multi-cloud experience

Vacancy Status:

This posting is for a currently vacant role, and the successful candidate will be hired into an existing open position.

Please Note: We may use artificial intelligence tools to assist with the screening, assessment, or selection of potential applicants for this position.

📌 Lead Data Engineer (Vancouver)
🏢 Insight Global
📍 Vancouver