Databricks Data Engineer with Azure, PySpark, Kafka and end-to-end automation of data engineering pipelines.

Self healing pipeline, streaming pipelines, automating pipelines and Databricks Certified

Description:

We are seeking a highly skilled Databricks Data Engineer to support the Retail Spine initiative, working with a wide range of retail datasets to design, build, and optimize scalable data solutions. This role requires deep hands-on expertise in Databricks and the ability to contribute immediately by developing new pipelines while enhancing existing data workflows.

The ideal candidate brings extensive experience building end-to-end data engineering pipelines with CDC (Change Data Capture) capabilities, and thrives in a fast-paced Azure-based environment leveraging Databricks, PySpark, and Kafka streaming.

Key Responsibilities

- Design, build, and optimize end-to-end ETL/ELT pipelines using Databricks and Apache Spark (PySpark).
- Develop and maintain scalable data solutions on Microsoft Azure, leveraging Databricks, Delta Lake, and Kafka streaming.
- Build robust pipelines with Change Data Capture (CDC) to support incremental data processing and real-time/near real-time data needs.
- Work across both new pipeline development and enhancement of existing pipelines, quickly understanding current codebases to deliver improvements.
- Integrate and process data from diverse sources including:
- Flat files, XML, JSON, SQL Server and JDBC systems, Kafka topics, APIs, MQ, Unity Catalog volumes and other enterprise platforms.




- Implement data quality frameworks, including writing technical data quality rules and quarantining error records.
- Apply canonical data modeling techniques, including SCD Type 1 and Type 2 implementations.
- Ensure high levels of data reliability, performance, scalability, and monitoring across all workflows.
- Automate workflows using Databricks Workflows and CI/CD pipelines (GitHub), focusing on reuse, efficiency, and standardization.
- Adhere to ADUSA architecture, governance, and security standards.
- Collaborate with developers, architects, and product owners, and engage with stakeholders, source teams, and downstream consumers as needed.
- Participate actively in Agile/Scrum ceremonies and deliver high-quality solutions within defined timelines.

Required Qualifications

- Extensive hands-on experience with Databricks, with ability to contribute from day one.
- Proven experience building end-to-end data engineering pipelines with CDC capabilities.
- Strong proficiency in PySpark and SQL (required); Python experience is a plus.
- Experience working in Microsoft Azure cloud environments.
- Knowledge of:
- Distributed data processing frameworks
- Streaming technologies (Kafka)
- Data modeling (SCD Type 1 & Type 2)
- Experience integrating data from multiple structured and unstructured data sources.
- Familiarity with GitHub and CI/CD pipelines for version control and deployment.
- Robust understanding of data architecture, performance optimization, and pipeline scalability.

📌 Data Bricks Engineer (Canada)
🏢 V-Soft Consulting Group
📍 Canada