Experience: 5+ Years

Primary Skills: Azure Databricks, PySpark, SQL, ETL/Data Engineering, Azure Data Factory, Streaming

Cloud Platform: Microsoft Azure

Role Summary

We are looking for an experienced Azure Data Engineer to design, develop, and support scalable data ingestion and processing solutions on the Microsoft Azure platform. The candidate will be responsible for building reusable data engineering frameworks, integrating end-to-end data pipelines, supporting batch and streaming workloads, and ensuring data quality, reliability, and consistency across the data platform.

The role also requires strong troubleshooting capabilities and experience supporting production applications across L1, L2, and L3 support activities.

Key Responsibilities

Design, develop, and implement data ingestion pipelines from multiple structured and unstructured data sources using Azure Databricks.

Develop scalable, configurable, and reusable data ingestion frameworks that can support multiple datasets and source systems.

Build and maintain end-to-end data pipelines, moving data from source systems through transformation layers into target data repositories.

Develop data transformation and processing solutions using PySpark, Spark SQL, and SQL.

Implement data validation, reconciliation, error handling, audit logging, and data-quality controls to maintain accuracy and consistency throughout the pipeline.

Design and develop both batch and streaming data pipelines.

Work with event-based and streaming technologies such as Apache Kafka, Azure Event Hubs, or Spark Structured Streaming, depending on project architecture.





Develop and orchestrate data pipelines using Azure Data Factory (ADF) and integrate them with Azure Databricks.

Work with Azure data services such as Azure Data Lake Storage Gen2 (ADLS Gen2) and related target repositories.

Optimize Databricks/Spark workloads for performance, scalability, and reliability.

Collaborate with cross-functional teams responsible for components such as APIs, search capabilities, source applications, and downstream consuming systems.

Evaluate appropriate data engineering tools and technologies against functional, technical, performance, and customer requirements.

Troubleshoot data pipeline failures and production issues, perform root-cause analysis, and implement permanent fixes.

Participate in L1, L2, and L3 production support, including operational monitoring, incident resolution, defect fixing, and problem management.

Maintain technical documentation, operational procedures, and data pipeline specifications.

Communicate effectively with technical teams, business stakeholders, and customer teams.

Mandatory Technical Skills

Strong hands-on experience with Azure Databricks

Strong knowledge of Apache Spark / PySpark

Advanced SQL skills

Experience with Azure Data Factory (ADF)





Experience building ETL/ELT and data ingestion pipelines

Experience with ADLS Gen2 / Azure Storage

Robust understanding of Delta Lake / Delta tables

Experience developing reusable/configuration-driven data engineering frameworks

Knowledge of batch and real-time/streaming data processing

Experience with streaming technologies such as Kafka, Azure Event Hubs, or Spark Structured Streaming

Good understanding of data warehousing and data lake concepts

Experience implementing data-quality, reconciliation, logging, auditing, and exception-handling mechanisms

Experience troubleshooting production data pipelines

Experience with L1/L2/L3 application or data-platform support

Good to Have

Databricks Workflows / Jobs

Delta Live Tables / Lakeflow Declarative Pipelines

Databricks Unity Catalog

CI/CD experience with Azure DevOps or Git

Python

Azure Key Vault and secrets management

REST API integration

Experience With JSON, CSV, Parquet, And Other Data Formats Data migration or modernization experience

Understanding of Medallion Architecture: Bronze, Silver, Gold

Monitoring, alerting, and operational support experience

Soft Skills

Strong verbal and written communication skills

Strong analytical and problem-solving abilities

Ability to independently troubleshoot complex data issues

Comfortable working with distributed and cross-functional project teams

Ability to communicate technical issues clearly to both technical and non-technical stakeholders

Strong ownership mindset for production systems

📌 Azure Data Engineer (Toronto)
🏢 Ju0026M Group
📍 Toronto