Develop enterprise ETL pipelines with Databricks, Azure Data Factory, and PySpark in Toronto. Transform data from diverse systems into curated downstream feeds.
Apex Systems seeks a Databricks Data Engineer for a large-scale data migration and centralization initiative. The assignment emphasizes data ingestion, transformation, and pipeline maintenance across medallion architecture layers. You will work with SQL Server, Db2, structured feeds, card networks, and file-based sources while supporting data consumers.
Key Responsibilities: Develop ETL and ELT ingestion pipelines
Transform data using SQL, Python, and PySpark
Build materialized and metric views
Maintain Bronze, Silver, and Gold datasets
Resolve pipeline, data, and transformation problems
Requirements: Robust Databricks data engineering background
Experience maintaining ETL and ingestion pipelines
Hands-on Python, PySpark, and SQL skills
Azure Data Factory pipeline experience
Understanding of Databricks tables and workflows
Use Databricks ETL, ADF, SQL, and PySpark skills to deliver migration-focused data pipelines for Apex Systems in Toronto.