Job Description Principal Data Engineer Databricks | Spark | Delta Lake | PySpark | Data Lakehouse | AWS/Azure Job Description Location: Toronto Work Model: Onsite (4 days/week) Key Requirements 1218 years of overall Data Engineering experience. 8+ years of experience with Enterprise Data Warehouse and Data Lake platforms. 5+ years of hands-on experience with Databricks and Apache Spark at scale.

Strong experience modernizing legacy Cloudera platforms (CDH/CDP, Hive, HBase, Impala, Spark) to Databricks Lakehouse.

Experience redesigning ingestion, transformation, and consumption patterns from HDFS-based architecture to cloud object storage and Delta Lake.

Experience refactoring legacy Hive/Impala logic into PySpark and Spark SQL ELT pipelines.

Experience ensuring data reconciliation, audit integrity, and consistency during migration.

Experience designing and governing Enterprise Data Warehouse and Data Lake/Lakehouse architectures.

Experience implementing layered architectures including: Raw/Landing Layer Curated/Conformed Layer Semantic/Consumption Layer Experience modernizing traditional Enterprise Data Warehouse platforms into scalable Lakehouse architectures.

Strong experience with finance and risk data models, including: General Ledger Sub-ledger Financial Hierarchies Credit Risk Models Liquidity Risk Models Market Risk Models Experience enabling reporting use cases including aggregation, drill-down, and drill-back capabilities.

Experience building and managing semantic/consumption layers for BI, reporting, and analytics.

Ability to define business metrics, dimensions, hierarchies, and KPIs.

Experience with Databricks SQL, Delta Tables, and dbt or similar frameworks.

Robust experience developing and optimizing large-scale data pipelines using:



PySpark Spark SQL Delta Lake Experience implementing Medallion Architecture: Bronze Layer Silver Layer Gold Layer Experience optimizing workloads using Z-ORDER, OPTIMIZE, caching, and cluster configurations.

Experience implementing data governance, data quality frameworks, reconciliation controls, and exception handling.

Experience establishing data lineage and metadata management.

Knowledge of data security, access control, and compliance standards.

Experience with cloud platforms such as AWS or Azure.

Experience with CI/CD pipelines using: Git Terraform Jenkins Azure Dev

Ops Familiarity with orchestration tools such as: Apache Airflow Databricks Workflows Experience with dbt is a plus.

Ability to act as a technical authority and lead architecture decisions.

Experience mentoring senior engineers and establishing engineering standards.

Strong stakeholder management skills with finance, risk, analytics, and governance teams.

Ability to translate complex data structures into business-ready insights.

Nice to Have Experience in Banking, Financial Services, Insurance (BFSI), Capital Markets, or regulatory reporting.

Exposure to: SAP Finance Oracle Financials SAP S/4HANA Experience supporting AI/ML workloads.

Databricks or cloud certifications.

Key Responsibilities Lead Cloudera to Databricks transformation initiatives.

Design and implement enterprise Data Lakehouse and Data Warehouse solutions.

Build scalable, high-performance data pipelines and modern data architectures.

Drive data modernization, governance, quality, and security initiatives.

Support regulatory, management, and analytical reporting platforms.

Provide technical leadership, mentor engineering teams, and establish best practices.

Requirements Sailpoint

📌 Principal Data Engineer – Databricks (Toronto)
🏢 Astra North Infoteck
📍 Toronto