Principal Data Engineer – Databricks | Spark | Delta Lake | PySpark | Data Lakehouse | AWS/Azure Location: Toronto
Work Model: Onsite (4 days/week)
Key Requirements 12–18 years of overall Data Engineering experience.
8+ years of experience with Enterprise Data Warehouse and Data Lake platforms.
5+ years of hands‑on experience with Databricks and Apache Spark at scale.
Strong experience modernizing legacy Cloudera platforms (CDH/CDP, Hive, HBase, Impala, Spark) to Databricks Lakehouse.
Experience redesigning ingestion, transformation, and consumption patterns from HDFS-based architecture to cloud object storage and Delta Lake.
Experience refactoring legacy Hive/Impala logic into PySpark and Spark SQL ELT pipelines.
Experience ensuring data reconciliation, audit integrity, and consistency during migration.
Experience designing and governing Enterprise Data Warehouse and Data Lake/Lakehouse architectures.
Experience implementing layered architectures including: Raw/Landing Layer
Curated/Conformed Layer
Semantic/Consumption Layer

Experience modernizing traditional Enterprise Data Warehouse platforms into scalable Lakehouse architectures.
Solid experience with finance and risk data models, including: General Ledger
Sub‑ledger
Financial Hierarchies
Credit Risk Models
Liquidity Risk Models
Market Risk Models

Experience enabling reporting use cases including aggregation, drill‑down, and drill‑back capabilities.
Experience building and managing semantic/consumption layers for BI, reporting, and analytics.
Ability to define business metrics, dimensions, hierarchies, and KPIs.
Experience with Databricks SQL, Delta Tables, and dbt or similar frameworks.
Strong experience developing and optimizing large‑scale data pipelines using: PySpark




Spark SQL
Delta Lake

Experience implementing Medallion Architecture: Bronze Layer
Silver Layer
Gold Layer

Experience optimizing workloads using Z‑ORDER, OPTIMIZE, caching, and cluster configurations.
Experience implementing data governance, data quality frameworks, reconciliation controls, and exception handling.
Experience establishing data lineage and metadata management.
Knowledge of data security, access control, and compliance standards.
Experience with cloud platforms such as AWS or Azure.
Experience with CI/CD pipelines using: Git
Terraform
Jenkins
Azure DevOps

Familiarity with orchestration tools such as: Apache Airflow
Databricks Workflows

Experience with dbt is a plus.
Ability to act as a technical authority and lead architecture decisions.
Experience mentoring senior engineers and establishing engineering standards.
Strong stakeholder management skills with finance, risk, analytics, and governance teams.
Ability to translate complex data structures into business‑ready insights.
Nice to Have Experience in Banking, Financial Services, Insurance (BFSI), Capital Markets, or regulatory reporting.
Exposure to: SAP Finance
Oracle Financials
SAP S/4HANA

Experience supporting AI/ML workloads.
Databricks or cloud certifications.
Key Responsibilities Lead Cloudera to Databricks transformation initiatives.
Design and implement enterprise Data Lakehouse and Data Warehouse solutions.
Build scalable, high‑performance data pipelines and modern data architectures.
Drive data modernization, governance, quality, and security initiatives.
Support regulatory, management, and analytical reporting platforms.
Provide technical leadership, mentor engineering teams, and establish best practices.

#J-18808-Ljbffr

📌 Principal Data Engineer – Databricks (Ontario)
🏢 Astra-North Infoteck Inc. ~ Conquering today’s challenges, achieving tomorrow’s vision!
📍 Ontario