Job Description Principal Databricks Data Engineer Experience Required: 1218 Years Key Requirements 1218 years of overall Data Engineering experience 8+ years of experience with enterprise Data Warehouse and Data Lake platforms 5+ years of hands-on experience with Databricks and Apache Spark at scale Robust experience modernizing legacy Cloudera platforms including: CDH/CDP Hive HBase Impala Spark Modernize Cloudera platforms to Databricks Lakehouse architecture Redesign ingestion, transformation, and consumption patterns from HDFS-based architecture to Cloud Object Storage and Delta Lake Refactor legacy Hive and Impala logic into PySpark and Spark SQL ELT pipelines Ensure data reconciliation, audit integrity, and consistency during migration Design and govern enterprise Data Warehouse and Data Lake/Lakehouse architectures Implement layered data architecture including: Raw / Landing Layer Curated / Conformed Layer Semantic / Consumption Layer Modernize traditional Enterprise Data Warehouse platforms into scalable Lakehouse architectures Strong experience with finance and risk data models including: General Ledger Sub-ledger Financial Hierarchies Credit Risk Liquidity Risk Market Risk Enable reporting capabilities including: Aggregation Drill-down Drill-back Build and manage semantic and consumption layers for BI, reporting, and analytics Define business metrics, dimensions, hierarchies, and KPIs Experience with: Databricks SQL Delta Tables dbt or similar frameworks Develop and optimize large-scale data pipelines using: PySpark Spark SQL Delta Lake Implement Medallion Architecture including:



Bronze Layer Silver Layer Gold Layer Optimize workloads using: Z-ORDER OPTIMIZE Caching Cluster Configuration Tuning Implement: Data Governance Data Quality Frameworks Reconciliation Controls Exception Handling Establish data lineage and metadata management Ensure data security, access control, and compliance standards Experience with AWS or Azure cloud platforms Experience with CI/CD pipelines using: Git Terraform Jenkins Azure Dev

Ops Familiarity with: Apache Airflow Databricks Workflows Experience with dbt is an advantage Act as a technical authority and lead enterprise architecture decisions Mentor senior engineers and establish engineering standards Collaborate with finance, risk, analytics, and governance stakeholders Translate complex data structures into business-ready insights Nice-to-Have Skills Experience in BFSI, Capital Markets, or Regulatory Reporting Exposure to: SAP Finance Oracle Financials SAP S/4HANA Experience supporting AI/ML workloads Databricks and Cloud Certifications Key Responsibilities Lead Cloudera to Databricks transformation initiatives Shape enterprise finance and risk data platforms Support regulatory, management, and analytical reporting systems Essential Skills Databricks Apache Spark PySpark Spark SQL Delta Lake Lakehouse Architecture Cloudera (CDH/CDP) Hive HBase Impala Data Warehouse & Data Lake Medallion Architecture Databricks SQL dbt AWS / Azure Airflow / Databricks Workflows CI/CD Terraform Jenkins Git Finance & Risk Data Models Enterprise Data Architecture Requirements Sailpoint

📌 Principal Databricks Data Engineer (Toronto)
🏢 Astra North Infoteck
📍 Toronto