Job Title Principal Data Engineer Databricks (Finance Risk Cloudera Modernization)Location Toronto| Canada (Hybrid Onsite as required) Experience 12-18 years overall data engineering experience 8 years across enterprise Data Warehouse Data Lake platforms 5 years with Databricks Spark at scale
Role Summary The Principal Databricks Data Engineer is a senior technical leader responsible for modernizing large scale finance and risk data platforms from legacy Cloudera ecosystems to cloud native Databricks lakehouse architectures.
This role demands deep hands on expertise in data warehousing| data lakes| finance risk data models| and semanticconsumption layers| with robust experience supporting regulatory| management reporting| and analytics use cases.
The individual will serve as a hands on architect and technical authority| leading platform modernization while partnering closely with Finance| Risk| Analytics| and Governance stakeholders.
Key Responsibilities
Cloudera Databricks ModernizationLead modernization of legacy Cloudera platforms (CDH CDP| Hive| HBase| Impala| Spark) into Databricks Lakehouse.
Redesign ingestion| transformation| and consumption patterns from HDFS centric architectures to cloud object storage and Delta Lake.
Refactor legacy HiveImpala logic into PySpark Spark SQL based ELT pipelines.
Ensure data parity| reconciliation| and audit integrity during platform migration.
Enterprise Data Warehouse Data Lake Architecture Design and govern enterprise Data Warehouse and Data Lake Lakehouse architectures.
Implement layered architectures spanning o Raw Landing zones o Curated conformed layers o Semantic consumption layers Modernize traditional EDW patterns into domain aligned| scalable lakehouse designs.
Finance Risk Data Modeling Support implementation of finance and risk data models| including o General Ledger Sub ledger data o Accounting events and financial hierarchies o Risk exposure| liquidity| credit| and market risk models Enable aggregation| drill down| and drill back from reports to transaction level data.
Support regulatory reporting| management reporting| and analytics use cases.
Semantic Consumption Layers Build and manage semantic consumption layers to ensure consistent business logic across o BI and reporting tools o Finance Risk analytics o Self service analytics platforms Define metrics| dimensions| hierarchies| and KPIs aligned to finance and risk definitions.
Implement semantic models using o Databricks SQL o Delta tables o dbt or equivalent transformation frameworks Databricks Engineering Optimization Engineer large scale pipelines using PySpark| Spark SQL| and Delta Lake.
Implement medallion architecture (Bronze Silver Gold) aligned to business domains.
Optimize Databricks workloads for cost| performance| and reliability (Z ORDER| OPTIMIZE| caching| cluster policies).
Data Governance| Quality Lineage Implement data quality frameworks| reconciliation co
Skills
Category
Name
Importance
Experience
SkillCategoryTest1_MN
Digital : Azure Databricks
Yes
1
>7 years
📌 CAN_Platform Architect (Toronto)
🏢 VARITE
📍 Toronto
Reply to this offer
Impress this employer describing Your skills and abilities, fill out the form below and leave Your personal touch in the presentation letter.