Key Requirements 12–18 years of overall data engineering experience
8+ years of experience in enterprise Data Warehouse and Data Lake platforms
5+ years of hands‑on experience with Databricks and Spark at scale
Strong experience in modernizing legacy Cloudera platforms (CDH/CDP, Hive, HBase, Impala, Spark) to Databricks Lakehouse
Redesign ingestion, transformation, and consumption patterns from HDFS‑based architecture to cloud object storage and Delta Lake
Refactor legacy Hive/Impala logic into PySpark and Spark SQL ELT pipelines
Ensure data reconciliation, audit integrity, and consistency during migration
Design and govern enterprise Data Warehouse and Data Lake/Lakehouse architectures
Implement layered architecture including Raw/Landing, Curated/Conformed, and Semantic/Consumption layers
Modernize traditional EDW platforms into scalable lakehouse architectures
Robust experience in finance and risk data models including General Ledger, Sub‑ledger, financial hierarchies, and risk exposure models (credit, liquidity, market risk)
Enable reporting use cases including aggregation, drill‑down, and drill‑back capabilities
Build and manage semantic/consumption layers for BI, reporting, and analytics
Define business metrics, dimensions, hierarchies, and KPIs
Experience with Databricks SQL, Delta tables, and dbt or similar frameworks
Develop and optimize large‑scale data pipelines using PySpark,



Spark SQL, and Delta Lake
Implement Medallion architecture (Bronze, Silver, Gold layers)
Optimize workloads using Z‑ORDER, OPTIMIZE, caching, and cluster configurations
Implement data governance, data quality frameworks, reconciliation controls, and exception handling
Establish data lineage and metadata management
Ensure data security, access control, and compliance standards
Experience with cloud platforms such as AWS or Azure
Experience with CI/CD pipelines using Git, Terraform, Jenkins, or Azure DevOps
Familiarity with orchestration tools such as Airflow or Databricks Workflows
Experience with dbt is a plus
Act as a technical authority and lead architecture decisions
Mentor and guide senior engineers and establish engineering standards
Strong stakeholder management with finance, risk, analytics, and governance teams
Ability to translate complex data structures into business‑ready insights
Nice to Have Experience in BFSI, Capital Markets, or regulatory reporting
Exposure to SAP Finance, Oracle Financials, or S/4HANA
Experience supporting AI/ML workloads
Databricks or cloud certifications
Impact Lead Cloudera to Databricks transformation initiatives
Shape enterprise finance and risk data platforms
Support regulatory, management, and analytical reporting systems

#J-18808-Ljbffr

📌 Databricks Engineer (Ontario)
🏢 PeoplePilot
📍 Ontario