Principal Databricks Data Engineer Experience Required: 12–18 Years
Key Requirements 12–18 years of overall Data Engineering experience
8+ years of experience with enterprise Data Warehouse and Data Lake platforms
5+ years of hands‑on experience with Databricks and Apache Spark at scale
Strong experience modernizing legacy Cloudera platforms including: CDH/CDP
Hive
HBase
Impala
Spark

Modernize Cloudera platforms to Databricks Lakehouse architecture
Redesign ingestion, transformation, and consumption patterns from HDFS‑based architecture to Cloud Object Storage and Delta Lake
Refactor legacy Hive and Impala logic into PySpark and Spark SQL ELT pipelines
Ensure data reconciliation, audit integrity, and consistency during migration
Design and govern enterprise Data Warehouse and Data Lake/Lakehouse architectures
Implement layered data architecture including: Raw / Landing Layer
Curated / Conformed Layer
Semantic / Consumption Layer

Modernize traditional Enterprise Data Warehouse platforms into scalable Lakehouse architectures
Robust experience with finance and risk data models including: General Ledger
Sub‑ledger
Financial Hierarchies
Credit Risk
Liquidity Risk
Market Risk

Enable reporting capabilities including: Aggregation
Drill‑down
Drill‑back

Build and manage semantic and consumption layers for BI, reporting, and analytics
Define business metrics, dimensions, hierarchies, and KPIs
Experience with: Databricks SQL
Delta Tables
dbt or similar frameworks

Develop and optimize large‑scale data pipelines using: PySpark
Spark SQL
Delta Lake

Implement Medallion Architecture including: Bronze Layer




Silver Layer
Gold Layer

Optimize workloads using: Z‑ORDER
OPTIMIZE
Caching
Cluster Configuration Tuning

Implement: Data Governance
Data Quality Frameworks
Reconciliation Controls
Exception Handling

Establish data lineage and metadata management
Ensure data security, access control, and compliance standards
Experience with AWS or Azure cloud platforms
Experience with CI/CD pipelines using: Git
Terraform
Jenkins
Azure DevOps

Familiarity with: Apache Airflow
Databricks Workflows

Experience with dbt is an advantage
Act as a technical authority and lead enterprise architecture decisions
Mentor senior engineers and establish engineering standards
Collaborate with finance, risk, analytics, and governance stakeholders
Translate complex data structures into business‑ready insights
Nice‑to‑Have Skills Experience in BFSI, Capital Markets, or Regulatory Reporting
Exposure to: SAP Finance
Oracle Financials
SAP S/4HANA

Experience supporting AI/ML workloads
Databricks and Cloud Certifications
Key Responsibilities Lead Cloudera to Databricks transformation initiatives
Shape enterprise finance and risk data platforms
Support regulatory, management, and analytical reporting systems
Essential Skills Databricks
Apache Spark
PySpark
Spark SQL
Delta Lake
Lakehouse Architecture
Cloudera (CDH/CDP)
Hive
HBase
Impala
Data Warehouse & Data Lake
Medallion Architecture
Databricks SQL
dbt
AWS / Azure
Airflow / Databricks Workflows
CI/CD
Terraform
Jenkins
Git
Finance & Risk Data Models
Enterprise Data Architecture

#J-18808-Ljbffr

📌 Principal Databricks Data Engineer (Ontario)
🏢 Astra-North Infoteck Inc. ~ Conquering today’s challenges, achieving tomorrow’s vision!
📍 Ontario