Job Description
We are hiring a Databricks ETL Developer to support a strategic Capital Markets Client Reporting initiative through the modernization and automation of enterprise data processes.

This consultant will build scalable data pipelines, develop reporting-focused data warehouse solutions, and deliver high-quality data assets within an Azure and Databricks environment.

The ideal candidate will possess strong data engineering and data warehousing expertise and be comfortable supporting reporting and analytics-focused use cases.

Key Responsibilities
- Architect and implement data pipelines following the Medallion (Bronze/Silver/Gold) architecture.
- Ingest data from diverse sources including APIs, flat files, binary files, and Databricks-to-Databricks transfers using Azure Data Lake Storage Gen2 (ADLS Gen2).
- Apply data modeling techniques including Data Vault 2.0 and Kimball Dimensional Modeling.
- Implement a Data Quality Framework to enforce data integrity and reliability standards.
- Handle complex data scenarios such as SCD Type 2, late-arriving data, carry-forward logic, backfill, reprocess, and restatement.
- Build ETL control and audit frameworks for lineage, monitoring, and traceability.
- Develop and optimize Delta Lake tables leveraging ACID transactions, Z-ordering, and OPTIMIZE for performance.
- Build real-time data pipelines using Structured Streaming for low-latency ingestion and processing.
- Orchestrate and schedule pipeline workflows using Databricks Workflows (Jobs).
- Tune Spark jobs through query optimization, partitioning strategies,



and cluster/compute sizing.
- Enforce data security policies including Row-Level Security (RLS), Column-Level Security (CLS), and data masking.
- Manage data governance and access control using Unity Catalog.

We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day.

We are an equal opportunity/affirmative action employer that believes everyone matters.

Qualified candidates will receive consideration for employment regardless of their race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances.

If you need assistance and/or a reasonable accommodation due to a disability during the application or recruiting process, please send a request to [email protected].

To learn more about how we collect, keep, and process your private information, please review Insight Global's Workforce Privacy Policy:



https://insightglobal.com/workforce-privacy-policy/.

Skills and Requirements
- Hands-on experience with Medallion architecture on Azure (ADLS Gen2).
- Robust proficiency in PySpark and Databricks SQL.
- Deep knowledge of Delta Lake, including ACID transactions, Z-ordering, and incremental processing.
- Expertise in Structured Streaming for real-time pipeline development.
- Proficiency with Databricks Workflows (Jobs) for orchestration and scheduling.
- Experience with Spark Declarative Pipelines (Lakeflow, formerly Delta Live Tables).
- Advanced Spark performance tuning, partitioning, caching, and compute sizing.
- Deep knowledge of Data Vault 2.0 and Kimball Dimensional Modeling.
- Solid understanding of Unity Catalog for data governance and security.
- Expertise in exception handling and data quality enforcement.
- Familiarity with Genie Space for natural language data exploration. - Strong data warehousing experience, including designing, building, and supporting a data warehouse rather than working exclusively on ETL pipelines.
- Experience developing data models and warehouse structures that support downstream reporting and analytics.
- Experience supporting Power BI reporting teams and optimizing data for reporting consumption.
- Experience automating manual data uploads and integrating multiple source systems.
- Capital Markets, Corporate Banking, or broader Financial Services experience.
- Experience working in an Agile delivery environment using Jira and Confluence.

📌 Databricks ETL Developer (Toronto)
🏢 Insight Global
📍 Toronto