Azure Databricks

Location: Toronto, Canada

:

We are seeking a highly skilled Backend/API Developer with robust hands-on experience in Azure Databricks and Apache Spark (PySpark/Scala). The ideal candidate will have a solid background in SQL, data transformation techniques, and cloud platforms (Azure/AWS/GCP). You will be responsible for building and optimizing ETL pipelines, ensuring data integrity, and implementing data governance practices.

Responsibilities

Design and develop ETL pipelines using Azure Databricks and Delta Lake, focusing on batch processing (autoloader) and Spark structured streaming.

Create and manage end-to-end environments, including catalogs, schemas, tables, materialized views, functions, and volumes using Unity Catalog.

Implement slowly changing dimensions (SCD1 and SCD2)



on dimension tables and build change data capture (CDC) pipelines.

Utilize Lakehouse federation to create foreign catalogs for accessing data from external sources.

Optimize data processing through effective partitioning and liquid clustering in Databricks.

Collaborate with cross-functional teams to ensure data governance and security practices are adhered to.

Participate in CI/CD pipeline development and DevOps practices to enhance deployment efficiency.

Mandatory Skills

Expertise in Azure Databricks

Expert-level proficiency in SQL

Regular experience with CI/CD practices

📌 Azure Databricks (Toronto)
🏢 Net2Source (N2S)
📍 Toronto