Job Title Backend/API Developer

Location PCS CA~TORONTO

Years of Experience 7-10 Years

Job Summary

We are seeking a highly skilled Backend/API Developer with robust hands-on experience in Azure Databricks and Apache Spark (PySpark/Scala). The ideal candidate will have a solid background in SQL, data transformation techniques, and cloud platforms (Azure/AWS/GCP). You will be responsible for building and optimizing ETL pipelines, ensuring data integrity, and implementing data governance practices.

Responsibilities

- Design and develop ETL pipelines using Azure Databricks and Delta Lake, focusing on batch processing (autoloader) and Spark structured streaming.
- Create and manage end-to-end environments, including catalogs, schemas, tables, materialized views, functions, and volumes using Unity Catalog.
- Implement slowly changing dimensions (SCD1 and SCD2) on dimension tables and build change data capture (CDC) pipelines.




- Utilize Lakehouse federation to create foreign catalogs for accessing data from external sources.
- Optimize data processing through effective partitioning and liquid clustering in Databricks.
- Collaborate with cross-functional teams to ensure data governance and security practices are adhered to.
- Participate in CI/CD pipeline development and DevOps practices to enhance deployment efficiency.

Mandatory Skills
- Expertise in Azure Databricks
- Expert-level proficiency in SQL
- Regular experience with CI/CD practices

Preferred Skills

- Knowledge of data warehousing concepts
- Experience with data modeling and performance tuning in Spark
- Familiarity with data governance and security practices

Qualifications Bachelor's degree in Computer Science, Information Technology, or a related field. 10+ years of experience in Data Engineering projects is required.

📌 Backend/API Developer (Toronto)
🏢 Virtusa
📍 Toronto