Senior Data Engineer –Databricks

3 months Contract with possibility of extension.

Work Arrangement - Remote

Please find below the details JD:

Key Responsibilities:

- Develop and implement data solutions using Databricks, with hands-on experience in specific Databricks platform features such as Delta Lake, Uniform (Iceberg), Delta Live Tables (Lakeflow Declarative Pipelines), and Unity Catalog.
- Collaborate with cross-functional teams to design and optimize data pipelines for both batch and streaming data, ensuring data quality and efficiency.
- Stay updated with emerging technologies, latest Databricks platform features and continuously improve your skills in Databricks and other relevant tools.

Mandatory Requirements:

- 5+ years of data engineering experience, with at least 2 years of hands-on data pipeline design and development experience with Databricks, including specific -platform features like Delta Lake, Uniform (Iceberg), Delta Live Tables (Lakeflow Declarative pipelines), and Unity Catalog.
- Proficiency in designing and building robust, scalable, YAML configuration driven data pipelines with batch, micro-batch,



and streaming data ingestion and processing patterns using tools like Auto Loader demonstrating a strong understanding of modern data engineering practices.Experienced in building complex job/workflow orchestration patterns using either Databricks jobs/workflows or with other orchestration tools like Airflow, dbt etc.
- Exposure in building robust data quality, pipeline audit/observability solutions with
- Databricks native features and/or other data quality tools/frameworks like Outstanding Expectations, Collibra, dbt etc.
- Proficiency in Big Data Platforms: Apache Spark, Presto, Amazon EMR.
- Experience with Cloud Data Warehouses: Amazon Redshift, Snowflake, Google BigQuery.
- Strong programming skills using SQL, Stored Procedures and Object-Oriented Programming languages (like Java, Python, PySpark etc.).
- Familiarity with building DevOPs, CI/CD pipelines using Databricks Asset Bundles with automated validation and testing.
- Exposure with Infrastructure as Code (IaC) tools like Terraform is a big plus.
- Familiarity with NoSQL Databases and Container Management Systems.
- Exposure to AI/ML tools (like mlflow), prompt engineering, and modern data and AI agentic workflows.
- An ideal candidate will have Databricks Data Engineering Associate and/or Professional certifications completed with multiple Databricks project delivery experience.

📌 Senior Data Engineer –Databricks (Canada)
🏢 Ideaon
📍 Canada