Databricks Developer with AWS

Location: Toronto, ON- Work Mode: Hybrid – Tuesday, Wednesday & Thursday onsite

Contract: 12 Months contract

Experience: 7+ Years

Job Summary

We are seeking an experienced Databricks Developer with strong AWS expertise to join a data engineering team supporting enterprise-scale data platforms for a leading financial services/investment management organization.

The ideal candidate will have 7+ years of experience in data engineering, with strong hands-on expertise in Databricks, Apache Spark, PySpark, Python, SQL, and AWS cloud services. The candidate will be responsible for designing, developing, optimizing, and maintaining scalable data pipelines and data processing solutions.

This role requires someone who can work effectively in a large enterprise environment, collaborate with data architects, business stakeholders, developers, and QA teams, and follow strong standards around data quality, security, performance, and governance.

Key Responsibilities

- Design, develop, and maintain scalable ETL/ELT data pipelines using Databricks and Apache Spark.
- Develop complex data transformations using PySpark, Python, and SQL.
- Build and optimize data pipelines on AWS cloud platforms.
- Work with AWS services such as S3, Glue, Lambda, EMR, Redshift, IAM, and related data services.
- Develop data processing solutions using Databricks notebooks, workflows, jobs, clusters, and Delta Lake.
- Implement reliable and reusable data ingestion frameworks for structured and semi-structured data.
- Perform data cleansing, transformation, validation, aggregation, and enrichment.
- Work with Delta Lake and implement efficient storage and data management strategies.
- Optimize Databricks/Spark jobs for performance, scalability, and cost efficiency.
- Troubleshoot production data pipeline failures and resolve data quality or performance issues.




- Implement appropriate error handling, logging, monitoring, and recovery mechanisms.
- Collaborate with data architects to implement enterprise data architecture and engineering standards.
- Participate in data modeling and development of analytical data structures.
- Work with large-volume datasets and implement distributed data processing solutions.
- Ensure data pipelines meet data quality, security, availability, and governance requirements.
- Participate in code reviews and follow established development standards.
- Develop and maintain technical documentation for data pipelines and processes.
- Work closely with QA teams to support data validation and testing.
- Support deployment of Databricks and AWS solutions across development, test, and production environments.
- Participate in Agile ceremonies including sprint planning, daily stand-ups, reviews, and retrospectives.

Required Technical Skills

Databricks

- Strong hands-on experience with Databricks.
- Databricks notebooks, jobs, workflows, clusters, and job scheduling.
- Experience with Delta Lake.
- Experience optimizing Databricks/Spark workloads.
- Understanding of Databricks data engineering best practices.

AWS Robust hands-on experience with AWS data services, including:

- AWS S3
- AWS Glue
- AWS Lambda
- AWS EMR
- AWS Redshift
- AWS IAM
- Experience with AWS-based data architecture and cloud data pipelines.

Programming & Data Processing





- Strong Python / PySpark development experience.
- Strong Apache Spark knowledge.
- Advanced SQL skills.
- Experience working with large-scale datasets and distributed processing.

Data Engineering

- Strong experience developing ETL/ELT pipelines.
- Experience with batch and/or near-real-time data processing.
- Data ingestion from multiple source systems.
- Data transformation and data quality validation.
- Experience working with structured and semi-structured data.
- Understanding of data warehousing and up-to-date data lake/lakehouse architectures.

Preferred Skills

- Experience with Databricks Unity Catalog.
- Experience with AWS Glue Data Catalog.
- Knowledge of Delta Live Tables / Lakeflow.
- Experience with CI/CD for Databricks.
- Git/GitHub or similar source-control systems.
- Experience with Terraform or Infrastructure as Code.
- Knowledge of DevOps practices.
- Experience with data governance and data security.
- Experience with enterprise data platforms in the financial services, banking, or investment management domain.
- Familiarity with Agile/Scrum development methodologies.

Financial Services / Enterprise Experience Experience working in a large financial services, banking, investment management, or capital markets environment is highly desirable.

Candidates should be comfortable working with enterprise-level data, security requirements, regulatory considerations, data governance standards, and production-critical data pipelines.

Education & Experience

- Bachelor's degree in Computer Science, Information Technology, Engineering, or a related discipline preferred.
- 7+ years of overall IT/data engineering experience.
- Strong recent hands-on experience with Databricks and AWS.
- Proven experience developing and supporting enterprise-scale data pipelines.

📌 Databricks Developer with AWS (Toronto)
🏢 Q1 Technologies
📍 Toronto