20 Nov
|
KTek Resourcing
|
Toronto
20 Nov
KTek Resourcing
Toronto
- Design, develop, and maintain scalable ETL/ELT pipelines for structured and unstructured data.
- Build and optimize data solutions using
Databricks
,
Spark
, and
Spark SQL
.
- Implement data ingestion frameworks from various source systems including databases, APIs, files, and streaming platforms.
- Develop and maintain data lakehouse architectures leveraging Databricks best practices.
- Design and implement Medallion Architecture (Bronze, Silver, Gold layers) for enterprise data platforms.
- Collaborate with business stakeholders, data scientists, architects, and application teams to understand data requirements.
- Optimize Spark jobs and data pipelines for performance, scalability, and cost efficiency.
- Implement data quality, governance, security, and monitoring frameworks.
- Support AI/ML initiatives by preparing and engineering datasets for model development and deployment.
- Develop CI/CD pipelines and automate deployment processes for data engineering workloads.
- Participate in architecture reviews and establish data engineering standards and best practices.
- Mentor junior engineers and provide technical leadership across projects.
Required Qualifications
- Bachelor's or Master's degree in Computer Science, Information Technology, Data Science, or related field.
- 8+ years of experience in Data Engineering and Data Platform development.
- Strong hands-on experience with
Databricks and
Apache Spark
.
- Proficiency in
Python
,
Spark
, and
SQL
.
- Experience with cloud platforms such as:
- Microsoft Azure (preferred)
- AWS
- Google Cloud Platform
- Experience with Delta Lake, Unity Catalog, and Databricks workflows.
- Robust understanding of Data Lake, Data Warehouse, and Lakehouse architectures.
- Experience working with large-scale distributed data processing systems.
- Knowledge of data modeling, partitioning, indexing, and query optimization techniques.
- Experience with Git, CI/CD pipelines, and DevOps practices.
📌 Senior Data Engineer (Toronto)
🏢 KTek Resourcing
📍 Toronto