Our clint is looking for a Senior Big Data Engineer to design, build, and optimize large-scale data pipelines and distributed data systems that power critical business intelligence across the organisation.

Responsibilities

- Build and maintain scalable data pipelines using PySpark within a Big Data workplace to process and transform large volumes of structured and unstructured data.
- Knowledge / experience with DataBricks.
- Design and develop solutions across the Hadoop ecosystem — including Hive, HDFS, Sqoop, Spark, Impala, and Scala — to enable efficient data ingestion, processing, and storage.
- Develop and manage real-time and batch data workflows using streaming data platforms, ensuring high availability and low-latency data delivery.
- Write complex SQL queries to extract, validate, and analyse data across distributed systems, supporting data-driven decision-making.
- Design and implement data models and data architecture patterns aligned with data warehouse principles, ensuring scalability, accuracy, and consistency.
- Automate pipeline scheduling and orchestration using shell scripting and Autosys, reducing manual intervention and improving operational reliability.

Qualifications & Skills 6-8 yrs. of experience require.





- Hands-on expertise in PySpark and Big Data processing, with the ability to build and optimise distributed data workflows at scale.
- Practical knowledge of the Hadoop ecosystem, including Hive, HDFS, Sqoop, Spark, Impala, and Scala, applied in a production environment.
- Proficiency in complex SQL query development for data analysis, transformation, and validation across large datasets.
- Solid understanding of distributed systems architecture and how data flows across interconnected processing layers.
- Demonstrated knowledge of data modelling and data design, with familiarity in data warehouse concepts and dimensional modelling techniques.
- Competence in shell scripting and job scheduling using Autosys or equivalent workflow automation tools.
- Strong analytical and problem-solving ability, with a track record of working independently to diagnose and resolve complex data engineering challenges.
- Clear and effective communication skills, with the ability to articulate technical concepts to both technical and non-technical audiences.

This is a contract role and work is hybrid (3 days on side)

📌 Sr. Big Data Engineer (Greater Toronto Area)
🏢 IT Connex
📍 Greater Toronto Area