Design batch and streaming pipelines at Index Exchange using Spark, Kafka, and Flink for billions of daily events. Build reliable data systems at scale.

As a Staff Data Engineer, you will evolve the data processing and pipeline layer supporting reporting, billing, and real-time data streams. You will transform high-volume event data into clean, queryable, production-grade datasets. The work spans distributed processing, SQL, data modeling, workflow orchestration, and performance optimization.

Key Responsibilities:

- Design and operate Spark, Kafka, and Flink pipelines
- Build canonical datasets and reporting tables
- Maintain Airflow DAGs for scheduling and backfills
- Optimize runtime, throughput, compute efficiency, and scalability
- Define data processing standards across teams

Requirements:

- Robust experience in data engineering at scale
- Expertise in distributed processing frameworks, SQL, and data modeling
- Experience with Airflow, Kafka, or streaming systems
- Experience managing Kubernetes
- Ability to lead projects and debug complex data issues

Bring staff-level data engineering leadership to Index Exchange through scalable pipelines, streaming systems, and optimized data platforms.

📌 Staff Data Engineer Pipelines (Toronto)
🏢 Index Exchange
📍 Toronto