Design batch and streaming pipelines at Index Exchange using Spark, Kafka, and Flink for billions of daily events. Build reliable data systems at scale.
As a Staff Data Engineer, you will evolve the data processing and pipeline layer supporting reporting, billing, and real-time data streams. You will transform high-volume event data into clean, queryable, production-grade datasets. The work spans distributed processing, SQL, data modeling, workflow orchestration, and performance optimization.
Key Responsibilities: Design and operate Spark, Kafka, and Flink pipelines
Build canonical datasets and reporting tables
Maintain Airflow DAGs for scheduling and backfills
Optimize runtime, throughput, compute efficiency, and scalability
Define data processing standards across teams
Requirements: Solid experience in data engineering at scale
Expertise in distributed processing frameworks, SQL, and data modeling
Experience with Airflow, Kafka, or streaming systems
Experience managing Kubernetes
Ability to lead projects and debug complex data issues
Bring staff-level data engineering leadership to Index Exchange through scalable pipelines, streaming systems, and optimized data platforms.

#J-18808-Ljbffr

📌 Staff Data Engineer Pipelines (Ontario)
🏢 Index Exchange
📍 Ontario