Design batch and streaming pipelines at Index Exchange using Spark, Kafka, and Flink for billions of daily events. Build reliable data systems at scale.As a Staff Data Engineer, you will evolve the data processing and pipeline layer supporting reporting, billing, and real-time data streams. You will transform high-volume event data into clean, queryable, production-grade datasets. The work spans distributed processing, SQL, data modeling, workflow orchestration, and performance optimization.Key Responsibilities:
Design and operate Spark, Kafka, and Flink pipelinesBuild canonical datasets and reporting tablesMaintain Airflow DAGs for scheduling and backfillsOptimize runtime, throughput, compute efficiency, and scalabilityDefine data processing standards across teamsRequirements:
Robust experience in data engineering at scaleExpertise in distributed processing frameworks, SQL, and data modelingExperience with Airflow, Kafka, or streaming systemsExperience managing KubernetesAbility to lead projects and debug complex data issuesBring staff-level data engineering leadership to Index Exchange through scalable pipelines, streaming systems, and optimized data platforms.
#J-18808-Ljbffr

📌 Staff Data Engineer Pipelines (Winnipeg)
🏢 Index Exchange
📍 Winnipeg