Senior Data EngineerThe Senior Data Engineer at HackerOne is responsible for leading the end-to-end design and delivery of scalable, secure, and intelligent data products and solutions to support the company’s transformation into an AI-first organization. This role involves partnering across business and engineering teams to identify opportunities for automation, integration, and system modernization, driving the architecture and execution of platform-level capabilities by leveraging AI and modern tooling to reduce manual effort, improve decision-making, and increase system resilience. The engineer will provide technical leadership to internal engineers and external development partners to ensure design quality, operational excellence, and long-term maintainability, shape and contribute to incident and on-call response strategies, playbooks, and processes to build systems that fail gracefully and recover quickly, mentor other engineers and advocate for technical excellence, and promote a culture of innovation and continuous improvement. Additionally, the role includes championing effective change management to ensure systems are successfully launched, adopted, understood, and evolved.
As a Data Engineer on the Foundational team, you will build ETL/ELT pipelines to extract, decode, and store raw Electro-Optical (EO) and Infrared (IR) video from field logs into optimized formats like WebDataset, TFRecords, or Parquet. You will develop algorithms to synchronize EO and IR frames temporally and spatially to provide paired inputs for model training. You will architect storage-to-GPU pipelines to ensure multi-node training clusters maintain over 90% GPU utilization without I/O bottlenecks.



Your role includes writing and optimizing distributed data processing jobs using tools such as Apache Spark, Ray, or Apache Beam to process thousands of hours of tactical video logs. You will implement automated quality checks to filter corrupted or blank frames and maintain reproducible training runs through robust versioning and lineage tracking. Additionally, you will assess and implement advanced storage solutions like MinIO and S3 tiering to manage growing datasets while optimizing cost and latency.
Collaborate with Machine Learning, Full-stack engineers and Science to solve complex document mining challenges, capture and model additional scientific experiments, scale data pipelines for rapid and reliable data transfer from research to platform, work with semi-structured and unstructured data, define and apply best practices for technologies in a cloud-based setting, architect and maintain robust data pipelines that ingest diverse sources and use large language models (LLMs) for high-fidelity entity extraction into structured formats, implement evaluation frameworks to monitor extraction models’ accuracy, drift, and hallucination rates in production pipelines, lead or consult on engineering design proposals according to the Platform Stream roadmap, make independent technical decisions based on business context and team goals, proactively identify new opportunities and implement project improvements,



respond with urgency to operational issues and own resolution within one’s responsibility, and challenge the status quo by proposing new technologies or ways of working.
Member of Technical Staff, Pre-training DataAs a Software Engineer on the Pre-training Data team, you will design and operate the systems that define the model’s training corpus at scale. Your work will focus on large-scale data acquisition, processing, filtering, mixture design, and ablation-driven iteration. You will handle infrastructure and experimental loops that decide the data used for training and thereby influence what the model learns. Responsibilities include building and operating large-scale web crawling, scraping, and ingestion pipelines;

designing filtering, deduplication, quality controls, and dataset versioning systems;

running data ablations across sources, rewrites, mixtures, and long-sequence strategies;

optimizing distributed data processing systems for throughput and cost efficiency;

improving observabilityand reliability of large ETL and dataflow jobs;

and collaboratingwith Research and Training Systems teams to ensure corpus design aligns with model behavior.
Set technical strategy and oversee development of a high scale, reliable data platform to manage, visualize, and serve large-scale datasets for ML model training and validation. Build the data lakehouse for autonomous driving scene datasets, including sensor data, calibration data, and annotation data. Drive the Autonomous Driving Data SDK development, including scene data search, datasets preparation, and dataset loading. Identify and resolve performance bottlenecks in the data processing pipelines, including data processing latency, data search latency,

📌 Ai Data Engineer Jobs (Toronto)
🏢 HackerOne
📍 Toronto