Overview

Rex.zone is hiring for Senior Data Labeling roles supporting AI/ML training workflows end to end, including data labeling, RLHF, prompt evaluation, QA evaluation, and training data quality audits for large language model evaluation and computer vision annotation. This is a full-time, remote prospect aligned with Ottawa talent.

What You Will Do

- Execute senior-level data labeling across text, image, and multimodal tasks with consistent decision-making and high accuracy

- Perform RLHF-style preference labeling/ranking using rubric-based guidelines

- Run prompt evaluation and QA evaluation to identify failure modes and policy gaps

- Audit training data quality via sampling, error categorization, and root-cause analysis

- Ensure annotation guidelines compliance; propose clarifications and edge-case examples

- Support NER/span labeling and content safety labeling as needed

- Resolve disagreements, calibrate annotators,



and stabilize inter-annotator agreement

- Document decisions and acceptance criteria for downstream model training teams

Core Workflows

- Gold set creation, consensus labeling, disagreement analysis

- Quality systems: spot checks, batch acceptance, error tracking, corrective actions

- Evaluation datasets for LLMs (rubric scoring, adversarial/safety probes)

- Computer vision annotation (bounding boxes, polygons, segmentation) where applicable

Who You Are

- Experienced in data labeling/annotation with strong edge-case handling and consistency

- Comfortable with RLHF, prompt evaluation, and QA evaluation concepts

- Able to mentor peers and communicate clearly when guidelines conflict with real data

📌 Senior Data Labeling Jobs in Ottawa (Remote)
🏢 Rex.zone
📍 Ottawa