Overview Senior data annotation jobs in Montreal focus on creating and evaluating high-quality training data for AI/ML systems. At Rex.zone, you will support LLM training pipelines through data labeling, RLHF evaluation, prompt evaluation, and QA review to improve model performance and safety in a remote, full time role.

Key Responsibilities
Deliver high-accuracy data labeling and review across NLP, LLM evaluation, computer vision annotation, and content safety labeling
Design, maintain, and enforce annotation guidelines compliance and taxonomy standards
Lead QA evaluation workflows including sampling plans, inter-annotator agreement checks, and adjudication
Execute RLHF tasks such as pairwise ranking, preference labeling, and consistency audits
Conduct prompt evaluation and rubric-based scoring aligned to model requirements
Perform dataset audits, ambiguity logging, and label error analysis to improve training data quality
Partner with engineering/ML stakeholders to define acceptance criteria and feedback loops into LLM training pipelines
Document edge cases and escalation paths to reduce guideline drift




Support onboarding and calibration using examples, gold sets, and retraining plans
Track quality metrics and recommend process improvements that improve throughput without sacrificing accuracy

Required Qualifications
Senior data annotation or data labeling operations experience with QA evaluation ownership
Solid understanding of taxonomy design, guideline adherence, and ambiguity resolution
Hands-on experience with LLM evaluation, RLHF, or prompt evaluation workflows
Familiarity with NLP tasks such as named entity recognition and text classification
Comfortable working remotely with cross-functional stakeholders

Nice to Have
Content safety labeling or policy-driven evaluation experience
Inter-annotator agreement metrics and quality sampling methodologies
Computer vision annotation exposure (bounding boxes, polygons, segmentation)
Multilingual evaluation or locale-specific guidelines

Work Model Remote, Full time. Apply with relevant annotation experience and examples of QA evaluation or guideline development work.

📌 Senior Data Annotation Jobs In Montreal
🏢 Rex.zone
📍 Montreal