Remote Data Annotator (Full-Time, Remote)

Rex.zone helps candidates discover and apply to remote data annotator roles supporting contemporary AI/ML training pipelines. In this role, you will create labeled datasets and evaluation signals that improve training data quality and large language model performance across NLP, computer vision, and content safety workflows.

What You Will Do
Apply detailed annotation guidelines to label and review text, images, audio, and multi-turn conversations
Perform LLM evaluation and prompt evaluation (helpfulness, harmlessness, factuality, reasoning quality)
Complete RLHF-style preference labeling, rubric scoring, and critique writing to generate high-signal feedback
Run QA evaluation using sampling plans, inter-annotator agreement checks, and corrective actions
Document edge cases, ambiguity, and rationale to improve rubrics, taxonomies, and consistency
Flag dataset issues (duplicates, leakage, label noise)



and follow confidentiality/data-handling standards

Required Qualifications
Mid-to-senior experience in data annotation, data labeling, or QA evaluation
Solid guideline compliance, consistency, and written reasoning for critique-based tasks
Comfort with structured labeling tools, taxonomies, and metrics-driven work from home
Familiarity with NLP/LLM evaluation or content moderation taxonomies

Preferred
RLHF workflows, preference ranking, and rubric-based scoring experience
Computer vision annotation exposure (bounding boxes, polygons, segmentation QA)
Understanding of training data quality concepts (bias, representativeness, label noise)

Compensation

Base hourly rate: $30–$50/hr (USD), depending on project needs and qualifications.

📌 Remote Data Annotator Montreal
🏢 Rex.zone
📍 Montreal