About The Job Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark , General Catalyst , Peter Thiel , Adam D'Angelo , Larry Summers , and Jack Dorsey .

Position: Generalist Annotator — Health AI Conversation Quality Evaluation

Type: Contract Compensation: $20–$160/hour Location: Remote Role Responsibilities

- Evaluate the relevance and clarity of AI responses. Ensure the AI answers the question in understandable language for non-experts.
- Detect sycophancy in AI interactions. Identify when the AI tells users what they want to hear instead of what they need.
- Assess the tone and outcome of conversations. Determine if the user leaves informed and calm or anxious and uninformed.
- Provide written justification for ratings below the top option. Explain what went wrong and suggest improvements.
- Work independently to review conversations ranging from 1 to 7 turns. Spend 2–13 minutes per conversation based on length.

Qualifications Must-Have

- Robust written English skills.
- Ability to explain judgments in two to four specific sentences.




- Careful reading to notice omissions in responses.
- Comfort with applying a detailed rubric consistently.
- Reliable availability for time-boxed batches.

Preferred

- Experience in annotation, human-feedback, evaluation, or content review.
- Background in explaining technical concepts to non-experts.
- Experience with rubric-based grading or quality assurance.

Important

- Using an AI tool to write comments is prohibited. This work captures human judgment that AI lacks. AI-written feedback corrupts the dataset and is checked for.

Application Process (Takes 20–30 mins to complete)

- Upload resume
- AI interview based on your resume
- Submit form

Resources & Support

- For details about the interview process and platform information, please check: https://talent.docs.mercor.com/welcome
- For any help or support, reach out to: [email protected]

PS: Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity. ,

📌 Health AI Conversation Evaluator (Montreal)
🏢 Mercor
📍 Montreal