Applied AI Evaluation & Product Researcher (12 Month Contract)

Reports to

Director, Applied Research

Salary range

$95,000 - $110,000

Job type

Full-Time, Contract, Hybrid working 1-2 days per week

Posting period

Tuesday, November 17, 2026

The opportunity

Are you motivated by the chance to help ensure all young people in Canada can access mental health support and Feel Out Loud? Kids Help Phone (KHP) is Canada’s Unlock The Hope Youth Charity. We’re the first and Canada's only national 24/7, free, confidential, and multilingual e-mental health service for youth and a global leader in innovation for youth and their well-being.

KHP is looking for an Applied AI Evaluation & Product Researcher to lead product evaluation and user-centred research for its youth-facing generative AI product. This first-of-its-kind experience will offer low-intensity support through AI, incorporating human oversight informed by KHP’s 37+ years of supervising quality mental health support at scale.

Working with Data and Technology, Innovation, Applied Research, Clinical Services, and other internal teams, the researcher will develop and implement an evaluation strategy that defines success, examines user experience and risk, and translates findings into practical recommendations. They will also support coordination across teams and help develop clinical governance mechanisms for the initiative. It’ll take courage, collaboration, innovation, resilience and trust. If you’re interested in transforming e-mental health solutions with KHP, you can explore this prospect today

The responsibilities

Evaluation Strategy and Measurement

- Develop and implement a comprehensive evaluation strategy for the system,



from early testing through launch and post-launch monitoring
- Create and maintain an evaluation matrix linking objectives, user scenarios, outcomes, indicators, data sources, timing, and decision criteria
- Define measures of success for service users including user experience, adoption, trust, safety, equity, operational performance, and impact
- Design evaluation activities and coordinate the collection, analysis, and interpretation of appropriate data
- Establish processes for ongoing monitoring, learning, and continuous improvement, and communicate findings to project teams and leadership
- Coordinate evaluation activities across internal workstreams and ensure that key evaluation decisions, findings, risks, dependencies, and resulting actions are documented and communicated to the appropriate teams

User Experience Research

- Support user research to understand user needs, behaviours, expectations, barriers, and experiences across the end-to-end service journey
- Design and conduct post-interaction surveys, usability testing, journey mapping, co-design activities, and other appropriate research methods
- Ensure research activities are inclusive, trauma-informed, accessible, and appropriate for the intended users and service context
- Translate findings into clear recommendations for product design, user flows,



service pathways, content, and implementation

Safety, Risk and Responsible AI

- Co-develop evaluation criteria, testing approaches, monitoring requirements, and escalation pathways for identified risks(e.g., user, clinical, operational, ethical, privacy, accessibility, and equity-related risks)
- Assess unintended outcomes, inappropriate or unclear recommendations, potential bias, and other issues that may affect user safety or access to support
- Contribute research evidence to responsible AI, privacy, and data governance processes and decisions
- Collaborate with designated clinical leaders and subject-matter experts to ensure evaluation criteria reflect relevant clinical standards, service practices, and safety considerations

Implementation and Launch Readiness

- Work with internal teams to define launch-readiness criteria, decision points, evidence requirements, and timelines, and provide evaluation findings to support decisions by accountable product, operational and clinical leaders
- Identify user-experience issues that may affect implementation and recommend strategies for improvement
- Support iterative testing throughout design, development and implementation and evaluate whether changes address identified user needs and risks

Reporting and Knowledge Translation

- Synthesize complex research and evaluation findings into practical recommendations, reports, presentations, briefings, and decision materials
- Conduct literature reviews, environmental scans, and evidence syntheses relevant to Generative AI, digital services, user experience, safety, and implementation
- Communicate resear

📌 Applied AI Evaluation (Toronto)
🏢 Kids Help Phone
📍 Toronto