Applied AI Evaluation & Product Researcher (Contract) Applied Research FT Contract Toronto, Toronto, ON, CA

Salary Range: $95,000.00 To $110,000.00 Annually

Job title

Applied AI Evaluation & Product Researcher (12 Month Contract)

Department

Reports to

Director, Applied Research

Salary range

$95,000 - $110,000

Job type

Full-Time, Contract, Hybrid working 1-2 days per week

Location

The opportunity

Are you motivated by the chance to help ensure all young people in Canada can access mental health support and Feel Out Loud? Kids Help Phone (KHP) is Canada’s Unlock The Hope Youth Charity. We’re the first and Canada's only national 24/7, free, confidential, and multilingual e-mental health service for youth and a global leader in innovation for youth and their well-being.

KHP is looking for an Applied AI Evaluation & Product Researcher to lead product evaluation and user‑centred research for its youth-facing generative AI product. This first‑of‑its‑kind experience will offer low‑intensity support through AI, incorporating human oversight informed by KHP’s 37+ years of supervising quality mental health support at scale.

Working with Data and Technology, Innovation, Applied Research, Clinical Services, and other internal teams, the researcher will develop and implement an evaluation strategy that defines success, examines user experience and risk, and translates findings into practical recommendations. They will also support coordination across teams and help develop clinical governance mechanisms for the initiative. It’ll take courage, collaboration, innovation, resilience and trust. If you’re interested in transforming e‑mental health solutions with KHP, you can explore this prospect today





The responsibilities

Evaluation Strategy and Measurement

Develop and implement a comprehensive evaluation strategy for the system, from early testing through launch and post-launch monitoring

Create and maintain an evaluation matrix linking objectives, user scenarios, outcomes, indicators, data sources, timing, and decision criteria

Define measures of success for service users including user experience, adoption, trust, safety, equity, operational performance, and impact

Design evaluation activities and coordinate the collection, analysis, and interpretation of appropriate data

Establish processes for ongoing monitoring, learning, and continuous improvement, and communicate findings to project teams and leadership

Coordinate evaluation activities across internal workstreams and ensure that key evaluation decisions, findings, risks, dependencies, and resulting actions are documented and communicated to the appropriate teams

User Experience Research

Support user research to understand user needs, behaviours, expectations, barriers, and experiences across the end-to-end service journey

Design and conduct post-interaction surveys, usability testing, journey mapping, co-design activities, and other appropriate research methods

Ensure research activities are inclusive, trauma-informed, accessible,



and appropriate for the intended users and service context

Translate findings into clear recommendations for product design, user flows, service pathways, content, and implementation

Safety, Risk and Responsible AI

Co‑develop evaluation criteria, testing approaches, monitoring requirements, and escalation pathways for identified risks(e.g., user, clinical, operational, ethical, privacy, accessibility, and equity‑related risks)

Assess unintended outcomes, inappropriate or unclear recommendations, potential bias, and other issues that may affect user safety or access to support

Contribute research evidence to responsible AI, privacy, and data governance processes and decisions

Collaborate with designated clinical leaders and subject‑matter experts to ensure evaluation criteria reflect relevant clinical standards, service practices, and safety considerations

Implementation and Launch Readiness

Work with internal teams to define launch‑readiness criteria, decision points, evidence requirements, and timelines, and provide evaluation findings to support decisions by accountable product, operational, and clinical leaders

Identify user‑experience issues that may affect implementation and recommend strategies for improvement

Support iterative testing throughout design, development and implementation and evaluate whether changes address identified user needs and risks

Reporting and Knowledge Translation

Synthesize complex research and evaluation findings into practical recommendations, reports, presentations, briefings, and decision materials

Conduct literature reviews, environmental scans, and

📌 Applied AI Evaluation (Winnipeg)
🏢 Kids Help Phone
📍 Winnipeg