Veeva Systems is a mission-driven organization and pioneer in industry cloud, helping life sciences companies bring therapies to patients faster. Do the Right Thing, Customer Success, Employee Success, and Speed. We’re not just any public company – we made history in 2021 by becoming a public benefit corporation (PBC), legally bound to balancing the interests of customers, employees, society, and investors. As a Work Anywhere company, we support your flexibility to work from home or in the office, so you can thrive in your ideal environment. Join us in transforming the life sciences industry, committed to making a positive impact on its customers, employees, and communities. This role is responsible for ensuring the reliability, accuracy, and safety of our Veeva AI Agents through rigorous evaluation and systematic validation methodologies. A meticulous, critical, and curious mindset with a dedication to product quality in a rapidly evolving technological domain Excellent ability to communicate technical findings to both engineering and product management audiences We support your flexibility to work remotely or in the office within Canada or the US, ensuring seamless collaboration within your product team’s time zone. Evaluation Strategy & Planning: Define and establish comprehensive evaluation strategies for new AI Agents. Prioritize the integrity and coverage of test data sets to reflect real-world usage and potential failure modes. Programmatically and manually evaluate the quality of LLM-generated content against predefined metrics (e.g., Creating High-Fidelity Datasets: Design, curate,



and generate diverse, high-quality test data sets, including challenging prompts and scenarios. Evaluate LLM outputs to proactively identify system biases, unsafe content, hallucinations, and critical edge cases. Automation of Evaluation Pipelines: Root Cause Analysis: Understand model behaviors and assist in the trace and root-cause analysis of identified defects or performance degradations. Reporting & Performance Metrics: Clearly document, track, and communicate performance metrics, validation results, and bug status to the broader development and product teams. Data Integrity & Validation: A strong, specialized understanding of data quality principles, including methods for validating datasets against bias, integrity concerns, and quality standards. Ability to craft diverse and adversarial test data to uncover AI edge cases. Demonstrated skill in advanced prompt engineering techniques to create evaluation scenarios that test the AI’s reasoning, action planning, and adherence to system instructions. Automated Evaluation Implementation: Proficiency in designing and deploying automated evaluation pipelines to assess complex, agentic AI behaviors.



Familiarity with quality metrics such as task success rate, semantic similarity, and sentiment analysis for output measurement. Programming & Frameworks: Proficiency in Python for developing custom evaluation frameworks, writing scripts, and integrating pipelines with CI/CD systems. Familiarity with standard test automation tools (e.g., Pytest, modern web automation tools). Bachelor’s degree in Data Science, Machine Learning, Computer Science, or a related field, with experience in Gen AI / LLMs. Medical, dental, vision, and basic life insurance PTO and company-paid holidays Retirement programs Base pay: $85,000
- $225,000 CAD The salary range listed here has been provided to comply with local regulations and represents a potential base salary range for this role. This position may also be eligible for other types of compensation in addition to base salary, such as variable bonus and/or stock bonus. All qualified applicants will receive consideration for employment without regard to race, color, sex, sexual orientation, gender identity or expression, religion, national origin or ancestry, age, disability, marital status, pregnancy, protected veteran status, protected genetic information, political affiliation, or any other characteristics protected by local laws, regulations, or ordinances. If you need assistance or accommodation due to a disability or special need when applying for a role or in our recruitment process, please contact us at talent_accommodations@veeva.
Mid-Senior level Full time #

📌 Arquitecto de datos SQL Senior (Canada)
🏢 Veeva Systems
📍 Canada