We are seeking an experienced Senior QE Automation Engineer to join our AI Development Lifecycle (AIDLC) team. This role focuses on ensuring quality and reliability of AI/ML systems through comprehensive test automation strategies, specialized AI model validation, and robust quality engineering practices.

Key Responsibilities Test Automation & Strategy

Create comprehensive test automation strategies covering unit, integration, system, and end-to-end testing

Implement continuous testing practices within CI/CD pipelines for AI model deployment

Develop automated tests for model training, inference, and monitoring systems

AI/ML Quality Assurance Validate AI model performance, accuracy, bias, fairness, and robustness

Design test cases for model drift detection and data quality validation

Implement automated testing for model versioning and A/B testing scenarios

Conduct performance and load testing for ML inference endpoints

Validate data pipelines, feature engineering, and ETL processes

Tools & Infrastructure Build and maintain test infrastructure for AI workloads

Integrate testing tools with MLOps platforms.

Implement monitoring and observability for test automation systems

Manage test data and synthetic data generation for AI testing

Required Qualifications Technical Skills 5+ years of experience in QA automation engineering





2+ years of hands-on experience testing AI/ML systems or data-intensive applications

Robust programming skills in Python (required) and familiarity with other languages (Java, JavaScript, Typescript)

Expertise in test automation frameworks: Pytest, Selenium, Playwright, Cypress, or similar

Experience with API testing tools: Postman, REST Assured, or similar

Proficiency with CI/CD tools: Jenkins, GitLab CI, GitHub Actions

Strong understanding of ML concepts: model training, evaluation metrics, inference, feature engineering

AI/ML Testing Experience Experience testing machine learning models (classification, regression, NLP)

Knowledge of model evaluation metrics (accuracy, precision, recall, etc.)

Understanding of data quality testing and validation techniques

Familiarity with ML frameworks

Experience with model monitoring and observability tools

Infrastructure & Cloud Experience with cloud platforms: AWS (Lambda), Azure ML

Knowledge of containerization: Docker, Kubernetes

Understanding of distributed systems and microservices architecture

Experience with version control systems (Git) and collaborative development workflows

API Framework – FastAPI

LLM Orchestration – LangGraph

Knowledge Graph – Neo4j

Vector Store - pgvector

📌 QE Automation Engineer with AI experience (Toronto)
🏢 Socket.dev
📍 Toronto