1. Quality Engineer – Test Automation, Java/Python & GenAI

Roles and Responsibilities:

- Partner with DevOps, Development, Product, and QE teams to design and execute exploratory, automated, and AI-focused tests that ensure product quality.
- Define quality strategies, test plans, test cases, and test data for functional, integration, system, regression, API, database, and acceptance testing.
- Design, develop, and maintain scalable test automation frameworks using Java, Python, or Groovy.
- Build automated tests for AI-enabled features, including prompt behavior, response accuracy, hallucination risks, safety, reliability, latency, and regression coverage.
- Validate AI integrations, APIs, microservices, data pipelines, model interactions, and third-party services.
- Establish testing approaches for both deterministic and non-deterministic AI behavior, including evaluation datasets, expected behavior, quality thresholds, and repeatability.
- Integrate automated testing and AI quality checks into CI/CD pipelines in partnership with DevOps teams.
- Use tools and frameworks such as Selenium, Cucumber/Gherkin, Playwright, RestAssured, JUnit, TestNG, ReadyAPI, and JMeter.
- Identify, document, prioritize, and track defects through resolution.
- Analyze test coverage, quality trends, AI evaluation results, and production feedback to identify gaps and drive continuous improvement.
- Collaborate with cross-functional Agile teams to deliver reliable, secure, and user-focused product features.
- Communicate test results, risks, quality metrics, and release recommendations clearly to technical and non-technical stakeholders.

Required Skills:

- 5+ years of relevant experience in Quality Engineering, Quality Assurance, test automation, or software development.
- 5+ years of programming experience with Java, Python, Groovy, or a comparable language.




- 5+ years of experience building and maintaining test automation frameworks.
- Hands-on experience with Selenium, Cucumber/Gherkin, Playwright, RestAssured, or similar automation technologies.
- Experience testing functional and non-functional requirements, APIs, integrations, databases, microservices, and regression suites.
- Experience with testing frameworks and tools such as JUnit, TestNG, qTest, ReadyAPI, and JMeter.
- Understanding of TDD, ATDD, BDD, continuous testing, and risk-based testing practices.
- Experience testing AI or machine learning-powered applications, APIs, conversational systems, or generative AI features.
- Understanding of AI quality concepts such as prompt evaluation, grounding, accuracy, relevance, toxicity, bias, hallucination detection, and response consistency.
- Ability to create effective test data and evaluation datasets for AI-driven functionality.
- Strong analytical, problem-solving, communication, and technical documentation skills.
- Ability to work effectively in a fast-paced, cross-functional Agile setting.

Nice to Have:

- Experience with AI evaluation frameworks, LLM testing, prompt engineering, retrieval-augmented generation, embeddings, or model observability.
- Experience with Git, Bitbucket, Jenkins, CI/CD pipelines, Artifactory, SonarQube, Jira, Confluence, AWS, Kubernetes, Maven, or Gradle.
- Experience testing cloud-native applications and distributed microservice architectures.
- Experience with security, performance, accessibility, or reliability testing.
- Experience developing automation scripts and reusable testing utilities.
- Experience with Tableau or other analytics and visualization platforms.
- CSTE certification or an equivalent quality engineering certification.
- Experience working directly with DevOps teams to implement automated quality gates and release controls.

📌 Senior Quality Engineer – Test Automation, AI Testing & Java/Python (Toronto)
🏢 Astra North Infoteck
📍 Toronto