Overall – 8+ | AI Exp – 3+

- Hands-on proficiency with libraries like

Ragas

,

DeepEval

,

TruLens

,

Promptfoo

, or

OpenAI Evals to measure:

- Context Precision & Context Recall

(retrieval quality from vector stores).

- Faithfulness / Groundedness
(checking against hallucination in bounded context).

- Answer Relevance and

Toxicity / Guardrail Compliance

.

- Analyzing evaluation datasets, confusion matrices, similarity score distributions, and benchmark trends.

- Computing classification metrics (Precision, Recall, F1, Cosine Similarity thresholds) for retrieval systems.

- Azure OpenAI SDK / OpenAI API and

LangChain / LlamaIndex for building automated

LLM-as-a-judge scoring scripts and synthetic test dataset generation.

- PyTest / Unittest:




Writing automated unit/regression tests for prompt templates and model version upgrades.

- Writing complex SQL queries to extract, curate, and version golden test datasets from relational databases (e.g., PostgreSQL, Azure SQL).

- Querying vector stores and telemetry databases (e.g., pgvector, Splunk/log tables) to evaluate real-world candidate/user query retrieval accuracy and fallback rates.

- Handling structured benchmark datasets and prompt configurations using

JSON / JSONL

,

YAML

, and

CSV

.

📌 LLM Evaluation Engineer (Canada)
🏢 HCLTech
📍 Canada