Overall – 8+ | AI Exp – 3+
• Hands-on proficiency with libraries like Ragas, DeepEval, TruLens, Promptfoo, or OpenAI Evals to measure:
• Context Precision & Context Recall (retrieval quality from vector stores).
• Faithfulness / Groundedness (checking against hallucination in bounded context).
• Answer Relevance and Toxicity / Guardrail Compliance.
• Analyzing evaluation datasets, confusion matrices, similarity score distributions, and benchmark trends.
• Computing classification metrics (Precision, Recall, F1, Cosine Similarity thresholds) for retrieval systems.
• Azure OpenAI SDK / OpenAI API and LangChain / LlamaIndex for building automated LLM-as-a-judge scoring scripts and synthetic test dataset generation.
• PyTest / Unittest:
Writing automated unit/regression tests for prompt templates and model version upgrades.
• Writing complex SQL queries to extract, curate, and version golden test datasets from relational databases (e.g., PostgreSQL, Azure SQL).
• Querying vector stores and telemetry databases (e.g., pgvector, Splunk/log tables) to evaluate real-world candidate/user query retrieval accuracy and fallback rates.
• Handling structured benchmark datasets and prompt configurations using JSON / JSONL, YAML, and CSV.