Job Summary:

The Senior Software Developer will be responsible for executing and monitoring batch inference jobs, supporting pipeline and evaluation processes, and managing data preparation and experimentation specifically for legal text datasets. This role requires expertise in large language model (LLM) frameworks and cloud ML environments, with a focus on maintaining reproducibility and tracking performance metrics.

Responsibilities:

- Execute and monitor batch inference jobs on Databricks and other platforms.
- Troubleshoot inference issues, log results, and escalate model anomalies appropriately.
- Assist with running DSPy pipelines and loading custom models and configurations.
- Support evaluation runs using internal metrics and track performance results.
- Prepare and clean legal text datasets for inference and evaluation purposes.
- Annotate outputs to facilitate error analysis and benchmarking.
- Maintain reproducible scripts and notebooks for inference runs.

Required Skills:
- Background in Computer Science, Data Science, or AI/ML.
- Proficiency in Python and relevant libraries such as pandas, scikit-learn, or PyTorch.
- Experience with LLM frameworks like DSPy, LangChain, or Hugging Face.
- Familiarity with Databricks, MLflow, or similar cloud ML environments.
- Solid organizational skills with attention to detail for experiment tracking.

Desired Skills:




- Experience working with legal text or domain-specific datasets.
- Knowledge of batch processing and scalable ML inference pipelines.
- Ability to annotate and analyze model outputs for benchmarking.
- Comfortable with troubleshooting and escalating model-related issues.
- Experience maintaining reproducible and well-documented codebases.

Rated Criteria:
- Technical Skills (Python, data libraries) - 50 Points
- Expertise in LLM Frameworks (DSPy, LangChain, Hugging Face) - 25 Points
- Experience with Databricks and MLflow - 25 Points

Total: 100 Points Deliverables:
- Successfully executed and monitored batch inference jobs with documented results.
- Reproducible scripts and notebooks for inference and evaluation.
- Cleaned and annotated legal text datasets prepared for model evaluation.
- Performance metrics summaries for inference experiments.
- Issue logs and escalations related to model anomalies.

Must Haves:
- Five days onsite availability.
- Strong proficiency in Python and ML frameworks.
- Experience with cloud ML platforms, specifically Databricks and MLflow.
- Detail-oriented approach to experiment tracking and documentation.
- Background in Computer Science, Data Science, or AI/ML.

Public Sector Experience:

Not Preferred

📌 Senior Software Developer (Ontario)
🏢 vTech Solution
📍 Ontario