The Intermediate Software Developer will support machine learning inference workflows, pipeline execution, and evaluation processes within cloud environments. This role focuses on running batch inference jobs, troubleshooting model performance, preparing and annotating legal text datasets, and maintaining reproducible experiment scripts. The position requires solid technical skills in Python programming, familiarity with LLM frameworks, and experience with cloud-based ML platforms such as Databricks and MLflow.
Responsibilities:
- Execute and monitor batch inference jobs on Databricks and other platforms.
- Troubleshoot inference issues, log results, and escalate anomalies in model behavior.
- Assist in running DSPy pipelines and loading custom models and configurations.
- Support evaluation runs using internal metrics and track performance outcomes.
- Prepare and clean legal text datasets for inference and evaluation purposes.
- Annotate model outputs to facilitate error analysis and benchmarking.
- Maintain reproducible scripts and notebooks for inference and evaluation workflows.
Required Skills:
- Background in Computer Science, Data Science, or AI/ML.
- Proficiency in Python and libraries such as pandas, scikit-learn, or PyTorch.
- Experience with large language model frameworks like DSPy, LangChain, or Hugging Face.
- Familiarity with cloud ML environments including Databricks and MLflow.
- Strong attention to detail and organizational skills for experiment tracking.
Desired Skills:
- Experience working with legal text datasets or domain-specific data preprocessing.
- Knowledge of batch processing and scalable inference techniques.
- Ability to annotate and benchmark model outputs effectively.
- Familiarity with maintaining reproducible data science workflows and notebooks.
Rated Criteria:
- Technical Proficiency in Python and ML Libraries - 40 Points
- Experience with LLM Frameworks (DSPy, LangChain, Hugging Face) - 25 Points
- Competence with Cloud ML Platforms (Databricks, MLflow) - 25 Points
- Data Preparation and Annotation Skills - 10 Points
Total: 100 Points Deliverables:
- Successfully executed and monitored batch inference jobs.
- Documented troubleshooting logs and escalated model anomalies.
- Cleaned and annotated legal text datasets prepared for inference and evaluation.
- Reproducible scripts and notebooks supporting inference and evaluation runs.
- Performance metric summaries for inference experiments.
Must Haves:
- Strong foundation in Computer Science, Data Science, or AI/ML.
- Proficiency in Python and relevant ML libraries.
- Hands-on experience with LLM frameworks.
- Comfortable working in cloud ML environments such as Databricks and MLflow.
- Detail-oriented with excellent organizational skills for experiment tracking.