247 Labs | Remote, Canada

About the role

247 Labs is looking for a hands-on AI Engineer to design, develop, deploy, and maintain production AI solutions for our clients.

We are looking for a software engineer with strong AI and data engineering expertise, not someone whose experience is limited to prompt engineering, chatbot configuration, or AI-assisted development.

The successful candidate will understand how AI models work, how to prepare and integrate data, and how to build reliable, scalable applications around machine learning and large language models.

You will work across the complete AI development lifecycle, from analyzing a client's business problem and data sources to selecting models, designing architectures, implementing services, evaluating results, and deploying solutions to production.

Key responsibilities:

1. AI solution architecture and development

- Design and implement end-to-end AI applications using LLMs, machine learning models, retrieval systems, and agentic workflows.
- Evaluate business requirements and determine when to use deterministic logic, traditional ML, RAG, fine-tuning, or agent-based architectures.
- Integrate commercial and open-source AI models through APIs or model-serving infrastructure.
- Implement tool calling, structured outputs, state management, and multi-step AI workflows.
- Select appropriate AI frameworks and technologies based on scalability, reliability, maintainability, and cost.

2. Data engineering and integration

- Design and develop ingestion and transformation pipelines for structured, semi-structured, and unstructured data.
- Work with relational databases, APIs, documents, object storage, and external enterprise systems.
- Use Python and SQL to extract, clean, validate, transform, and prepare data for AI applications.
- Implement reliable ETL/ELT workflows, incremental synchronization, data validation, and error recovery.
- Design data storage, metadata management, indexing, and access-control strategies.

3. Retrieval and knowledge systems

- Build and optimize RAG pipelines, including document processing, chunking, embeddings, indexing, retrieval, reranking, and generation.
- Implement semantic, keyword, and hybrid search.
- Evaluate retrieval performance and generated-answer quality using measurable metrics.
- Maintain accurate, current, permission-aware knowledge bases with document citations and source traceability.

4. AI evaluation and optimization

- Develop automated evaluation frameworks and representative test datasets.
- Evaluate model outputs, retrieval relevance, grounding, tool execution, and end-to-end task completion.
- Establish baselines and regression tests to measure improvements and detect failures.
- Optimize AI solutions for quality, latency, reliability, and operational cost.




- Investigate production issues using traces, metrics, logs, and systematic error analysis.

5. Backend engineering and production deployment

- Develop Python services and REST APIs using FastAPI or equivalent frameworks.
- Implement robust integrations with databases, client applications, and third-party systems.
- Build automated tests, CI/CD pipelines, and containerized services.
- Deploy, monitor, and troubleshoot AI applications on AWS, Azure, or GCP.
- Implement appropriate caching, retries, timeouts, concurrency controls, and background processing.
- Apply secure software engineering practices, including authentication, authorization, secrets management, and data protection.

6. Client collaboration and technical ownership

- Participate in technical discovery and assess client data readiness and integration requirements.
- Translate business goals into AI architecture, implementation plans, and measurable acceptance criteria.
- Explain architectural decisions, technical limitations, risks, and costs to clients and project managers.
- Develop technical proofs of concept, demonstrate solutions, and evolve successful prototypes into production systems.
- Collaborate with backend, frontend, QA, and DevOps engineers.

Must-have qualifications

Software and backend engineering

- 4+ years of professional experience in software, data, ML, or AI engineering, including meaningful hands-on production LLM development.
- Strong Python programming skills, including API development, asynchronous programming, automated testing, and debugging.
- Robust SQL knowledge and experience with relational databases.
- Experience with Git, Docker, CI/CD, and at least one major cloud platform.

Data engineering

- Practical experience designing and maintaining data ingestion and transformation pipelines.
- Experience integrating databases, REST APIs, and structured and unstructured data sources.
- Understanding of data quality, schema design, storage, indexing, and access control.

AI and machine learning

- Solid understanding of modern ML and LLM fundamentals, including embeddings, transformers, tokens, context windows, inference, and model limitations.
- Experience integrating AI models into real applications using model APIs or inference services.
- Ability to explain and evaluate architectural trade-offs between prompting, RAG, fine-tuning, agents, and conventional software.





LLM and retrieval engineering

- Hands-on experience building and deploying at least one substantive AI system used in a real production environment.
- Experience developing RAG pipelines using vector databases or search engines.
- Experience with tool calling, workflow orchestration, structured outputs, or agent frameworks.
- Understanding of retrieval strategies and techniques to improve answer accuracy and grounding.

AI evaluation and operations

- Experience developing evaluation datasets, automated tests, and measurable AI quality metrics.
- Experience troubleshooting LLM applications using logs, traces, and performance metrics.
- Understanding of AI security risks, including prompt injection, sensitive data exposure, and excessive tool permissions.

Communication

- Ability to explain technical decisions to clients and non-technical stakeholders.
- Ability to independently investigate complex engineering problems and take ownership of solutions from design to production.

Nice-to-have qualifications

- Experience with LangGraph, LangChain, Semantic Kernel, or equivalent AI orchestration technologies.
- Experience with model evaluation frameworks such as LangSmith, Ragas, DeepEval, or Langfuse.
- Experience with data orchestration tools such as Airflow, Dagster, or Prefect.
- Knowledge of Hugging Face, PyTorch, and model fine-tuning techniques such as LoRA/PEFT.
- Experience deploying open-source models using vLLM or comparable serving technologies.
- Experience with streaming data, distributed processing, or scalable data infrastructure.
- Experience implementing multi-tenant enterprise AI applications.
- Experience developing secure AI solutions for healthcare, finance, insurance, or other regulated industries.
- Experience working with multiple clients in a consultancy, agency, or software delivery environment.

What success looks like An effective AI Engineer at 247 Labs can independently take a client use case from discovery to production, including:
- Understanding the business problem and defining measurable success criteria.
- Identifying, accessing, preparing, and validating the required data.
- Selecting an appropriate AI architecture and technology stack.
- Implementing a secure, maintainable, and testable solution.
- Evaluating solution quality against an agreed baseline.
- Deploying and monitoring the solution in a real production workplace.
- Explaining the solution, its limitations, and its operational costs to the client.

We value practical engineering experience, strong technical fundamentals, and demonstrated problem-solving ability over familiarity with a long list of AI frameworks. Pay: $55.00-$60.00 per hour

Expected hours: 40.0 per week

Work Location: Remote

📌 AI Engineer – Applied AI, Data & LLM Systems (Toronto)
🏢 247 Labs
📍 Toronto