What you will be working on

The AI Performance Engineer (Team Lead) is a player-coach role within the AI Enablement Office. You will lead a small team focused on a deceptively important problem: making AI systems at CPF Board actually work well.

Across government and industry, organisations are investing heavily in GenAI and agentic AI. But the discipline of systematically evaluating and improving these systems is still nascent. Most teams know how to build traditional software—they have CI/CD pipelines, unit tests, code reviews, and well-understood quality bars. For AI systems, these equivalents are still being invented. Evaluation is non-trivial when outputs are non-deterministic. Debugging is harder when failure modes are subtle, context-dependent, and probabilistic. Improvement requires a different toolkit—prompt engineering, context engineering, workflow redesign, evaluation framework design—that most engineering teams have not yet developed.

Your job is to lead the effort to close this gap: both by doing the hands-on work of improving AI systems, and by building the organisation’s capability to do so independently.

This is a new function—not just at CPF Board, but in the broader technology ecosystem. There is no established playbook. You will help define the practices, standards, and culture of AI performance engineering within the organisation, and should be comfortable operating in ambiguity as we figure things out together.

In this role, you will:

Evaluating and Improving AI Systems (~50% of the role)

• Lead the design and implementation of evaluation frameworks for GenAI and agentic AI systems, incorporating techniques such as LLM-as-judge, automated evaluation pipelines, human evaluation protocols, and regression benchmarking.
• Personally diagnose and resolve complex AI performance issues—from prompt failures and context engineering gaps to tool selection errors and workflow design flaws—setting the standard for the team.
• Drive systematic improvement of AI systems by iterating on prompts, agent architectures, tool definitions, and orchestration logic. Frameworks in use include LangGraph and N8N.
• Define what “good” looks like for AI outputs across different business contexts, translating domain requirements into measurable evaluation criteria and quality bars.
• Establish and maintain observability practices for AI systems—monitoring drift, regressions, and emergent failure modes in production.

Building Organisation Capability (~30% of the role)

• Design and deliver training programmes, workshops, and hands-on labs on AI evaluation and performance improvement for IT teams across CPF Board.
• Develop comprehensive playbooks, reference architectures, and reusable frameworks that enable other teams to independently evaluate and improve their AI systems.
• Act as an internal evangelist and trusted advisor on AI performance, consulting with project teams on evaluation strategy and improvement approaches.
• Build a community of practice around AI quality, connecting practitioners across the organisation and curating knowledge from the rapidly evolving external ecosystem.
• Over time, extend capability-building efforts beyond IT to business stakeholders, enabling broader AI literacy across the organisation.

Leading a Team (~20% of the role)

• Manage and mentor a small team of AI Performance Engineers, providing technical guidance, career development support, and a psychologically safe environment for experimentation.
• Shape the team’s roadmap and priorities in collaboration with the AI Enablement Office leadership, balancing hands-on improvement work with capability-building initiatives.
• Establish team practices, rituals, and quality standards for a new and evolving discipline.

What are we looking for

We value the diverse talents and experiences that each individual brings to the table. While mastery of every requirement may not be necessary, familiarity and expertise in some of the following areas will position you for success within this team.

• Some software development experience, with the ability to write and debug code effectively. Fresh graduates with strong technical foundations and demonstrated curiosity are welcome.
• Familiarity with Large Language Model (LLM) APIs and core concepts (tokens, context windows, temperature, tool use). Hands-on experience with frameworks like LangChain, LangGraph, or LlamaIndex is advantageous but not essential.
• Understanding of—or strong interest in—AI evaluation methods, prompt engineering, and agentic AI patterns (e.g. ReAct, tool calling, multi-step workflows).
• Strong analytical and problem-solving skills. You should be comfortable reasoning about why an AI system is underperforming and forming hypotheses to test.
• Excellent written communication skills. Much of this work involves writing—prompts, evaluation criteria, documentation, and playbooks—so clarity of expression matters.
• Intellectual curiosity and a genuine interest in how AI systems work under the hood.
• Comfort with ambiguity and willingness to help define a new function from the ground up. There is no established playbook for this role—you will help write it.
• Proactive, self-driven attitude with the ability to work both independently and collaboratively.
• Experience with AI evaluation and observability platforms (e.g. LangSmith, Braintrust, or similar).
• Familiarity with workflow orchestration tools such as LangGraph, N8N, or similar.
• Cloud-native development experience.
• Desire and aptitude to be full-stack—comfortable spanning from infrastructure to UX when needed.

The seniority of appointment and actual corporate job title will commensurate with individual work experiences.

Position is on a 2-year full-time contract directly under the payroll of CPF Board with an option to renew, contingent upon confirmation and subject to organisational needs. Additionally, there is potential for emplacement into a permanent position.