AI / LLM Evaluation Engineer

About the Role

We're looking for a Senior LLM Evaluation Engineer to join our partner companies shipping AI products to real users. Most of these teams can change a prompt in five minutes and have no way to tell if it broke something. You'll fix that. You'll build the evaluation harness, curate golden datasets from production traffic, write and calibrate LLM-as-judge graders, and wire regression evals into CI so a bad change gets caught before a customer sees it. Part of the job is red-teaming: finding the inputs that make the system fail, leak data, or say something it shouldn't.

What You'll Do

• Design and build eval harnesses for LLM features, agents, and RAG pipelines
• Build and maintain golden datasets from production logs, edge cases, and expert labels
• Write LLM-as-judge graders and check them against human ratings
• Run regression evals in CI on every prompt, model, or retrieval change
• Red-team the system for jailbreaks, prompt injection, data leaks, and harmful output
• Report eval results to product and engineering in a form they can act on

Requirements

• 4+ years of software or ML engineering, 1+ year building evals for LLM systems
• Strong Python and experience with data tooling (pandas, notebooks, SQL)
• Hands-on experience with LLM APIs (OpenAI, Anthropic, open-weight models)
• Understanding of eval design: sampling, inter-rater agreement, metric choice, and judge bias
• Experience with at least one eval or tracing tool (promptfoo, LangSmith, Braintrust, Ragas, OpenAI Evals)
• Excellent written and verbal English (B2+)

Nice to Have

• Background in ML research, data science, or QA automation
• Experience evaluating agents with tool calls and multi-step tasks
• Knowledge of LLM security and red-teaming frameworks (OWASP LLM Top 10, garak)
• Experience fine-tuning models or training reward models

Tech Stack

Benefits & Perks