Anthropic
Research Engineer, Model Evaluations
About this role
Anthropic seeks a Research Engineer to design and operationalize evaluations that measure Claude's capabilities across reasoning, safety, and behavior. You'll build scalable evaluation infrastructure, create dashboards for model monitoring during training, and partner with researchers to turn abstract notions of AI performance into precise, defensible metrics.
What you'll do
- Design evaluations for Claude's reasoning, safety, agentic behavior, and knowledge; create visualizations of results for researchers and leaders
- Build and maintain a distributed evaluation platform that reliably executes hundreds of evals against training checkpoints
- Own monitoring dashboards used during training to detect model regressions and health issues in real-time
- Debug anomalous evaluation results during live training runs and determine whether issues stem from model changes or infrastructure
- Develop tooling, libraries, and workflows to help researchers implement and iterate on evaluations
- Run experiments to characterize how prompting, sampling, and scaffolding affect performance on internal and industry benchmarks
What they're looking for
- Python programming (production and research code)
- Distributed systems and infrastructure design
- Data pipeline development and management
- Technical communication to non-specialist audiences
- Data visualization and dashboarding
- Debugging and incident response under time pressure
- ML evaluation and metrics design
- On-call production support readiness
Benefits
- Remote-friendly with locations in San Francisco, NYC, and distributed options
- Work on foundational AI safety and evaluation research
- Collaborate with committed researchers and engineers advancing beneficial AI
- Impact how the world understands and trusts AI capabilities
Opens the official application on the employer’s site. No login required.
Anthropic
Anthropic builds Claude, an AI assistant, and is hiring for engineering roles across infrastructure, data systems, and security that support both AI research operations and the company's internal technology needs. The company seeks infrastructure engineers, systems integrators, data scientists, and security specialists to build production-scale systems for training data pipelines, financial operations, developer productivity measurement, research infrastructure, and server firmware security.
- Website
- anthropic.com
Likely interview questions
- Describe your experience building or operating distributed systems at scale—what challenges did you face and how did you ensure reliability?
- Walk us through how you've debugged a complex production issue under time pressure. How did you communicate findings to stakeholders?