Backend Software Engineer (Evals)

About this role

Join OpenAI's Support Automation team to design and build evaluation infrastructure for AI-powered support systems. You'll develop backend services and evaluation pipelines that measure quality and enable continuous monitoring of support automation at scale, working closely with data science and research teams.

What you'll do

• Design reliable, reproducible, and extensible evaluation pipelines for support automation
• Build infrastructure for continuous monitoring frameworks including regression detection and golden dataset management
• Develop backend services and APIs supporting intelligent automation and knowledge systems
• Integrate and structure data across platforms for downstream AI workflows
• Collaborate with data, research, and engineering teams to integrate OpenAI models into high-leverage workflows
• Own the full development lifecycle of new backend systems and internal platform capabilities

What they're looking for

• Backend engineering (4+ years at product companies)
• Python, FastAPI, and PostgreSQL
• Distributed systems and API design
• ML/LLM evaluation methods and metrics
• AI agent and application development
• Data pipeline design and scaling
• Production ML systems and performance measurement
• Pragmatic iterative development approach