Town
AI Engineer, Evals & Agent Quality
- Confirmed live in the last 24 hours
- $250k–$300k
- Mid level
- Full-time
- On-site · San Francisco
- Added 1 month ago
About this role
Town is seeking an AI Engineer to build and own the evaluation and quality systems that ensure their personalized AI assistant consistently delivers exceptional performance. This is a unique opportunity to create a foundational framework from scratch, defining what "best" means for their assistant and driving measurable improvements across multiple tools and workflows. You will collaborate closely with product and engineering teams to instrument quality and implement data-driven fixes.
What you'll do
- Develop a comprehensive evaluation system for assistant quality across various platforms.
- Create and maintain golden datasets and labeling loops for continuous validation.
- Design and implement model routing and online evaluation tooling.
- Establish measurable metrics for prompt and system changes.
- Collaborate with engineers to instrument quality and drive improvements.
What they're looking for
- LLM evaluation
- Quality measurement
- Model routing
- Applied ML
- MLE
Opens the application — the Jobs AI extension fills it for you. Set up autofill
Opens the official application on the employer’s site. No login required.
Town
Likely interview questions
- Describe your experience building or owning LLM evaluation systems at scale.
- How do you approach the challenge of defining and measuring 'better' performance for an AI assistant?