AI Evaluation Engineer

About this role

Judi Health seeks an AI Evaluation Engineer to design and operate testing frameworks, metrics, and tooling that assess the safety, reliability, and accuracy of AI models and autonomous agents in production. You'll bridge model development and real-world deployment by building evaluation pipelines, continuous benchmarking systems, and self-service tools that enable teams to confidently measure quality impact.

What you'll do

• Build ETL pipelines to collect and reconstruct production conversations from traces, logs, transcripts, and user feedback signals
• Design and own continuous evaluation frameworks that run weekly benchmarks against staging and production to track accuracy, reliability, and safety metrics
• Develop dashboards and monitoring tools that clearly surface whether deployments improve or degrade quality and risk
• Create self-service evaluation APIs and internal tools enabling data scientists and engineers to add scenarios to test suites with minimal infrastructure overhead
• Apply LLM-as-judge patterns and design metrics appropriate for non-deterministic AI systems
• Partner with data science, engineering, and product teams to translate ambiguous product goals into measurable quality targets and success criteria

What they're looking for

• Data engineering and ETL pipeline development
• Designing metrics and evaluation frameworks for AI/ML systems
• Building dashboards and observability tooling
• CI/CD integration and automation
• LLM evaluation techniques and LangSmith familiarity
• Python or similar backend development languages
• Session reconstruction and complex data stitching
• Statistical analysis for probabilistic systems