Skip to main content

Artificial Analysis

Forward Deployed Engineer - Language Models

San Francisco (On-site)fulltimemidAdded today

About this role

Forward Deployed Engineer to operate and maintain Artificial Analysis's language model benchmarking pipeline while serving as the primary technical contact for AI lab customers. You'll onboard models, run evaluations, debug issues across the stack, and communicate results to sophisticated industry partners.

What you'll do

  • Operate and maintain Python-based language model benchmarking pipeline end-to-end, including model onboarding and evaluation execution
  • Debug and resolve issues across APIs, data pipelines, and benchmark outputs with quick turnaround
  • Serve as primary technical contact for AI lab customers via Slack and video conferencing, explaining results and fielding methodology questions
  • Monitor benchmark runs for anomalies, investigate discrepancies, and ensure result accuracy and integrity
  • Maintain documentation of processes, known issues, and model-specific configurations
  • Collaborate with engineering team to identify pipeline improvements and contribute to process refinements

What they're looking for

  • Python programming and complex codebase navigation
  • AI/ML model API integration (OpenAI, Anthropic, Google, Meta, etc.)
  • Advanced debugging across APIs, data pipelines, and distributed systems
  • Technical communication and explanation of complex concepts to stakeholders
  • Customer relationship management and issue ownership
  • Operational excellence and attention to detail
  • LLM inference infrastructure knowledge (preferred)
  • AI evaluation and benchmarking methodologies (preferred)

Benefits

  • Direct influence on how leading AI labs develop their models
  • Opportunity to gain deep expertise in AI evaluation and the language model ecosystem
  • Work at the cutting edge of AI benchmarking and evaluation
  • Remote or San Francisco office flexibility
  • Backing by prominent AI industry leaders including Andrew Ng, OpenAI executives, and Hugging Face founders
Apply with Autofill

Opens the application — the Jobs AI extension fills it for you. Set up autofill

Opens the official application on the employer’s site. No login required.

Artificial Analysis

Artificial Analysis builds AI benchmarking infrastructure that evaluates and compares language models and media generation systems. The company is hiring Solutions Engineers to operate benchmarking pipelines, manage evaluations, and serve as technical liaisons between its infrastructure and AI lab customers.

View all jobs at Artificial Analysis

Likely interview questions

  • Walk us through a time you debugged a complex issue spanning multiple systems or APIs. How did you approach it and resolve it?
  • Describe your experience working directly with AI/ML model APIs. Which ones have you worked with and what challenges did you encounter?
Unlock all 7 questions free — and practice them live →