Machine Learning Scientist

About this role

Arena Intelligence seeks a Machine Learning Scientist to design experiments and develop novel evaluation methodologies for understanding AI model behavior through human preference data. You'll conduct rigorous research on model performance across multiple dimensions while collaborating with engineers and product teams to translate findings into production systems.

What you'll do

• Design and conduct experiments evaluating AI model behavior across reasoning, style, robustness, and user preference dimensions
• Develop new metrics, methodologies, and evaluation protocols beyond traditional benchmarks
• Analyze large-scale human voting and interaction data to uncover model performance insights
• Collaborate with engineers to implement and scale research findings into production
• Author internal reports and external publications contributing to ML research community
• Partner with model providers on evaluation questions and responsible model testing

What they're looking for

• Training large-scale models including reward models and RLHF/DPO fine-tuning
• Statistical rigor in experiment design and analysis
• Python proficiency with PyTorch, JAX, or TensorFlow
• LLM and modern deep learning architecture expertise
• Real-world data analysis and custom metric design
• Research publication and open-source contribution experience
• Cross-functional collaboration with engineering and product teams
• Rapid prototyping balanced with methodological rigor

Benefits

• Competitive compensation and equity
• Remote-friendly work in Bay Area
• Opportunity to contribute to open science and reproducibility
• Collaboration with researchers from UC Berkeley, Google, Stanford, and DeepMind
• Work on problems with real-world AI impact