Office Hours
Software Engineer- Benchmarking
- Confirmed live in the last 24 hours
- $160k–$210k
- Mid level
- Full-time
- San Francisco, NYC, or Fully Remote
- 4+ yrs exp
- Added 1 month ago
About this role
Office Hours is seeking a Software Engineer to develop and maintain the infrastructure behind their AI model evaluations. You'll collaborate closely with researchers to build data pipelines, evaluation environments, and publish results, ensuring consistency and reproducibility across various models and benchmarks. This role requires strong engineering skills, a focus on data rigor, and experience with containerization.
What you'll do
- Prepare and maintain benchmark datasets.
- Build and maintain evaluation pipelines.
- Create containerized evaluation environments.
- Support model experiments and fine-tuning.
- Develop scoreboards and leaderboards.
- Build analysis tools to identify failure modes.
What they're looking for
- Python
- Docker
- Data cleaning
- Data maintenance
- Reproducible environments
- React
- Next.js
Benefits
- Competitive salary and equity
- Medical, dental, and vision coverage
- 401(k)
- Wellness stipend
- Paid time off
- Company off-sites
Opens the application — the Jobs AI extension fills it for you. Set up autofill
Opens the official application on the employer’s site. No login required.
Office Hours
Likely interview questions
- Describe your experience preparing and cleaning datasets for analysis.
- Walk me through a time you built a reproducible execution environment.