OpenAI
Software Engineer, Model Runtime
About this role
Build the inference runtime that executes frontier LLM models on OpenAI's custom silicon, optimizing for throughput, latency, and hardware utilization. You'll design scheduling, memory management, and distributed execution strategies while collaborating across kernel, compiler, and hardware teams to deliver production-grade model serving infrastructure.
What you'll do
- Design and implement LLM inference runtime for custom silicon, including scheduling, batching, and KV-cache management
- Develop distributed execution strategies across chips and racks, including model partitioning and synchronization
- Optimize end-to-end latency, throughput, memory efficiency, and hardware utilization across model architectures
- Partner with kernel, compiler, and silicon teams to co-design interfaces and eliminate performance bottlenecks
- Build profiling, observability, and benchmarking tools to measure and improve runtime behavior
- Debug correctness, performance, and reliability issues spanning model code, runtime, communication, and hardware
What they're looking for
- Systems programming in C++, Rust, or Python
- LLM inference fundamentals (prefill/decode, batching, KV-cache, model parallelism)
- Distributed systems and runtime design
- Performance profiling and hardware-software stack debugging
- Quantitative reasoning about latency, throughput, and compute efficiency
- Compiler or kernel optimization experience
- Cross-team collaboration on complex technical problems
- Production systems reliability and observability
Opens the application — the Jobs AI extension fills it for you. Set up autofill
Opens the official application on the employer’s site. No login required.
OpenAI
OpenAI builds AI infrastructure and products, including large-scale data center campuses for AI computing and generative AI applications for enterprise customers. The company is hiring civil engineers, project engineers, electrical design engineers, data center R&D engineers, and AI deployment engineers to expand its infrastructure capabilities and help customers deploy AI solutions.
View all jobs at OpenAILikely interview questions
- Walk us through how you'd design scheduling and continuous batching for an LLM inference runtime targeting custom silicon.
- Describe your experience optimizing a complex system across multiple hardware-software layers. What bottlenecks did you identify and how did you resolve them?