Software Engineer, Model Routing & Inference

About this role

Build the inference platform powering Cursor's AI interactions, focusing on making model serving faster, more reliable, and cost-effective at scale. You'll own the complete inference path from routing to failover, handling millions of requests daily across multiple AI providers.

What you'll do

• Design and maintain the inference gateway abstracting multiple provider APIs
• Implement cross-provider failover mechanisms to prevent outages
• Build routing, backpressure, and admission control systems for traffic spikes
• Optimize for cost, latency, and reliability tradeoffs in production
• Handle the inference path for agent sessions, completions, and chat messages
• Scale the system to support millions of AI requests

What they're looking for

• Distributed systems design and implementation
• High-throughput, low-latency system architecture
• Inference serving or traffic routing experience
• Real-time data pipeline design
• Cost and capacity planning optimization
• Production systems at scale
• API abstraction and integration
• Software engineering fundamentals