Software Engineer- GPU Fabric Observability

About this role

Baseten is seeking a Lead Software Engineer to design and build an observability and root-cause analysis system for GPU fabric infrastructure. This role requires solving complex distributed systems challenges to diagnose failures across NVIDIA networks, RDMA paths, and inference services at large scale.

What you'll do

• Own GPU fabric observability and root-cause analysis architecture end-to-end
• Build high-volume telemetry pipelines across switches, NICs, hosts, GPUs, Kubernetes, and inference services
• Model network topology, flow paths, service ownership, and failure domains
• Distinguish between fabric faults and failures in host, NIC, GPU, kernel, driver, RDMA, scheduler, and workload layers
• Develop real-time signal reduction, correlation, and anomaly detection systems
• Create operator workflows for incident triage, remediation, and post-incident analysis

What they're looking for

• Distributed systems and streaming telemetry pipeline design
• High-performance networking and RDMA fundamentals
• GPU infrastructure and data center networking
• Low-level diagnostics and root-cause analysis system building
• Handling high-cardinality, high-volume operational data
• Kubernetes and container orchestration
• Real-time correlation and anomaly detection algorithms
• Systems programming and infrastructure software development

Benefits

• Competitive compensation with meaningful equity
• 100% medical, dental, and vision coverage for employee and dependents
• Flexible PTO with company-wide winter break closure
• Paid parental leave
• Fertility and family-building stipend through Carrot
• Company-facilitated 401(k)

Likely interview questions

• Can you describe a production incident you debugged involving multiple infrastructure layers? How did you isolate the root cause?
• Tell us about your experience building or operating telemetry systems at scale. What were the biggest challenges with high-cardinality data?