Software Engineer, Web Crawling

About this role

Exa, a well-funded AI lab, seeks a Software Engineer to build Google-scale web crawling infrastructure that processes hundreds of millions of pages daily to power next-generation search. You'll design and optimize distributed systems handling dynamic content, anti-bot detection, and intelligent scheduling across thousands of domains.

What you'll do

• Build distributed crawler systems capable of processing 100M+ pages per day
• Optimize crawl politeness, rate limiting, and domain-specific scheduling
• Design systems to detect and handle dynamic content, JavaScript rendering, and anti-bot measures
• Create intelligent crawl prioritization algorithms for maximum coverage efficiency
• Scale and maintain massive web crawling infrastructure

What they're looking for

• Web crawler development and scaling
• High-performance languages (C++, Rust)
• TypeScript and Playwright
• Chrome DevTools Protocol (CDP)
• Distributed systems design
• Performance optimization
• JavaScript rendering and dynamic content handling
• System architecture and scalability

Benefits

• Premium healthcare (medical, dental, vision)
• Fertility benefits
• 16 weeks fully paid parental leave
• Monthly wellness stipend
• Visa sponsorship available (STEM OPT, H1B, O1, E3)