Principal Software Engineer - AI Infrastructure

Own the architecture and technical direction of major components of the training platform for Large Language Models (LLMs), Small Language Models (SLMs), multimodal, code-specific, and open-weight models, ensuring designs meet scalability, reliability, security, and cost-efficiency requirements. Drive ambiguous, multi-quarter initiatives from proposal to production, breaking work into milestones, identifying dependencies, and negotiating ownership and delivery commitments with partner teams across Cloud + AI (C+AI). Build on and extend open-source training frameworks (for example, PyTorch, DeepSpeed, Megatron-LM, and Hugging Face libraries), adapting them for proprietary and open-weight models and contributing improvements back to the community where appropriate. Lead debugging and resolution of complex issues in distributed training jobs and production services; participate in the on-call rotation as a Designated Responsible Individual (DRI), lead incident retrospectives, and improve troubleshooting guides, testing, safe deployment, and monitoring. Mentor engineers and raise the technical bar through design reviews, code reviews, and ownership of the team's test strategy, while championing AI-assisted engineering tools and practices across the software development lifecycle. Bachelor's Degree in Computer Science or related technical field AND 6+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python OR equivalent experience. Master's Degree in Computer Science or related technical field AND 8+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python OR Bachelor's Degree in Computer Science or related technical field AND 12+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python OR equivalent experience. 3+ years building machine learning training infrastructure, such as Graphics Processing Unit (GPU) cluster scheduling, data pipelines, and checkpointing and storage systems, using open-source training frameworks such as PyTorch Fully Sharded Data Parallel (FSDP), DeepSpeed, Megatron-LM, Hugging Face Transformers and Accelerate, or Ray Train; contributions to open-source projects are a strong plus. 2+ years training, fine-tuning, or evaluating open-weight models (for example, Llama, Mistral, Phi, Qwen, or DeepSeek model families) at multi-node scale. 2+ years building or operating Reinforcement Learning (RL) post-training pipelines for Large Language Models (LLMs) and agentic models using verl, slime, MILES, or comparable frameworks, with a demonstrated understanding of the end-to-end training stack, including pretraining and post-training frameworks, Reinforcement Learning Environments (RLEs), rollout inference engines (for example, vLLM or SGLang), evaluation, and checkpointing, and how design choices at each layer affect throughput, stability, and cost. 3+ years serving as an on-call or Designated Responsible Individual (DRI) engineer for production services operating against defined availability or latency targets, including incident response, postmortems, structured logging, distributed tracing, dashboards, alerting, automated validation, and incident playbooks. 2+ years partnering directly with internal or external customers to define requirements, instrument customer experience (for example, adoption, satisfaction, or time-to-first-result metrics), and prioritize engineering work by customer impact. Experience leading technical design across 2+ teams, driving ambiguous multi-quarter initiatives to completion, and mentoring senior engineers.