Senior Principal AI Engineer

Overview

Design and operate distributed training systems for large neural networks across GPU clusters. Partner with research and applied ML teams to productionize reliable, high-performance large-model training pipelines.

What you'll do

• Design and operate distributed training systems across GPU clusters.
• Optimize multi-node, multi-GPU execution for throughput and utilization.
• Diagnose bottlenecks across compute, memory, and network resources.
• Improve training stability and fault tolerance at scale.
• Build and optimize GPU cluster orchestration.
• Scale large-model training with distributed frameworks.
• Own launch configurations, failure recovery, and performance tuning.

What you'll need

• Deep hands-on experience with distributed systems or ML systems.
• Production experience with PyTorch distributed training.
• Experience running large-scale workloads on GPU clusters.
• Strong understanding of data, tensor, and pipeline parallelism.
• Low-level understanding of GPU communication and networking.
• Experience with Slurm, Kubernetes, Ray, or RunAI.
• Experience with NCCL, RDMA, InfiniBand, NVLink, Megatron-LM, or DeepSpeed.

Nice to have

• Experience with large language models or foundation models.
• HPC engineering experience transitioning into ML.

Details

• Remote - USA.
• Salary range: USD 141,400 to USD 226,300 yearly.
• Remote and/or hybrid work is available depending on the position.