Software Engineer, ML Platform

• Architect end-to-end model serving pipelines and integrate new model architectures from our research team into our core, high-throughput inference engine.
• Build robust and sophisticated scheduling systems to manage jobs based on cluster availability and user priority, ensuring we optimally leverage thousands of expensive GPU resources.
• Design and implement dynamic, traffic-based systems for hotswapping models on our GPU workers to maximize fleet efficiency and meet product SLOs.
• Own the end-to-end CI/CD pipelines, including creating a resilient artifact store to manage all model checkpoints across multiple versions and providers.
• Develop and maintain user-friendly APIs and interaction patterns that empower our product and research teams to ship groundbreaking features at high velocity.
• Manage and optimize our complex inference workloads at scale, operating across multiple clusters and hardware providers.

• 5+ years of professional engineering experience with deep, hands-on proficiency in Python and complex distributed systems architecture.
• Extensive, practical experience building and managing systems at scale, specifically with queues, scheduling, traffic-control, and fleet management.
• Deep expertise in our core infrastructure stack: Linux, Docker, and Kubernetes.
• Strong experience with Redis, S3-compatible storage, and public cloud platforms (AWS).

• Experience with high-performance, large-scale ML systems (managing >100 GPUs).
• Deep familiarity with PyTorch and CUDA.
• Experience with modern networking stacks, including RDMA (RoCE, Infiniband, NVLink).
• Familiarity with FFmpeg and multimedia processing pipelines.

Compensation

About Luma