Staff Software Engineer - AI Traffic & Inference Infrastructure

Bengaluru3d ago

Seniority: Staff

About the role

Please complete the attached <a href="https://coupang.service-now.com/sp?id=kb_article&sysparm_article=KB0010204">Internal Transfer Request Form</a> and submit.    Please make sure to apply with your Coupang e-mail address.    <hr>  Company Introduction   We exist to wow our customers. We know we’re doing the right thing when we hear our customers say, “How did we ever live without Coupang?” Born out of an obsession to make shopping, eating, and living easier than ever, we’re collectively disrupting the multi-billion-dollar e-commerce industry from the ground up. We are one of the fastest-growing e-commerce companies that established an unparalleled reputation for being a dominant and reliable force in South Korean commerce.    We are proud to have the best of both worlds — a startup culture with the resources of a large global public company. This fuels us to continue our growth and launch new services at the speed we have been since our inception. We are all entrepreneurs surrounded by opportunities to drive new initiatives and innovations. At our core, we are bold and ambitious people that like to get our hands dirty and make a hands-on impact. At Coupang, you will see yourself, your colleagues, your team, and the company grow every day.   Our mission to build the future of commerce is real. We push the boundaries of what’s possible to solve problems and break traditional tradeoffs. Join Coupang now to create an epic experience in this always-on, high-tech, and hyper-connected world.     Role Overview As a Staff Engineer on our Coupang intelligent Cloud Infrastructure team, you will design and scale the intelligent nervous system of our CIC Cloud AI platform. You won't just be moving packets; you’ll be building the orchestration and routing layers that ensure our LLMs and foundation models are highly available, low-latency, and cost-efficient. You will own the end-to-end lifecycle of traffic management from global load balancing to hardware-aware request routing across thousands of accelerators. What You Will Do <ul data-pm-slice="3 3 []"> <li> Intelligent Routing: Design and implement sophisticated load-balancing algorithms tailored for AI workloads (training, inference), optimizing request distribution based on model availability, and accelerator health.  </li> </ul> <ul> <li> Inference Orchestration: Architect and evolve our inference infrastructure to support seamless model deployment, auto-scaling, and multi-AZ failover.  </li> </ul> <ul> <li> Performance Engineering: Drive initiatives to minimize tail latency (P95 /P99) and maximize throughput using advanced batching, caching, and streaming token delivery techniques.  </li> </ul> <ul> <li> Fleet Automation: Build robust infrastructure-as-code and CI/CD pipelines to manage dynamic compute fleets, ensuring they automatically scale to meet production and research demands.  </li> </ul> <ul> <li> Observability & Optimization: Leverage deep telemetry data to tune system performance and hardware-agnostic scheduling across diverse GPU/TPU environments.  </li> </ul> <ul> <li> Technical Leadership: Lead cross-functional initiatives across infrastructure and SW team, ML teams, providing mentorship and setting up the long-term technical roadmap for traffic management.  </li> </ul>   Basic Qualifications <ul> <li>Education: Bachelor’s or Master’s degree in Computer Science, Engineering, or a related technical field. </li> </ul> <ul data-pm-slice="3 3 []"> <li> Experience: 8–12 years of progressive software engineering experience, with a heavy emphasis on distributed systems, cloud-native architectures, or platform operations.  </li> </ul> <ul> <li> Programming: Strong proficiency in Go or Python, with a deep understanding of networked systems and performance optimization.  </li> </ul> <ul> <li> Orchestration: Expert-level knowledge of Kubernetes internals (scheduling, controllers) and containerization ecosystems.  </li> </ul> <ul> <li> Traffic Management: Proven experience with load balancing, service mesh, and request routing at scale.  </li> <li> Operational Excellence: A strong "ownership" mindset with a track record of maintaining mission-critical, high-availability systems in production.  </li> </ul>   Preferred Qualifications <ul data-pm-slice="3 3 []"> <li> AI/ML Domain Knowledge: Prior experience building infrastructure specifically for LLM inference or large-scale training clusters.  </li> </ul> <ul> <li> Low-Level Optimization: Familiarity with inference, including mixed precision, kernel tuning, or custom hardware accelerators.  </li> </ul> <ul> <li> Public/Private Cloud: Experience managing hybrid-cloud or multi-AZ deployments across AWS, Azure, or GCP.  </li> </ul> <ul> <li> Compliance: Experience operating in regulated environments with strict security and compliance requirements.  </li> </ul>   Type of work model  <ul> <li data-leveltext="" data-font="Symbol" data-listid="38" data-list-defn-props="{"335552541":1,"335559685":880,"335559991":440,"469769226":"Symbol","469769242":[8226],"469777803":"left","469777804":"","469777815":"hybridMultilevel"}" data-aria-posinset="1" data-aria-level="1">Hybrid</li> </ul> Details to consider  <ul> <li data-leveltext="" data-font="Symbol" data-listid="38" data-list-defn-props="{"335552541":1,"335559685":880,"335559991":440,"469769226":"Symbol","469769242":[8226],"469777803":"left","469777804":"","469777815":"hybridMultilevel"}" data-aria-posinset="2" data-aria-level="1">Those eligible for employment protection (recipients of veteran’s benefits, the disabled, etc.) may receive preferential treatment for employment in accordance with applicable laws.   </li> </ul> Privacy Notice   <ul> <li data-leveltext="" data-font="Symbol" data-listid="35" data-list-defn-props="{"335552541":1,"335559685":720,"335559991":360,"469769226":"Symbol","469769242":[8226],"469777803":"left","469777804":"","469777815":"multilevel"}" data-aria-posinset="1" data-aria-level="1">Your personal information will be collected and managed by Coupang as stated in the Application Privacy Notice located below. <a href="https://privacy.coupang.com/en/land/jobs/">https://privacy.coupang.com/en/land/jobs/</a> </li> </ul>

741,000+ hidden jobs like this

coupanginternal and thousands of companies post here first — often days before LinkedIn or Indeed. Your first 5 applications are free; go Pro to apply without limits.

Everything Pro unlocks:

Unlimited applications — free stops at 5
Track every application in one place
Apply straight to the source, one click
Save & organize roles you love
Roles pulled from company boards before the big sites

Weekly

$9.99

$4.99/week

For an active search. Cancel anytime.

Get Weekly

Monthly

$24.99

$12.99/month

The smart pick. Save 35% vs weekly.

Get Monthly

Lifetime

$99

$49.99once

Pay once. Every future feature, forever.

Get Lifetime