Overview
Senior Specialist, Inference Engineering at MSD in Hyderabad. The first technical lead of InferenceOps, setting up how self-hosted models are served in production, covering the serving stack, release path from the Model Bakery, capacity planning and the support model, while doing hands-on engineering and guiding two engineers.
What you'll do
- Design and own the inference runtime: serving engines (vLLM, SGLang, Triton/TensorRT-LLM), orchestration (NVIDIA Dynamo or equivalent on Kubernetes), model storage and versioning.
- Design serving for large MoE models: expert parallelism, data-parallel attention, multi-node deployments, and when disaggregated prefill/decode is worth its latency cost.
- Take models from the Model Bakery to production behind apiGPTeal, with repeatable, automated releases, rollbacks and model upgrades.
- Plan GPU capacity: model placement, sizing and utilisation, and decide when to scale or have the gateway spill traffic over to cloud models.
- Own the support model, SLOs and on-call design for the inference layer.
- Act as escalation point in major incidents, approve post-incident reviews and drive their follow-ups.
- Run production changes and incidents through IT service management (ServiceNow or similar) and register each model deployment in the CMDB.
- Own the security of the serving stack: patch serving engines and container images for vulnerabilities and check the provenance of model weights (safetensors only, scanned before release).
- Manage the technical relationship with NVIDIA, hardware vendors and implementation partners: support cases, software licences such as NVIDIA AI Enterprise, and taking over deployments that partners build.
- Work with the hardware and cluster team on node, driver and GPU Operator changes, with the apiGPTeal team on routing and spill-over, and with the HPC and security teams on integration, network and access controls.
- Mentor the team and set engineering standards: infrastructure as code, CI/CD and documentation.
What you'll need
- Bachelor's degree in information technology, Computer Science or related field.
- 7 to 11 years of hands-on experience working in full-stack development.
- Infrastructure or ML engineering, including 2+ years running ML or LLM inference in production.
- Deep hands-on experience with at least one LLM serving engine (vLLM, SGLang, TensorRT-LLM or Triton).
- Experience serving large models across multiple GPUs or nodes.
- Strong Kubernetes skills for GPU workloads, with enough knowledge of the NVIDIA GPU Operator, device plugins and scheduling to debug them.
- Linux, containers and Helm.
- Working knowledge of inference performance: batching, KV/prefix caching, quantisation, tensor and expert parallelism, and latency versus throughput trade-offs.
- Experience running services with SLOs, monitoring, alerting and on-call.
- Infrastructure as code (Terraform or similar) and CI/CD (GitHub Actions or similar).
Nice to have
- NVIDIA Dynamo, NIM or NVIDIA AI Enterprise; disaggregated prefill/decode serving.
- High-speed networking for inference: RDMA, InfiniBand or RoCE, NCCL, NIXL.
- AWS (EKS, EC2 GPU instances, SageMaker) alongside on-premises GPU clusters.
- API gateways for LLM traffic (Kong or similar); OpenAI- and Anthropic-compatible APIs.
- Cybersecurity: threat modelling and hardening of ML serving platforms (container and Kubernetes security, network segmentation, secrets management), model supply-chain security, and LLM-specific threats such as prompt injection and data exfiltration (OWASP Top 10 for LLM Applications).
- Regulated-industry experience (pharma, finance) and model governance.
Details
- Location: Hyderabad (Hitec City Raidurg), India.
- Flexible work arrangement: Hybrid.
- No travel required.
- Employee status: Regular.
- Domestic relocation; no visa sponsorship.
Read the full description and apply on the company’s own careers page.