MSD logo

Senior Specialist, Inference Engineering

MSD
NewPosted today

LOCATION

Hyderabad · Hybrid

EXPERIENCE

7 - 11 Years

TYPE

FullTime

SKILLS REQUIRED

Model servingKubernetesLarge Language ModelsInfrastructure as CodePerformance Optimization

Job description

Overview

Senior Specialist, Inference Engineering at MSD in Hyderabad. The first technical lead of InferenceOps, setting up how self-hosted models are served in production, covering the serving stack, release path from the Model Bakery, capacity planning and the support model, while doing hands-on engineering and guiding two engineers.

What you'll do

  • Design and own the inference runtime: serving engines (vLLM, SGLang, Triton/TensorRT-LLM), orchestration (NVIDIA Dynamo or equivalent on Kubernetes), model storage and versioning.
  • Design serving for large MoE models: expert parallelism, data-parallel attention, multi-node deployments, and when disaggregated prefill/decode is worth its latency cost.
  • Take models from the Model Bakery to production behind apiGPTeal, with repeatable, automated releases, rollbacks and model upgrades.
  • Plan GPU capacity: model placement, sizing and utilisation, and decide when to scale or have the gateway spill traffic over to cloud models.
  • Own the support model, SLOs and on-call design for the inference layer.
  • Act as escalation point in major incidents, approve post-incident reviews and drive their follow-ups.
  • Run production changes and incidents through IT service management (ServiceNow or similar) and register each model deployment in the CMDB.
  • Own the security of the serving stack: patch serving engines and container images for vulnerabilities and check the provenance of model weights (safetensors only, scanned before release).
  • Manage the technical relationship with NVIDIA, hardware vendors and implementation partners: support cases, software licences such as NVIDIA AI Enterprise, and taking over deployments that partners build.
  • Work with the hardware and cluster team on node, driver and GPU Operator changes, with the apiGPTeal team on routing and spill-over, and with the HPC and security teams on integration, network and access controls.
  • Mentor the team and set engineering standards: infrastructure as code, CI/CD and documentation.

What you'll need

  • Bachelor's degree in information technology, Computer Science or related field.
  • 7 to 11 years of hands-on experience working in full-stack development.
  • Infrastructure or ML engineering, including 2+ years running ML or LLM inference in production.
  • Deep hands-on experience with at least one LLM serving engine (vLLM, SGLang, TensorRT-LLM or Triton).
  • Experience serving large models across multiple GPUs or nodes.
  • Strong Kubernetes skills for GPU workloads, with enough knowledge of the NVIDIA GPU Operator, device plugins and scheduling to debug them.
  • Linux, containers and Helm.
  • Working knowledge of inference performance: batching, KV/prefix caching, quantisation, tensor and expert parallelism, and latency versus throughput trade-offs.
  • Experience running services with SLOs, monitoring, alerting and on-call.
  • Infrastructure as code (Terraform or similar) and CI/CD (GitHub Actions or similar).

Nice to have

  • NVIDIA Dynamo, NIM or NVIDIA AI Enterprise; disaggregated prefill/decode serving.
  • High-speed networking for inference: RDMA, InfiniBand or RoCE, NCCL, NIXL.
  • AWS (EKS, EC2 GPU instances, SageMaker) alongside on-premises GPU clusters.
  • API gateways for LLM traffic (Kong or similar); OpenAI- and Anthropic-compatible APIs.
  • Cybersecurity: threat modelling and hardening of ML serving platforms (container and Kubernetes security, network segmentation, secrets management), model supply-chain security, and LLM-specific threats such as prompt injection and data exfiltration (OWASP Top 10 for LLM Applications).
  • Regulated-industry experience (pharma, finance) and model governance.

Details

  • Location: Hyderabad (Hitec City Raidurg), India.
  • Flexible work arrangement: Hybrid.
  • No travel required.
  • Employee status: Regular.
  • Domestic relocation; no visa sponsorship.

Read the full description and apply on the company’s own careers page.

Stay safe

Hiring on Abekus is free for applicants

We never charge a fee, and employers are prohibited from doing so. If a recruiter asks for payment, please report them right away.

Senior Specialist, Inference Engineering