MSD logo

Specialist, Inference Engineering

MSD
NewPosted today

LOCATION

Hyderabad · Hybrid

EXPERIENCE

3 - 6 Years

TYPE

FullTime

SKILLS REQUIRED

PythonKubernetesMachine LearningCI/CDPerformance OptimizationObservabilityCUDA

Job description

Overview

Deploy, tune and upgrade the models hosted on the platform, making each model fast, cost-efficient and correct on GPUs, with model releases automated through pipelines.

What you'll do

  • Deploy and configure models on vLLM, SGLang or Triton, including engine settings, quantisation, tensor and expert parallelism, multi-node layouts and caching.
  • Make each model behave as clients expect on OpenAI- and Anthropic-compatible APIs: chat templates, tool-call and reasoning parsers, structured output, and correct token usage in every response so the gateway can meter and cost each request.
  • Handle very large checkpoints (hundreds of GB): pre-stage and cache weights, cut start-up time, and plan rollbacks around it.
  • Benchmark latency (time to first token, inter-token latency), throughput and cost per token, and tune deployments to meet SLOs at the best GPU utilisation.
  • Build and maintain release pipelines from the Model Bakery: packaging, weight checks (safetensors only, scanned), validation tests, canary releases and rollback.
  • Keep serving engines and container images current, with CUDA versions compatible with the host drivers and security fixes applied promptly.
  • Validate model behaviour after each change, working with the evaluation and observability teams.
  • Write runbooks, take part in on-call for the serving layer, and support the apiGPTeal team when they onboard new models to the gateway.

What you'll need

  • Bachelor's degree in information technology, Computer Science or related field
  • 3-6 years of hands-on experience working in full-stack development
  • ML engineering, MLOps or platform engineering, with hands-on experience serving models on GPUs
  • Practical experience with vLLM, SGLang, TensorRT-LLM or Triton
  • Python, Linux and containers; working Kubernetes knowledge
  • Understanding of transformer inference: tokens, batching, KV cache, quantisation, multi-GPU parallelism and mixture-of-experts models
  • Familiarity with OpenAI-compatible chat APIs, including tool calling
  • Experience with CI/CD and infrastructure as code

Nice to have

  • Profiling GPU workloads (Nsight, DCGM) and CUDA basics
  • Experience with small classifier, guardrail, routing or embedding models as well as large LLMs
  • AWS GPU instances and SageMaker; Hugging Face model packaging and model registries
  • Cybersecurity: ML supply-chain security (model and container image scanning, SBOMs, signed artefacts), vulnerability management, and LLM application threats such as prompt injection (OWASP Top 10 for LLM Applications)

Details

  • Location: Hyderabad (Hitec City Raidurg), Telangana, India
  • Flexible work arrangement: Hybrid
  • No travel required
  • Visa sponsorship: No
  • Relocation: Domestic
  • Employee status: Regular

Read the full description and apply on the company’s own careers page.

Stay safe

Hiring on Abekus is free for applicants

We never charge a fee, and employers are prohibited from doing so. If a recruiter asks for payment, please report them right away.

Specialist, Inference Engineering