Overview
Deploy, tune and upgrade the models hosted on the platform, making each model fast, cost-efficient and correct on GPUs, with model releases automated through pipelines.
What you'll do
- Deploy and configure models on vLLM, SGLang or Triton, including engine settings, quantisation, tensor and expert parallelism, multi-node layouts and caching.
- Make each model behave as clients expect on OpenAI- and Anthropic-compatible APIs: chat templates, tool-call and reasoning parsers, structured output, and correct token usage in every response so the gateway can meter and cost each request.
- Handle very large checkpoints (hundreds of GB): pre-stage and cache weights, cut start-up time, and plan rollbacks around it.
- Benchmark latency (time to first token, inter-token latency), throughput and cost per token, and tune deployments to meet SLOs at the best GPU utilisation.
- Build and maintain release pipelines from the Model Bakery: packaging, weight checks (safetensors only, scanned), validation tests, canary releases and rollback.
- Keep serving engines and container images current, with CUDA versions compatible with the host drivers and security fixes applied promptly.
- Validate model behaviour after each change, working with the evaluation and observability teams.
- Write runbooks, take part in on-call for the serving layer, and support the apiGPTeal team when they onboard new models to the gateway.
What you'll need
- Bachelor's degree in information technology, Computer Science or related field
- 3-6 years of hands-on experience working in full-stack development
- ML engineering, MLOps or platform engineering, with hands-on experience serving models on GPUs
- Practical experience with vLLM, SGLang, TensorRT-LLM or Triton
- Python, Linux and containers; working Kubernetes knowledge
- Understanding of transformer inference: tokens, batching, KV cache, quantisation, multi-GPU parallelism and mixture-of-experts models
- Familiarity with OpenAI-compatible chat APIs, including tool calling
- Experience with CI/CD and infrastructure as code
Nice to have
- Profiling GPU workloads (Nsight, DCGM) and CUDA basics
- Experience with small classifier, guardrail, routing or embedding models as well as large LLMs
- AWS GPU instances and SageMaker; Hugging Face model packaging and model registries
- Cybersecurity: ML supply-chain security (model and container image scanning, SBOMs, signed artefacts), vulnerability management, and LLM application threats such as prompt injection (OWASP Top 10 for LLM Applications)
Details
- Location: Hyderabad (Hitec City Raidurg), Telangana, India
- Flexible work arrangement: Hybrid
- No travel required
- Visa sponsorship: No
- Relocation: Domestic
- Employee status: Regular
Read the full description and apply on the company’s own careers page.