Principal Application Support Engineering – AI Products & Agents

Overview

Principal Engineer leading operations, reliability and AI platforms for enterprise-scale production systems, cloud platforms and AI-powered applications at Lilly in Hyderabad.

What you'll do

• Define and drive the operational engineering strategy for enterprise applications, cloud-native platforms and AI-enabled systems.
• Establish standards for reliability, scalability, availability, performance, security and operational excellence.
• Lead architecture reviews focused on supportability, observability, resiliency, disaster recovery and long-term sustainability.
• Provide technical leadership during major incidents, service disruptions and high-severity production events.
• Influence product and engineering roadmaps to address technical debt, operational risks and reliability improvements.
• Own reliability outcomes through SLOs, SLIs, error budgets and operational health metrics.
• Drive adoption of Site Reliability Engineering and operational excellence practices across teams.
• Lead complex root cause investigations involving distributed systems, cloud platforms, integrations, databases and AI services.
• Establish operational readiness reviews, production acceptance criteria, release governance and supportability standards.
• Define and implement resiliency engineering practices including failover validation, disaster recovery testing, capacity planning and business continuity programs.
• Drive continuous improvement through incident trend analysis, reliability metrics and operational insights.
• Provide technical leadership for AI-powered applications, copilots, virtual assistants, chatbots and agent-based systems in production.
• Establish operational standards for LLMs, RAG, AI agents, vector databases, MCP ecosystems and orchestration platforms.
• Lead investigations involving AI model failures, retrieval issues, prompt orchestration, tool-calling workflows, agent interactions and integration failures.
• Define monitoring strategies for AI quality, token consumption, latency, throughput, reliability and operational health.
• Drive AI governance, responsible AI practices, security, compliance and operational risk management.
• Lead initiatives to eliminate manual operational effort through automation, intelligent workflows, self-healing capabilities and autonomous remediation.
• Design and implement operational automation using scripting, Infrastructure as Code, CI/CD pipelines, observability platforms and AIOps solutions.
• Establish enterprise runbooks, playbooks, operational standards and knowledge management frameworks.
• Drive adoption of platform engineering principles to improve reliability, developer productivity and operational efficiency.
• Partner with Engineering, Product, Platform, Security, Compliance and Operations teams to improve service quality and customer outcomes.
• Mentor engineers and technical leaders in reliability engineering, distributed systems, cloud operations and AI platform support.
• Communicate technical risks, operational health and strategic recommendations to technical and executive stakeholders.
• Lead cross-functional initiatives that improve reliability, resilience, supportability and operational maturity.

What you'll need

• 10+ years of experience in Software Engineering, Site Reliability Engineering, DevOps, Platform Engineering, Cloud Engineering, Production Operations or related disciplines.
• Proven experience operating and supporting large-scale enterprise production environments.
• Experience leading reliability, operational excellence and platform transformation initiatives.
• Demonstrated success driving operational improvements across multiple products, systems and teams.
• Experience operating within regulated industries such as Life Sciences, Healthcare, Financial Services or similar environments.
• Deep expertise in distributed systems, microservices, APIs, event-driven architectures and cloud-native platforms.
• Strong understanding of software architecture, performance engineering, resiliency, security and operational excellence.
• Advanced troubleshooting capabilities across applications, integrations, databases, networking, infrastructure and cloud services.
• Experience leading major incident management, post-incident reviews, root cause analysis and corrective action programmes.
• Strong understanding of governance, compliance, risk management and secure engineering practices.
• Expert knowledge of AWS, Azure and/or GCP cloud platforms.
• Experience designing and supporting highly available, fault-tolerant, enterprise-scale cloud architectures.
• Strong expertise with Kubernetes, Docker and container orchestration technologies.
• Experience with Infrastructure as Code technologies such as Terraform, CloudFormation, Bicep or similar platforms.
• Deep understanding of networking, identity management, security and cloud governance.
• Experience implementing platform engineering principles and Internal Developer Platforms.
• Strong knowledge of high availability, failover, disaster recovery, backup and resilience engineering concepts.
• Expert understanding of SRE principles, reliability engineering, error budgets, SLOs and SLIs.
• Strong experience with GitHub Actions, Azure DevOps, Jenkins, GitLab CI/CD and modern deployment automation platforms.
• Proven experience implementing self-healing systems, automated remediation and operational automation solutions.
• Deep expertise with observability platforms such as OpenTelemetry, Grafana, Datadog, Splunk, ELK, Prometheus, Dynatrace or equivalent technologies.
• Experience implementing AIOps, intelligent alerting, event correlation and operational analytics platforms.
• Strong automation and scripting capabilities using Python, PowerShell, Bash, JavaScript or similar technologies.
• Strong understanding of Generative AI ecosystems and enterprise AI architecture patterns.
• Experience supporting LLM-based applications, RAG, AI agents and multi-agent systems, agentic workflows, semantic search platforms, vector databases and AI orchestration frameworks.
• Familiarity with Azure OpenAI, AWS Bedrock, Google Vertex AI, OpenAI APIs, Anthropic Claude, Azure AI Services or similar AI platforms.
• Understanding of MCP Server architecture, MCP Clients, tool calling, enterprise AI integrations and AI platform operations.
• Experience monitoring AI reliability, latency, throughput, token consumption, usage patterns and operational health metrics.
• Knowledge of Responsible AI, AI governance, security, compliance, model evaluation and operational risk management.

Nice to have

• Experience leading enterprise-wide operational transformation initiatives.
• Experience building reliability engineering or operational excellence programmes.
• Experience supporting Kubernetes-based cloud-native architectures at scale.
• Familiarity with LangChain, LangGraph, Semantic Kernel, AutoGen, CrewAI, OpenAI Agents SDK or similar orchestration frameworks.
• Experience supporting MCP-based enterprise integrations and AI ecosystems.
• Experience implementing AIOps capabilities, autonomous operations and intelligent automation solutions.
• Cloud, Kubernetes, DevOps, SRE or AI/Generative AI certifications.
• Experience supporting mission-critical, global, 24x7 production environments.

Details

• Location: Hyderabad, India.
• Non-standard work hours: the team supports continuous operations across two shifts, with some work on weekends and holidays.
• Candidates should be open to working different shifts, 6AM to 2PM and 2PM to 11PM, when required.