Principal Engineer - MCP or ACP Layer (API & AI Orchestration)

Overview

Serve as a senior individual contributor and technical leader building ABC Fitness's MCP / ACP connectivity layer, which enables AI agents to interact safely and reliably with business systems. Architect platform services, orchestration patterns, schemas, and control mechanisms for scalable AI execution, including independently leading delivery of an LLM feature area with evaluation strategy and production readiness.

What you'll do

• Architect, build, and evolve the MCP / ACP ecosystem that exposes ABC Fitness capabilities as standardized, agent-consumable tools.
• Design schemas, tool definitions, and abstraction layers that shield AI agents from underlying system complexity and provide structured, hallucination-resistant interfaces.
• Define and implement reusable orchestration patterns for multi-step AI workflows, including tool use, validation, retries, state management, fallback handling, and long-running execution flows.
• Build for deterministic execution by making AI-driven tool interactions predictable, safe, observable, auditable, and reversible where necessary.
• Establish company-wide standards for AI interaction patterns, structured outputs, input/output validation, schema governance, and agent orchestration workflows.
• Design and implement security, permissions, and access-control patterns for non-human actors and autonomous systems, including auditability, rate controls, and operational boundaries.
• Build instrumentation and observability capabilities to trace agent interactions, identify schema friction, measure execution quality, and improve orchestration success rates.
• Partner with API modernization, platform engineering, infrastructure, security, and SRE teams to define AI-ready integration patterns and ensure scalable production operations.
• Lead technical design reviews and influence architecture decisions across teams building AI-native systems.
• Define and document reusable patterns, reference implementations, SDKs, and paved paths that accelerate safe AI adoption.

What you'll need

• Typically 10+ years of software engineering experience, including deep expertise building distributed backend systems, APIs, orchestration layers, or platform infrastructure.
• Experience designing and operating high-scale backend systems, with strong understanding of state management, concurrency, reliability, and distributed execution patterns.
• Strong understanding of how LLMs interact with tools and workflows, including schema design, structured outputs, predictable execution patterns, and orchestration reliability.
• Deep expertise in abstraction design and wrapping legacy or complex systems with clean, scalable, and reusable interfaces.
• Experience building workflow orchestration systems handling retries, failure states, long-running processes, and multi-step execution logic.
• Ability to define technical standards and influence engineering direction across multiple teams through architecture, design review, and strong engineering judgment.
• Experience designing for observability, operational resilience, auditability, scalability, and performance optimization.
• Excellent cross-functional collaboration skills and ability to work effectively with Engineering, Product, Platform, Infrastructure, Security, and SRE teams.
• A growth mindset and strong learning agility in a rapidly evolving AI and platform engineering landscape.
• Ability to enable other engineering teams, establish scalable paved paths, and create reusable platform capabilities rather than isolated solutions.

Nice to have

• Experience with Model Context Protocol (MCP), ACP patterns, LangChain, LangGraph, or similar AI orchestration frameworks.
• Experience with middleware platforms, API gateways, service mesh architectures, or distributed control planes.
• Experience building internal developer platforms, SDKs, or shared engineering tooling.
• Experience implementing AI observability, tracing, prompt tracking, model evaluation, or orchestration telemetry systems.
• Familiarity with policy enforcement, safe model execution, permissioning systems, and AI governance patterns in production environments.
• Experience helping engineering organizations adopt new technical standards through documentation, mentoring, and reference implementations.

Details

• Work from the Hyderabad office 2-3x a week.
• Location: Hyderabad, India.