Director, Data Engineer (Python, Pyspark, MLOPs, LLMOPs, SQL, Databricks)

Overview

Serve as a core engineering leader for Gartner's Tech Market Economics team by architecting next-generation data, analytics, and MLOps platforms. Build scalable AI-powered insight capabilities and establish the engineering systems, telemetry, and automated evaluation frameworks needed for complex AI workflows.

What you'll do

• Architect the destination state for high-volume data pipelines supporting model training and complex, multi-prompt inference workflows at enterprise scale.
• Design evaluation frameworks, guardrails, and automated quality gates to manage consistency, reliability, and variance in non-deterministic AI outputs.
• Establish MLOps/LLMOps practices and system telemetry to monitor pipeline health, detect data and model drift, and ensure continuous performance in production.
• Partner with Data Science, Product, and Research leaders to define technical roadmaps and move prototype models into scalable, production-grade capabilities.
• Architect solutions for data isolation, metadata management, and data classification across internal, external, and unstructured data sources.
• Establish engineering standards, reusable code design principles, and CI/CD governance frameworks that accelerate engineering velocity while reducing technical debt.
• Lead the modernization of production data platforms by identifying architectural bottlenecks, improving system resilience, and expanding end-to-end automation.

What you'll need

• 9+ years of progressive experience in data engineering, advanced analytics, MLOps, and complex distributed system architecture.
• Bachelor's degree in Computer Science, Data Science, Software Engineering, or a related quantitative discipline.
• Expertise in designing and optimizing robust enterprise pipelines, data workflows, and modern orchestration frameworks for complex AI/ML systems.
• Proven track record of deploying, monitoring, and scaling AI models in production, including experience managing output consistency and implementing automated regression/evaluation suites.
• Hands-on experience architecting scalable data systems within major cloud environments and modern enterprise data platforms.
• Strong proficiency in Python, with deep experience using numerical processing, orchestration, and automated testing libraries to develop scalable enterprise solutions.
• Proven ability to build evaluation tooling, system monitoring dashboards, and alerting frameworks to catch performance degradation in production.
• Drive enterprise value by creating scalable systems, reusable assets, and durable architectures rather than just executing isolated tasks.
• Take loosely defined requirements or prototype models and translate them into actionable engineering architecture.
• Balance multiple priorities while effectively managing risk, technical debt, and architectural trade-offs.
• Translate complex architectural concepts, system behaviors, and MLOps methodologies to diverse business, research, and technical audiences.

Nice to have

• Master's degree in Computer Science, Data Science, Software Engineering, or a related quantitative discipline.

Details

• Location: Gurgaon.
• The source describes Gartner's work environment as hybrid, with flexibility to work virtually when productive and get together with colleagues in a community.