Lead - Data Engineering

Overview

Lead Data Engineering role working as an AI Data Engineer to architect high-performance data infrastructure for autonomous systems.

What you'll do

• Design and scale RAG pipelines to turn unstructured IT logs and documentation into vector embeddings.
• Monitor and optimize the health and performance of vector databases for sub-second retrieval.
• Build knowledge graphs and semantic layers to provide agents with infrastructure context.
• Create automated data guardrails to detect noise, bias, and PII before it reaches the model.
• Identify and resolve data quality issues from raw, messy data sources.
• Build, deploy, and maintain CI/CD pipelines for data infrastructure and keep context reliable.

What you'll need

• Expertise in ELK (ElasticSearch, Logstash, Kibana).
• Expertise in Phython.
• Experience with data mining, data storage, and ETL processes.
• Experience developing data pipelines and tooling.
• Experience with relational and NoSQL databases, including PostgreSQL, DB2, and MongoDB.
• Excellent problem-solving, analytical, and critical thinking skills.

Nice to have

• Experience as a ELK Observability SME / Data Engineer.
• Experience in data modelling for conceptual models used in business processes.
• Professional certification (example given: Open Certified Technical Specialist with Data Engineering Specialization).
• Cloud platform certification (examples given: AWS, Elastic, Google Cloud, or Azure data engineering certs).
• Understanding of social coding and IDEs (examples given: GitHub, Visual Studio).
• Degree in a scientific discipline (examples given: Computer Science, Software Engineering, or Information Technology).

Details

• Location: Mumbai, Maharashtra, India.