Overview
Act as software detectives, providing a dynamic service that identifies and solves issues within multiple components of critical business systems. The role focuses on designing, building and maintaining robust, scalable and highly observable data pipelines that ensure reliable data flow across the enterprise.
What you'll do
- Build and maintain monitoring dashboards for data pipelines (latency, failures, SLAs).
- Design, develop, monitor and troubleshoot Apache NiFi data flows for enterprise-scale data integration and ingestion.
- Implement distributed tracing and logging across ingestion, processing and storage layers.
- Configure real time alerts using tools like Grafana, Datadog, Azure Monitor, CloudWatch, etc.
- Enhance pipeline performance through profiling, optimization and scalability improvements.
- Ensure consistent metadata capture using Data Catalog tools.
- Investigate pipeline failures and performance issues.
- Conduct RCA and implement preventive automation.
- Work closely with Data Engineers, SRE/DevOps, Platform Teams and Product Owners.
- Provide insights into pipeline health to business stakeholders.
- Influence the development of platform-wide observability standards.
What you'll need
- Bachelor's/master's degree in computer science or equivalent with score 6 GPA and above.
- 4–8 years of experience in Data Engineering, Observability or Platform Engineering.
- Strong understanding of modern data pipelines (batch and streaming).
- Experience with at least one cloud ecosystem: Azure / AWS / GCP.
- Proficiency in monitoring/observability platforms: Datadog, Grafana, CloudWatch, Azure Monitor, Dynatrace, etc.
- Hands-on experience with streaming frameworks (Kafka, Spark Streaming, EventHub, etc.).
- Mandatory hands-on experience with Apache NiFi, including flow design and development, performance tuning, troubleshooting and production support, monitoring and observability, and error handling and flow optimization.
- Hands-on experience with Apache NiFi, Kafka, Spark Streaming, EventHub or equivalent streaming technologies.
- Good understanding of CI/CD, Docker, Kubernetes, APIs and Microservices.
- Programming/scripting skills in Python, SQL, Bash or similar.
- Experience with modern monitoring tools in the IT Ops domain.
- Proficiency in tools for logging, alerting and monitoring on-prem/cloud infrastructure, services and performance.
- Good understanding of analysis, design, implementation, testing, monitoring and SDLC.
- Strong knowledge of version control systems (e.g. GitHub).
- Good database knowledge, including understanding of RDBMS concepts such as nested queries and records maintenance.
- Excellent written and verbal communication skills.
- Agile methodologies and project execution.
- Minimum 7.5 years of experience is required; 15 years full time education is required.
Nice to have
- Knowledge of IT Operations space and Site Reliability Engineer job role.
Details
- Location: Bengaluru office.
- Educational qualification: 15 years full time education.
Read the full description and apply on the company’s own careers page.