Senior Data Engineer - Spark / Kafka / NiFi

Overview

Senior Data Engineer to develop and support large-scale batch and real-time data platforms using Apache Spark, Apache Kafka, and Apache NiFi.

What you'll do

• Develop, test, validate, and maintain scalable batch and real-time data processing applications using Spark, Kafka, and NiFi.
• Build validation suites for high-performance, fault-tolerant, distributed systems handling large data volumes.
• Implement validation for Spark batch and streaming applications for high-volume dataset processing.
• Validate and support Apache NiFi ingestion, transformation, and routing workflows.
• Build reliable real-time and near-real-time data pipelines with replay, recovery, checkpoint, and failure handling.
• Monitor production environments and troubleshoot incidents with root-cause analysis.

What you'll need

• 6+ years of hands-on software deployment, validation, testing, and development on large-scale data-intensive applications.
• Strong expertise in Apache Spark (Batch and Structured Streaming).
• Strong experience with Apache Kafka and event-driven architectures.
• Experience developing, testing, and supporting Apache NiFi data flows.
• Proficiency in Scala, pyspark, or Python.
• Hands-on experience with Linux, Git, shell scripting, and CI/CD pipelines.
• Strong SQL and data modeling skills.

Details

• Location: Pune, India.