Senior Data Engineer

Overview

Senior Data Engineer responsible for developing, validating, testing, and supporting large-scale batch and real-time data platforms using Apache Spark, Kafka, and NiFi.

What you'll do

• Develop and maintain scalable batch and real-time data applications.
• Build validation suites for distributed Spark applications and data platforms.
• Support NiFi data ingestion, transformation, and routing workflows.
• Implement replay, recovery, checkpoint management, and failure-handling solutions.
• Optimize performance, throughput, scalability, reliability, and resource utilization.
• Monitor production systems and participate in troubleshooting and incident resolution.
• Mentor junior engineers and contribute to code reviews, documentation, and CI/CD initiatives.

What you'll need

• 6+ years of software deployment, validation, testing, and development experience on large-scale data-intensive applications.
• Bachelor's degree in Computer Science, Engineering, or a related technical field.
• Strong expertise in Apache Spark Batch and Structured Streaming.
• Strong experience with Apache Kafka and event-driven architectures.
• Experience developing, testing, and supporting Apache NiFi data flows.
• Proficiency in Scala, PySpark, or Python.
• Strong SQL, data modeling, Linux, Git, shell scripting, and CI/CD skills.

Nice to have

• Experience with observability, monitoring, and logging platforms.
• Knowledge of data governance and metadata management.
• Experience with Docker and Kubernetes.
• Knowledge of performance engineering, resilience testing, and production readiness assessments.
• Experience in regulated, transaction-processing, or large-scale enterprise environments.

Details

• Location: Pune, India.