Senior Data Engineer - Spark / Kafka / NiFi

Overview

Senior Data Engineer to develop, validate, and support large-scale batch and real-time data platforms using Apache Spark, Apache Kafka, and Apache NiFi.

What you'll do

• Develop, test, validate, and maintain scalable batch and real-time data processing applications.
• Build validation suites for fault-tolerant distributed systems handling large data volumes.
• Create reusable frameworks, libraries, and shared platform components.
• Design streaming pipelines for real-time and near-real-time processing.
• Implement data replay, recovery, checkpoint management, and failure handling.
• Optimize performance, throughput, and resource utilization and support production operations.

What you'll need

• 6+ years of hands-on software deployment, validation, testing, and development on large-scale data-intensive applications.
• Strong expertise in Apache Spark (batch and Structured Streaming).
• Strong experience with Apache Kafka and event-driven architectures.
• Experience developing, testing, and supporting Apache NiFi data flows.
• Proficiency in Scala, pyspark, or Python.
• Strong SQL and data modeling skills.
• Hands-on experience with Linux, Git, shell scripting, and CI/CD pipelines.