Java Spark developer

Overview

Build and maintain scalable data processing pipelines and distributed applications using Java and Apache Spark. The role covers batch and real-time streaming architectures, performance optimization, and collaboration with data and cloud teams.

What you'll do

• Design and maintain Spark-based data ingestion and ETL/ELT pipelines.
• Develop low-latency microservices and distributed modules integrated with messaging platforms.
• Optimize Spark jobs, query execution, resource use, and data serialization.
• Design data schemas using formats such as Parquet, ORC, and Avro.
• Implement data validation, cleansing, governance, and error handling.
• Configure CI/CD pipelines and participate in code reviews and Agile ceremonies.
• Write unit, integration, and automated regression tests.

What you'll need

• 3–6 years of professional software engineering experience.
• 2–4 years building distributed data processing applications with Java and Apache Spark.
• Bachelor’s or Master’s degree in a relevant technical discipline.
• Deep proficiency in Core Java and Apache Spark.
• Strong knowledge of distributed architecture, HDFS, YARN, Hive, and storage systems.
• Experience with Kafka or RabbitMQ, SQL, relational databases, and NoSQL datastores.
• Proficiency with Maven or Gradle, Git, JUnit, and Mockito.

Nice to have

• Experience with AWS, Azure, or Google Cloud data platforms.
• Exposure to Apache Iceberg, Delta Lake, or Apache Hudi.
• Familiarity with Docker, Kubernetes, Airflow, or Luigi.
• Familiarity with Scala or Python.

Details

• Location: Chennai, Tamil Nadu, India.
• Employment type: Full time.