Hadoop Software Engineer

Overview

Hadoop Software Engineer focused on building and optimizing a next-generation lakehouse platform using Spark/lakehouse table format technologies. The role emphasizes Spark stability/performance and extending Iceberg capabilities.

What you'll do

• Build and optimize the next-generation lakehouse platform.
• Enhance Spark stability, usability, and performance.
• Extend Iceberg capabilities.
• Build internal storage and metadata services for lakehouse architecture.
• Troubleshoot and resolve production issues to minimize downtime.
• Maintain and improve system monitoring, logging, and alerting.

What you'll need

• 8+ years of experience as a successful software engineer.
• Strong programming skills in Java or Scala.
• Deep knowledge of Apache Spark internals (Catalyst optimizer, Tungsten, shuffle mechanics, AQE, memory management).
• Knowledge of file formats like Apache Parquet and table formats such as Apache Iceberg/Delta/Hudi.
• Linux/Unix familiarity including shell scripting and system commands.
• Network principles understanding for distributed environments.
• Ability to use AI coding agents/tools to build production-grade software (e.g., Claude Code, Codex).

Details

• Location: Bengaluru, India.
• Work type: not specified.