Software Development Engineer 5

Overview

Sr. Staff Data Platform Engineer who acts as the primary architect and visionary for the core data infrastructure powering the enterprise, building internal platforms, frameworks and self-service tools that enable other engineers to move faster.

What you'll do

• Define the 3-5 year technical roadmap for the Data Lakehouse, deciding how storage, compute and metadata layers (e.g., Apache Polaris, Unity Catalog or Datahub Catalog) interact.
• Build internal SDKs, CLI tools and automated orchestration frameworks that abstract cloud complexity via Control Planes and Custom Operators.
• Prototype and benchmark emerging technologies (e.g., specialized Spark extensions) to keep the platform at the forefront of performance and cost-efficiency.
• Architect compliance-by-design systems that automate data lineage, PII masking and fine-grained access control across petabyte-scale environments.
• Set the gold standard for code quality and system design across the company, leading Cross-Functional Architecture Reviews.
• Serve as the final escalation point for the most complex system outages or performance bottlenecks.
• Mentor individuals and foster an Engineering Community, influencing the hiring bar and professional development paths for the data engineering organization.

What you'll need

• 15+ years in software engineering and distributed systems, with at least 4 years in a principal or staff-level capacity leading platform-scale initiatives.
• Strong software engineering fundamentals: algorithms, data structures, system architecture and scalable production applications.
• Experience in the Java/J2EE ecosystem (Spring Boot, Microservices) and Python.
• Experience building platforms, framework engines and APIs that power data movement, rather than only ETL/ELT pipelines.
• Deep internal knowledge of Apache Iceberg, Hudi or Delta Lake (metadata management, manifest files and compaction strategies).
• Extensive experience with cloud architecture and services, including AWS (S3, EMR, Kubernetes, Lambda) and Azure.
• Deep understanding of CI/CD automation, Git Actions, Terraform and modern development tools and frameworks.
• Experience leveraging AI code generation platforms in the SDLC to automate code generation, reviews, unit tests and root-cause analysis of system failures.
• Ability to architect infrastructure supporting AI agent development by enabling vector database integration.
• Proven track record of leading by influence, driving adoption of new technologies across multiple autonomous teams.
• Ability to communicate complex architectural trade-offs to C-suite executives and junior engineers alike.

Nice to have

• Bachelor's or Master's degree in Computer Science (Distributed Systems focus), or equivalent deep industry experience.
• Experience contributing to or deeply customizing open-source data projects (e.g., Spark, dbt).

Details

• Location: Bangalore Office, India.