Java Scala Developer – Apache Spark & SQL

Overview

Intermediate Applications Development role responsible for participating in the establishment and implementation of new or revised application systems and programs, in coordination with the Technology team, and for designing, developing and optimizing large-scale Spark-based data processing pipelines and regulatory data platforms.

What you'll do

• Identify and define necessary system enhancements, including using script tools and analyzing/interpreting code.
• Consult with users, clients and other technology groups on issues, and recommend programming solutions; install and support customer exposure systems.
• Apply fundamental knowledge of programming languages for design specifications.
• Analyze applications to identify vulnerabilities and security issues, and conduct testing and debugging.
• Serve as advisor or coach to new or lower level analysts.
• Identify problems, analyze information, and make evaluative judgements to recommend and implement solutions.
• Act as SME to senior stakeholders and/or other team members.
• Design, implement and maintain scalable, high-throughput batch and stream data pipelines using Apache Spark, Java/Scala and PySpark.
• Create High-Level Designs (HLD), Low-Level Designs (LLD), data flow diagrams and interface specifications from business and regulatory requirements.
• Perform in-depth data analysis across the data lifecycle, including data sourcing, schema evolution, transformation logic, quality validation and insight generation.
• Write, tune and maintain complex Spark SQL queries, partitioned datasets and transformations for optimal performance over large enterprise datasets.
• Build enterprise-grade services and data ingestion microservices using Spring and Spring Boot.
• Implement automated build, test and deployment workflows using Git/GitHub, Jenkins / Lightspeed and Harness.
• Leverage GenAI developer tooling (e.g., GitHub Copilot, Devin AI) to accelerate prototyping, code review, unit testing and documentation.
• Partner with Product Owners, Data Analysts, Business Teams and Quality Assurance to ensure end-to-end alignment and timely delivery.

What you'll need

• 5-9 years of hands-on software and data engineering experience in enterprise environments; minimum 7+ years expected.
• Strong communication skills, both verbal and written.
• Strong in Java/Scala development, Spring, Spring Boot, Apache Spark, Spark SQL and shell scripting.
• Ability to work in Python.
• Strong in SQL programming, especially Spark SQL.
• Demonstrated data analysis skills: data sourcing, processing and insights.
• Deep practical expertise in Apache Spark architecture (RDDs, DataFrames, Datasets, execution plans, memory management, partitioning and performance tuning).
• Advanced proficiency in Java and/or Scala, with strong object-oriented and functional programming paradigms.
• Strong proficiency in Python with practical experience writing data transformations using PySpark.
• Expert-level proficiency in SQL / Spark SQL, window functions, complex joins, indexing and query plan execution analysis (EXPLAIN).
• Proven ability to source, cleanse, enrich, reconcile and validate large datasets to deliver high-integrity reporting and analytical outputs.
• Ability to understand requirements and develop high-level and low-level design.
• Experience with DevOps tools such as GitHub, Jenkins/Lightspeed and Harness.
• Experience using AI tools such as GitHub Copilot, Devin AI or Claude Code.
• Bachelor's degree/University degree or equivalent experience.

Nice to have

• Pyspark experience.
• Tableau experience.
• Python, Unix shell scripting and Tableau for visualization and reporting.
• Experience with Tableau or similar BI visualization tools for dashboarding and reporting validation.
• Prior exposure to Banking, Financial Services, Liquidity Reporting, Regulatory Data (e.g., NDTL/LCR) or Risk Management.

Details

• Location: Pune, Maharashtra, India.
• Employment type: Full time.
• Works with a limited level of direct supervision and exercises independence of judgement and autonomy.