Big Data and Apache Spark Developer - Assistant Vice President

Overview

Intermediate-level Applications Development role building and supporting large-scale data processing solutions with Apache Spark, participating in the establishment and implementation of new or revised application systems in coordination with the Technology team.

What you'll do

• Design, develop, test, and maintain data processing applications using Apache Spark.
• Build scalable ETL and data transformation pipelines for large datasets.
• Develop and support applications using Java or Python in a distributed data environment.
• Analyze requirements and collaborate with business analysts, QA teams, and stakeholders to deliver solutions.
• Troubleshoot production issues and optimize application performance.
• Participate in code reviews and follow engineering best practices.
• Contribute to application modernization, migration, and automation initiatives.
• Leverage AI-assisted development tools to improve productivity and quality.

What you'll need

• 8-10 years of software development experience.
• Strong hands-on experience with Apache Spark.
• Proficiency in either Java or Python.
• Candidates with primary expertise in Python should have a good understanding of Java fundamentals (OOP concepts, collections, exception handling, multithreading basics).
• Experience working with SQL and relational databases.
• Understanding of distributed computing concepts and big data processing.
• Strong analytical, problem-solving, and debugging skills.
• Demonstrated ability and willingness to adopt AI-assisted development tools (e.g., GitHub Copilot, Devin, Databricks Assistant, or similar) to improve productivity, code quality, and delivery efficiency.
• Ability to effectively leverage AI-generated recommendations for coding, testing, debugging, documentation, and analysis while maintaining accountability for final deliverables.
• Strong engineering judgment to critically review, validate, and optimize AI-generated code and solutions rather than relying solely on tool outputs.
• Capable of identifying potential functional, performance, security, and maintainability issues in AI-assisted implementations and making appropriate corrections.
• Passion for continuous learning and staying current with emerging AI technologies and software engineering best practices.

Nice to have

• Experience with Databricks (highly desirable).
• Exposure to cloud platforms and modern data engineering practices.
• Experience with CI/CD pipelines, Git, and Agile development methodologies.
• Knowledge of regulatory reporting or financial services domain is a plus.
• Experience with Spark performance tuning.
• Familiarity with data lake architectures.
• Experience leveraging AI tools such as Copilot or similar developer productivity solutions.

Details

• Location: Pune, Maharashtra, India.
• Employment type: Full time.