Lead Data Engineer - Assistant Vice President

Overview

Lead Data Engineer / SME in the SSIM Business Architecture vertical, working as a hands-on Individual Contributor to design, build, and support scalable, production-grade data pipelines and cloud data solutions.

What you'll do

• Design, develop, and maintain scalable ETL/ELT, data ingestion, and transformation pipelines.
• Build and optimize data processing solutions using Python and PySpark.
• Develop and support production-grade workloads and pipelines on Databricks.
• Design and implement cloud-native data engineering solutions on AWS.
• Work with large-scale structured and unstructured datasets.
• Optimize Spark and Databricks workloads for performance, scalability, reliability, and cost efficiency.
• Develop reusable data engineering components, frameworks, and engineering patterns.
• Implement data quality, validation, reconciliation, and monitoring controls.
• Troubleshoot complex production issues, perform root cause analysis, and implement sustainable fixes.
• Support CI/CD, testing, deployment, monitoring, and operational reliability.
• Act as a Data Engineering SME, providing technical guidance, design reviews, and code reviews.
• Collaborate with architecture, application, cloud, platform, and business teams to deliver enterprise data solutions.
• Drive automation, standardization, and continuous improvement across the data engineering landscape.

What you'll need

• 10+ years of overall experience in software engineering, data engineering, or related technology roles.
• 8+ years of hands-on data engineering & DevOps experience.
• Proven experience designing, developing, and supporting enterprise-scale production data pipelines and platforms.
• Demonstrated ability to operate as a hands-on technical SME in complex enterprise environments.
• Strong hands-on experience in Data Engineering.
• Strong hands-on experience with Databricks.
• Advanced programming skills in Python and PySpark.
• Strong experience with AWS cloud-based data engineering.
• Strong proficiency in SQL.
• Experience designing and supporting enterprise ETL/ELT and data pipelines.
• Strong understanding of Apache Spark and distributed data processing.
• Experience with data lake and lakehouse architecture.
• Strong understanding of data modeling, data integration, and data quality principles.
• Experience with CI/CD and modern engineering practices.
• Experience supporting business-critical production data platforms.
• Strong troubleshooting, debugging, and root cause analysis skills.
• Strong communication and stakeholder-management capabilities.
• Bachelor’s degree in Computer Science, Information Technology, Engineering, Data Science, or a related technical discipline.

Nice to have

• 7+ years of relevant experience across Databricks, PySpark/Spark, and cloud-based data engineering.
• Experience with Snowflake data engineering and integration.
• Knowledge of Snowflake databases, schemas, warehouses, security, and performance optimization.
• Experience integrating Databricks/AWS data pipelines with Snowflake.
• Experience with GoldenGate, AWS DMS, or similar data replication technologies.
• Knowledge of data governance, lineage, metadata management, and auditing.
• Experience with data quality and reconciliation frameworks.
• Exposure to enterprise monitoring and observability solutions.
• Experience working in financial services or other regulated environments.
• Master’s degree in a relevant discipline.

Details

• Location: Bangalore, India.
• Hybrid working requires 4 days per week working from the office.
• Primary working window: 12:00 PM to 9:00 PM India time.
• Must be flexible to work outside standard hours and, where required, over weekends to support critical deliveries, production issues, releases, DR, or other business needs.
• The role requires close collaboration with Architecture, Security, engineering, platform, and business technology stakeholders.