Data Engineer – Python, PySpark, SQL & ETL

Overview

Design, build and support scalable ETL pipelines and data marts using Python, PySpark, Spark, SQL and data-warehousing technologies, covering the full software development lifecycle from development through post-production support.

What you'll do

• Design, develop, maintain and optimize scalable ETL and data-pipeline solutions using Python and PySpark.
• Build and enhance data marts and data-transformation processes.
• Develop distributed data-processing solutions using Spark, Hadoop, Hive and MapReduce.
• Work with structured, semi-structured and unstructured data from multiple sources.
• Develop and optimize Oracle SQL queries for data extraction, transformation, analysis and validation.
• Analyze and debug data issues using Oracle SQL, PySpark, Pandas and structured data-analysis techniques.
• Perform end-to-end SDLC activities, including development, testing, UAT support, bug fixing, production deployment and post-production support.
• Collaborate with cross-functional teams to define data requirements, validate solutions, resolve defects and improve data quality.
• Implement data validation, reconciliation, completeness, accuracy and consistency checks.
• Support CI/CD implementation for data pipelines, applications and deployment workflows.
• Monitor production environments and troubleshoot pipeline failures, data-quality issues, performance problems and processing delays.
• Create and maintain technical documentation, data mappings, transformation logic, support procedures and operational runbooks.
• Apply software engineering best practices, coding standards, testing practices and maintainable design principles.
• Contribute to performance improvements through query optimization, efficient transformations, appropriate partitioning, resource management and pipeline tuning.
• Identify opportunities to improve automation, reliability, data quality, maintainability and sustainable use of processing resources.

What you'll need

• Minimum 5 years of professional experience in data engineering, ETL development, data warehousing, big-data engineering or a related technical role.
• Strong hands-on experience developing data-processing applications, ETL workflows, automation scripts and production services in Python.
• Strong experience building distributed data-processing and ETL solutions using PySpark and Spark.
• Experience with data preparation, analysis, transformation, validation and profiling using Pandas.
• Strong experience with SQL for data extraction, transformation, validation, analysis and query optimization.
• Strong knowledge of Oracle SQL, including complex queries, joins, aggregations, subqueries, stored procedures and performance optimization.
• Experience designing, developing, testing, deploying and supporting ETL and data-pipeline solutions.
• Strong understanding of data-warehouse concepts, dimensional modelling, data marts, transformations, fact tables, dimensions and data quality.
• Experience working with Hadoop, Hive and MapReduce.
• Experience working with SQL and NoSQL databases and with structured, semi-structured and unstructured data.
• Experience using Jupyter Notebook for data analysis, development, validation, troubleshooting and technical exploration.
• Familiarity with CI/CD pipelines, including automated build, testing, deployment and release processes.
• Experience validating data transformations, pipeline results, data quality and application behavior.
• Understanding of coding standards, version control, documentation, code reviews, testing and maintainable development practices.
• Experience troubleshooting data-pipeline failures, query issues, data-quality defects, performance problems and production incidents.
• Understanding of secure data access, authentication, authorization and least-privilege principles.
• Ability to protect sensitive business, customer, financial and operational data, including secure database connections, credential management, data masking, encryption and controlled data handling.
• Bachelor's degree in Computer Science, Information Technology, Engineering, Data Science or a related field; equivalent relevant experience may be considered.

Nice to have

• Experience with banking or financial-services data and reporting requirements.
• Exposure to cloud-based data platforms and data-processing services.
• Familiarity with workflow scheduling, orchestration, monitoring and alerting tools.
• Experience with automated data-quality checks, reconciliation, lineage and metadata management.
• Exposure to infrastructure automation, containerization and modern DevOps practices.
• Familiarity with tools supporting data pipeline observability and performance analysis.
• Banking and financial-services experience is highly preferred.

Details

• Location: Bengaluru - Bellandur (GTP).