Data Engineer

Overview

Senior Data Engineer to analyze, design and develop enterprise-scale data pipelines across legacy and strategic cloud data platforms, with a focus on analyzing existing Omnia-based Spark pipelines and translating their logic into scalable cloud solutions.

What you'll do

• Conduct detailed analysis of existing end-to-end data pipelines across multiple platforms, with a particular focus on Omnia PySpark-based processing.
• Review existing source code, SQL, transformation rules, mappings and data flows to understand the current business and technical logic.
• Derive, document, and implement new transformation logic based on the analysis of existing pipelines and business requirements.
• Design, develop and maintain reliable, scalable, and reusable data pipelines using AWS, Snowflake and dbt.
• Develop batch and, where required, event-driven data processing solutions using cloud-native and distributed data technologies.
• Translate existing Omnia data processing logic into strategic cloud-based data engineering solutions.
• Build ingestion, transformation, enrichment and data integration pipelines across data lakes, data warehouses and cloud lakehouse platforms.
• Develop complex and performance-optimized SQL for data transformation, reconciliation, validation and analytical processing.
• Analyze end-to-end data flows from source capture through transformation, enrichment and downstream consumption.
• Work with data designers, product owners, and subject matter experts to translate business requirements into technical data rules and durable engineering solutions.
• Implement data quality, reconciliation and balancing controls across data pipelines.
• Troubleshoot data pipeline failures, transformation issues, data discrepancies and performance bottlenecks across multiple environments.

What you'll need

• Strong hands-on experience designing, developing and supporting scalable enterprise data pipelines.
• Strong Spark-ETL experience, including the ability to analyze existing Omnia pipelines, understand their processing logic and derive new transformation and business rules.
• Strong working knowledge of AWS services, including S3, Glue, EMR, Lambda, Athena, Step Functions, SQS, IAM, Secrets Manager and CloudWatch.
• Hands-on experience orchestrating data pipelines using Apache Airflow.
• Practical experience with Snowflake, covering data ingestion, data modelling, transformation and performance optimization.
• Experience using dbt to develop data models, incremental processing, reusable macros, automated tests, documentation and dependency management.
• Excellent SQL skills, with experience in complex data transformations, query optimization, reconciliation and understanding existing business logic.
• Familiarity with Git, automated testing, CI/CD, Jenkins and DevSecOps practices within a data engineering environment.
• Good Python programming skills for data processing, automation, and reusable framework components.

Nice to have

• Previous CBA experience, particularly with Omnia or CBA data platforms and engineering standards.

Details

• Location: Bengaluru - Thanissandra (BCIT).