Data Engineer - Big Data, Python

Overview

Data Engineer (Applications Development Intermediate Programmer Analyst) responsible for building and improving scalable data pipelines and architectures in coordination with the technology team.

What you'll do

• Design, build, and optimize scalable data pipelines and architectures using data engineering and ETL/ELT concepts.
• Automate workflows and analyze/optimize data processing using Python, Bash, and SQL.
• Collaborate with business analysts and cross-functional tech teams to understand data requirements and deploy/support data platforms.
• Design and implement scalable data models, schemas, and pipeline specifications using programming and software engineering best practices.
• Ensure data quality, integrity, and security through testing and debugging of data workflows with governance/privacy compliance.
• Identify bottlenecks and recommend optimizations for data ingestion, processing, and storage performance.
• Resolve complex pipeline failures, data anomalies, and infrastructure bottlenecks using big data and cloud data warehouse experience.

What you'll need

• 5-7 years of relevant experience in the Financial Service industry.
• Intermediate level experience in an Applications Development role.
• Proficiency in Big Data technologies including Cloudera and Hive, plus Python and Java/PySpark.
• Proficiency in data analysis and data modelling.
• Knowledge of ETL concepts.
• Good understanding of relational databases such as Oracle and SQL Server.
• Bachelor’s degree/university degree or equivalent experience.

Details

• Location: Chennai, Tamil Nadu, India.
• Employment type: Full time.