Data Engineer

Overview

Applications Development Programmer Analyst 2 (C10) in Citi's Technology – Data Engineering function, focused on the PySpark / Spark-based Data Platform, designing and maintaining scalable data pipelines and analytical solutions across enterprise systems.

What you'll do

• Design, develop, test, and maintain PySpark and Apache Spark data pipelines for batch and streaming workloads.
• Manage and monitor Spark Streaming Jobs (running Sunday evening through Saturday afternoon) performing Eligibility and Data Quality (DQ) checks, publishing data to Kafka sender topics, and processing exception/XMEV requests.
• Manage and monitor Spark Batch Jobs (running hourly or every 2 hours) that store audit, inbound, and outbound data into Hive tables.
• Respond to ITRS Alerts and system monitoring events (e.g., job failures, kills, or jobs not triggered within 30 minutes of scheduled time).
• Work with large-scale data sources across Oracle, Hadoop, Impala, and SAS Server.
• Configure and manage JDBC connections from SAS, Python, and R to dedicated Sandbox/data platform environments.
• Collaborate on schema creation during code deployment; raise CMP tickets for permission grants and country-specific schema access.
• Support the retirement of legacy applications and champion adoption of modern data engineering technologies.
• Work within a full software development lifecycle (SDLC) environment spanning Development (NAM), UAT, and Production.

What you'll need

• Proficiency in Python and PySpark for distributed data engineering.
• Experience with Apache Spark (Streaming and Batch) in production environments.
• Strong understanding of SQL and ability to work with relational (Oracle) and columnar (Impala, Hive) databases.
• Familiarity with Kafka or event-driven architectures.
• Experience with job scheduling and monitoring tools (e.g., Autosys, ITRS).
• Knowledge of data governance, access management, and information security standards.
• Strong problem-solving skills for diagnosing and resolving complex data pipeline issues.
• 2-5 years of relevant experience.
• Experience in programming/debugging used in business applications.
• Working knowledge of industry practice and standards.
• Comprehensive knowledge of specific business area for application development.
• Working knowledge of program languages.
• Consistently demonstrates clear and concise written and verbal communication.
• Bachelor's degree/University degree in information technology or Engineering, or equivalent experience.

Nice to have

• Experience with SAS or equivalent enterprise analytical platforms.
• Familiarity with SDLC practices across Development, UAT, and Production environments.
• GitHub / Lightspeed.

Details

• Location: Pune, Maharashtra, India.
• Full time.