Data Engineer

Overview

Perform development work and technical support related to data transformation and ETL jobs in support of a global data warehouse, communicating results to internal customers.

What you'll do

• Develop new ETL/data transformation jobs using PySpark and IBM DataStage in AWS.
• Enhance and support existing ETL/data transformation jobs.
• Explain technical solutions and resolutions to internal customers and communicate feedback to the ETL team.
• Perform technical code reviews for peers moving code into production.
• Perform and review integration testing before production migrations.
• Provide a high level of technical support and perform root cause analysis for problems within your area of functional responsibility.
• Document technical specs from business communications.

What you'll need

• 5+ years of ETL and SQL experience.
• 3+ years of Python and PySpark experience.
• 3+ years of AWS and UNIX experience.
• 2+ years of IBM DataStage experience.
• 1+ years of Snowflake experience.
• Bachelor's degree in computer science or equivalent.
• Experience with core Python programming for data transformation.
• Intermediate-level PySpark skills: can read, understand and debug existing code and write simple PySpark code from scratch.
• Strong knowledge of SQL fundamentals, including subqueries, and ability to tune queries with execution hints to improve performance.
• Able to write SQL code sufficient for most business requirements for pulling data from sources, applying rules, and loading target data.
• Proven track record troubleshooting ETL jobs and addressing production issues such as performance tuning, reject handling, and ad-hoc reloads.
• Proficient in developing optimization strategies for ETL processes.
• Basic AWS technical support skills: can log in, find existing jobs, and check run status and logs.
• Will run and monitor jobs running via Control-M.
• Can create clear and concise documentation and communications.
• Ability to coordinate and aggressively follow up on incidents and problems, perform diagnosis, and provide resolution to minimize service interruption.
• Ability to prioritize and work on multiple tasks simultaneously.
• Effective in cross-functional and global environments, managing multiple tasks and assignments concurrently with effective communication.
• Self-starter who can work well independently and on team projects.
• Experienced in analyzing business requirements, defining granularity, source-to-target mapping of data elements, and full technical specification.
• Understands data dependencies and how to schedule jobs in Control-M.
• Experienced working at the command line in various flavors of UNIX, with basic understanding of bash and korn shell scripting.

Nice to have

• IBM DataStage experience.
• Snowflake background or certification.
• Preferred certifications: AWS Certified Cloud Practitioner; Certified DataStage Professional; Python and PySpark certifications; SnowPro Certification; DE with AI Certification; Control-M Scheduler Certification.

Details

• Location: Bangalore, Karnataka.