Data Engineer - PySpark

Overview

Data Engineer - PySpark builds and maintains data systems including data pipelines, warehouses, and lakes to keep data accurate, accessible, and secure.

What you'll do

• Build and maintain data architecture and pipelines for transferring and processing consistent data.
• Design and implement data warehouses and data lakes with security measures.
• Develop processing and analysis algorithms for specified data complexity and volume.
• Collaborate with data scientists to build and deploy machine learning models.
• Support timely, high-standard execution and drive continuous improvement.
• Lead and supervise a team (or act as an individual contributor advisor) and coordinate work and resources.
• Manage risk and strengthen controls for owned or contributed work.

What you'll need

• Hands-on experience with PySpark including DataFrames, RDD, and SparkSQL.
• Hands-on experience developing, testing, and maintaining applications on AWS Cloud.
• Knowledge of AWS Data Analytics stack including Glue, S3, Lambda, Lake Formation, and Athena.
• Experience designing and implementing scalable data transformation/storage solutions using Snowflake.
• Experience ingesting data into Snowflake from formats such as Parquet, Iceberg, JSON, and CSV.
• Experience using DBT (Data Build Tool) with Snowflake for ELT pipeline development.
• Ability to write advanced SQL and PL SQL programs.

Details

• Role based out of Pune.