Cloud Data Engineer

Overview

Cloud Data Engineer to build and maintain data pipelines, data warehouses, and data lakes that keep data accurate, accessible, and secure.

What you'll do

• Build and maintain data architecture pipelines for processing and transfer of durable, complete, consistent data.
• Design and implement data warehouses and data lakes with required security measures and scale for data volumes/velocity.
• Develop processing and analysis algorithms for intended data complexity and volumes.
• Collaborate with data scientists to build and deploy machine learning models.
• Lead and supervise a team, allocating work and coordinating resources (or act as an advisor as an individual contributor).
• Manage risk and strengthen controls for owned/contributed work, escalating policy/procedure breaches appropriately.

What you'll need

• Hands-on experience in pyspark and strong knowledge of Dataframes, RDD, and SparkSQL.
• Hands-on experience developing, testing, and maintaining applications on AWS Cloud.
• Strong knowledge of AWS data analytics technologies: Glue, S3, Lambda, Lake Formation, and Athena.
• Experience designing and implementing scalable data transformation/storage solutions using Snowflake.
• Experience ingesting data into Snowflake for formats including Parquet, Iceberg, JSON, and CSV.
• Experience using DBT (Data Build Tool) with Snowflake for ELT pipeline development.
• Experience writing advanced SQL and PL SQL programs.

Nice to have

• Exposure to data governance or lineage tools such as Immuta and Alation.
• Experience using orchestration tools such as Apache Airflow or Snowflake Tasks.
• Knowledge on Abinitio ETL tool.

Details

• Role location: Pune.