Lead Data Engineer - Data Engineering 4C

Overview

Lead Data Engineer role on Genpact's Data Engineering team, solving cutting-edge data problems in Databricks to meet functional and non-functional requirements.

What you'll do

• Work with architects and lead engineers to deliver solutions that meet functional and non-functional requirements.
• Maintain close awareness of new and emerging technologies and their potential application for service offerings and products.
• Demonstrate knowledge of relevant industry trends and standards.
• Demonstrate strong analytical and technical problem-solving skills.

What you'll need

• Experience in the Data Engineering domain.
• Bachelor's degree in CS, CE, CIS, IS, MIS, or an engineering discipline, or equivalent work experience.
• Overall 8+ years of experience in IT.
• Excellent coding skills in Python and Spark / Scala.
• Implemented at least 2 projects end-to-end in Databricks.
• Experience on Databricks components: AutoLoader, Delta Lake, Unity Catalog, Lakeflow Connect, DB API 2.0, Databricks workflows orchestration.
• Well versed with the Databricks Lakehouse concept and its implementation in enterprise environments.
• Good understanding to create complex data pipelines.
• Good knowledge of data structures and algorithms.
• Strong in SQL and Spark-SQL.
• Strong performance optimization skills to improve efficiency and reduce cost.
• Worked on both batch and streaming data pipelines.
• Extensive knowledge of Spark and Hive data processing frameworks.
• Worked on any cloud (Azure, AWS, GCP) and common services such as ADLS/S3, ADF/Lambda, CosmosDB/DynamoDB, ASB/SQS, cloud databases.
• Strong in writing unit tests and integration tests.
• Strong communication skills and experience working on a team of 15+ people.
• Great attitude towards learning new skills and upskilling existing ones.

Nice to have

• Good to have Genie One, MosaicAI, DQX, and basic governance knowledge.
• Good to have Databricks SQL Endpoint understanding.
• Good to have CI/CD experience to build pipelines for Databricks jobs.
• Good to have worked on a migration project to build a unified data platform.
• Good to have knowledge of DBT.
• Good to have knowledge of Docker and Kubernetes.

Details

• Work shift: Flex Time (India).
• Work mode: Hybrid.
• Location: 1401-G-India: GF/11, Pritech Park, Bangalore.
• Language: English, proficient (C2), required.