Lead Data Engineer - Data Engineering 4C

Overview

Lead a Databricks-focused data engineering role solving complex, real-world problems to meet functional and non-functional requirements, working with architects and lead engineers.

What you'll do

• Work with architects and lead engineers on solutions that meet functional and non-functional requirements.
• Maintain close awareness of new and emerging technologies and their potential application for service offerings and products.
• Apply demonstrated knowledge of relevant industry trends and standards.
• Apply strong analytical and technical problem-solving skills.
• Lead the team of size 15 plus.

What you'll need

• Experience in the Data Engineering domain.
• Bachelor's degree in CS, CE, CIS, IS, MIS, or an engineering discipline, or equivalent work experience.
• Overall 8+ years of experience in IT.
• Excellent coding skills in Python and Spark / Scala, preferably Python and Spark.
• Implemented at least 2 projects end-to-end in Databricks.
• Experience on Databricks components: AutoLoader, Delta Lake, Unity Catalog, Lakeflow Connect, DB API 2.0, and Databricks workflows orchestration.
• Well versed with the Databricks Lakehouse concept and its implementation in enterprise environments.
• Ability to create complex data pipelines.
• Good knowledge of data structures and algorithms.
• Strong in SQL and Spark SQL.
• Strong performance optimization skills to improve efficiency and reduce cost.
• Worked on both batch and streaming data pipelines.
• Extensive knowledge of Spark and Hive data processing frameworks.
• Worked on any cloud (Azure, AWS, GCP) and common services such as ADLS/S3, ADF/Lambda, CosmosDB/DynamoDB, ASB/SQS, and cloud databases.
• Strong in writing unit test cases and integration tests.
• Strong communication skills and experience working on a team of size 15 plus.
• Great attitude towards learning new skills and upskilling existing skills.
• Language: English (Required), Proficient - C2.

Nice to have

• Good to have Genie One, MosaicAI, DQX, and basic governance knowledge.
• Good to have Databricks SQL Endpoint understanding.
• Good to have CI/CD experience to build pipelines for Databricks jobs.
• Good to have worked on a migration project to build a unified data platform.
• Good to have knowledge of DBT.
• Good to have knowledge of Docker and Kubernetes.

Details

• Location: Hyderabad, India (STPI, Uppal).
• Remote type: Hybrid.
• Work shift: Flex Time (India).
• Job type: Regular.