Lead Data Engineer - Data Engineering 4C

Overview

Lead Data Engineer role to design, build, and operationalize a Data Quality Factory, a standardized, scalable framework for profiling, validating, monitoring, and remediating data quality issues across enterprise data platforms.

What you'll do

• Design and implement a reusable Data Quality Factory framework covering profiling, rule-based validation, anomaly detection, and reconciliation across source and target systems.
• Build and maintain data quality rules, scorecards, and dashboards to track completeness, accuracy, consistency, timeliness, and uniqueness metrics.
• Develop and deploy data quality pipelines using Databricks (PySpark/Spark SQL) and Snowflake (SQL, Snowpark) for large-scale data validation.
• Leverage Azure AI Foundry to build and integrate AI/ML-assisted data quality checks (e.g., anomaly detection, intelligent rule suggestions, GenAI-based DQ remediation).
• Write efficient, production-grade Python and SQL scripts for automated data profiling, cleansing, and exception handling.
• Integrate data quality checks into CI/CD pipelines and orchestration tools to enable automated, scheduled DQ runs.
• Collaborate with data engineering, governance, and business teams to define DQ rules, thresholds, and remediation workflows.
• Perform root cause analysis on data quality issues and drive resolution with upstream data owners.
• Document data quality standards, metadata, and lineage to support data governance initiatives.
• Support audit and compliance requirements by maintaining data quality logs, exception reports, and control evidence.

What you'll need

• Azure AI Foundry: hands-on experience building/deploying AI-assisted solutions or agents.
• Python: strong programming skills for data validation, automation, and scripting.
• SQL: advanced query writing, performance tuning, and complex data validation logic.
• Databricks: experience with Delta Lake, notebooks, and job orchestration.
• Snowflake: experience with Snowflake SQL, Snowpark, warehouses, and data sharing.
• Strong understanding of data quality dimensions (accuracy, completeness, consistency, timeliness, validity, uniqueness).
• Familiarity with data governance, data cataloguing, and metadata management concepts.
• Experience integrating DQ frameworks with CI/CD and orchestration tools (e.g., Airflow, ADF, DevOps pipelines).
• Bachelor's/master's degree in computer science, Information Systems, Engineering, or related field.
• [4-6] years of experience in data engineering / data quality roles.
• Strong analytical and problem-solving mindset.
• Ability to work with cross-functional teams (engineering, governance, business).
• Excellent communication skills to document and present data quality findings.
• Self-driven, detail-oriented, and comfortable working in a fast-paced environment.
• Language: English, Proficient (C2).

Nice to have

• Experience with data quality tools (e.g., Great Expectations, Collibra, Informatica DQ, Ataccama).
• Exposure to Azure Data Factory, Azure Synapse, or other Azure data services.
• Knowledge of data lineage and cataloguing tools.
• Experience working in regulated industries (finance, healthcare, insurance) with strict data quality/compliance needs.
• Relevant certifications (Databricks, Snowflake, Azure) are a plus.

Details

• Location: Bangalore (Prestige Technology Park IV), India.
• Work mode: Hybrid.
• Work shift: Flex Time (India).
• Job type: Regular.