Data Architect - Data Engineering 4D

Overview

Senior DataOps Engineer on the Global Digital Operations Data Platform team, building a governed, metadata-driven data platform on Azure Databricks and Unity Catalog.

What you'll do

• Build and operate Auto Loader–based ingestion pipelines (batch and incremental) from Unity Catalog volumes into managed Delta tables, using MERGE patterns for upsert/dedup logic
• Design and maintain metadata-driven pipeline generation, where dataset onboarding (new source, new schema, new retention rule) is a config change against a rules table, not new pipeline code per dataset
• Implement and maintain partition-based retention (year partitioning; rolling window of current + N prior years) and the archive extraction process for datasets requiring longer history
• Build and maintain a dataset routing-rules engine: a governed metadata table (splittable, target_systems[], row_filter_expr, column_mask_expr) that programmatically generates consumption-layer views per downstream system
• Own Unity Catalog governance: catalog/schema/storage-location design, access grants, lineage, and audit trail across landing, central, archive, and consumption layers
• Manage Databricks Asset Bundles (DAB) or equivalent IaC for repeatable environment and job deployment across dev/test/prod
• Monitor pipeline health, checkpoint integrity, schema drift/evolution, and job SLAs; build alerting for ingestion failures and data-freshness breaches
• Collaborate with upstream source-system owners to define data contracts (e.g. ensuring _source_changed_at semantics are honored) and with downstream teams to translate their access/filter requirements into routing rules
• Document pipeline architecture, retention policy, and routing logic for audit and onboarding purposes

What you'll need

• Azure Databricks and Unity Catalog (catalogs, schemas, managed/external volumes, managed/external tables, storage-location hierarchy, grants)
• Strong PySpark and SQL, including Structured Streaming / Auto Loader (cloudFiles, foreachBatch, trigger(availableNow=True)), MERGE INTO, and Delta Lake internals (transaction log, OPTIMIZE, VACUUM, ZORDER)
• Practical experience with partitioning strategy and retention management (partition-drop vs. rolling delete tradeoffs) at production data volumes
• Demonstrable background designing or maintaining metadata/config-driven code generation, e.g. a rules or registry table that drives dynamically generated SQL, views, jobs, or DDL, rather than hardcoded per-object pipelines
• Comfort with templated/parameterized pipeline patterns (e.g. DAB with variable substitution, dbt-style codegen, or custom Python/SQL generation from a config table)
• Understanding of how to keep generated artifacts (views, jobs) in sync with a source-of-truth ruleset, including versioning and change auditability
• Azure fundamentals: ADLS Gen2, Azure Data Factory or equivalent orchestration, networking/private endpoints, cost management basics
• CI/CD for data pipelines (Azure DevOps or GitHub Actions), ideally with Databricks Asset Bundles or Terraform for Databricks
• Git-based workflow discipline; comfortable owning pipeline code as a shared, reviewed codebase rather than notebook-only development
• Monitoring/observability for data pipelines (job alerting, data quality checks, schema drift detection)
• Working knowledge of data governance concepts: access control models, data lineage, retention/compliance-driven deletion, PII masking
• Experience working with multiple downstream consumers with differing data needs, building filtered/masked views rather than granting broad table access
• Clear written documentation habits; able to explain pipeline and governance logic to both technical and non-technical stakeholders
• Bachelor's in Business Analytics, Computer Science, or Statistics, or Master's in Data Science
• Required skills: Agile Methodology, Change Management, Consumer Packaged Goods (CPG), Databricks Platform, Data Engineering, Snowflake GenAI
• English language proficiency, Proficient - C2

Nice to have

• Experience with Delta Live Tables or Databricks Streaming Tables (declarative SQL ingestion)
• SnowFlake, Iceberg
• Exposure to multi-tenant data platforms serving several internal business-unit consumers with differing retention/access needs
• Certifications: Certified Data Management Professional (Riversand); Databricks Certified Associate Developer for Apache Spark 3.0 or Apache Spark (Databricks Academy); Microsoft Certified: Azure Data Engineer Associate; Oracle Database 12c Certified Implementation Specialist

Details

• Location: Hyderabad, India (Floor 1 to 12, Hafeezpet Phoenix)
• Work mode: Office
• Shift: Flex Time (India)
• Job type: Regular