Data Architect - Data Engineering 4D

Overview

Data Architect role in Data Engineering focused on designing and building real-time streaming data pipelines and enterprise data transformation frameworks using Apache Flink and Kafka.

What you'll do

• Design, develop, and maintain high-performance real-time data processing applications using Apache Flink.
• Develop in-flight transformation logic leveraging Apache Flink, Kafka Streams, KSQLDB, and Kafka Single Message Transforms (SMTs).
• Build and maintain scalable mapping frameworks for transforming source system data into harmonized enterprise data models.
• Implement complex event processing, enrichment, filtering, aggregation, and windowing logic for streaming workloads.
• Optimize streaming pipeline performance, ensuring low latency, high throughput, and efficient resource utilization.
• Design solutions to handle late-arriving and out-of-order events using event-time processing, watermarks, checkpointing, and state management.
• Integrate streaming applications with Kafka, Schema Registry, APIs, and downstream analytical platforms.
• Implement robust error handling, retry mechanisms, dead-letter queues (DLQ), and monitoring for streaming applications.
• Collaborate with Solution Architects, Data Architects, API teams, and business stakeholders to define transformation rules and streaming integration patterns.
• Develop automated testing frameworks and CI/CD pipelines for streaming applications.
• Monitor production streaming jobs and troubleshoot performance, scalability, and reliability issues.

What you'll need

• 6–10+ years of experience in Data Engineering with strong expertise in real-time streaming solutions.
• Hands-on experience with Apache Flink for enterprise-scale stream processing.
• Strong knowledge of Apache Kafka, Kafka Streams, KSQLDB, Kafka Connect, and Single Message Transforms (SMTs).
• Experience designing event-driven data architectures and real-time data integration pipelines.
• Strong understanding of event-time processing, watermarks, windowing, checkpointing, stateful stream processing, and fault tolerance.
• Experience developing data transformation and harmonization frameworks for enterprise data platforms.
• Proficiency in Java and/or Scala.
• Experience with Avro, Protobuf, JSON, Schema Registry, and serialization techniques.
• Strong SQL skills and experience working with large-scale structured and semi-structured datasets.
• Bachelor's in Computer Science, Data Science, or Statistics, or Master's in Data Science.
• English proficiency at C2 level.

Nice to have

• Python knowledge.
• Experience with cloud platforms such as Azure, AWS, or GCP.
• Familiarity with CI/CD, Git, Docker, Kubernetes, and monitoring tools.
• Experience with Confluent Platform and enterprise Kafka deployments.
• Exposure to modern data lakehouse platforms such as Databricks, Snowflake, or Delta Lake.
• Knowledge of CDC patterns, API integration, and enterprise integration architectures.
• Experience working in Agile/Scrum delivery environments.
• Excellent analytical, problem-solving, and stakeholder communication skills.
• Certified Data Management Professional - Riversand, or Databricks Certified Associate Developer for Apache Spark 3.0.

Details

• Location: 1401-GIPL: Prestige Technology Park IV, Bangalore.
• Work mode: Hybrid.
• Shift: Flex Time (India).
• Job type: Regular.