Senior Data Engineer

Overview

As a Senior Data Engineer on the Member Communication Platform team, you’ll design and operate secure, scalable data products that support audience selection, eligibility, preferences, consent, campaign inputs, delivery data, and operational reporting. You’ll deliver reliable Databricks-based pipelines and integrations, improve data quality and observability, and provide technical leadership across cross-functional initiatives.

What you'll do

• Lead the design and implementation of scalable, secure data architectures for member communications, defining data models, contracts, lineage, and ownership.
• Independently lead complex data engineering initiatives from technical requirements and design through production delivery, coordinating work across multiple engineers and stakeholders while managing dependencies, risks, and technical trade-offs.
• Design, develop, and optimize production-grade data pipelines and integrations using Databricks, Apache Spark, PySpark, Spark SQL, Delta Lake, and SQL, while modernizing appropriate legacy workloads.
• Build data-loading, reconciliation, retry, idempotency, recovery, and deployment patterns.
• Help operate data lakehouse and operational data assets for performance, integrity, availability, and security.
• Establish data-quality and governance practices, including validation, monitoring, alerts, dashboards, runbooks, and lineage.
• Partner with Privacy, Compliance, Risk, and Security on PII, preferences, consent, suppression, retention, and access controls.
• Provide technical leadership through design and code reviews, mentor junior engineers, lead complex production investigations, identify root causes, and implement durable fixes.
• Partner with Product, Communications, Analytics, Engineering, Risk, Privacy, Compliance, and other stakeholders to translate needs and technical trade-offs into actionable solutions.
• Drive release readiness through automated testing, CI/CD, rollout planning, and post-release measurement.
• Monitor pipeline performance, quality, timeliness, cost, and reliability.
• Use AI tools thoughtfully to work smarter, reduce manual work, and make better decisions faster.

What you'll need

• 6+ years of experience in data engineering, with a focus on data architecture, data pipelines, data platforms, and database or lakehouse management.
• A bachelor's or master's degree in Computer Science, Data Science, Engineering, or a related field, or equivalent practical experience.
• Strong proficiency in Python/PySpark and SQL.
• Strong leadership, problem-solving, decision-making, and communication skills.
• Strong hands-on experience with Databricks, Apache Spark, Spark SQL, Delta Lake, workflow orchestration, CI/CD, Git-based workflows, automated testing, and end-to-end data integrations or products in production environments.
• Experience with data quality, observability, alerting, lineage, cloud platforms and services, especially AWS.
• Experience with secure access controls, secrets management, encryption, auditability, and sensitive member data.
• Experience with Agile ways of working.

Nice to have

• Experience in financial services, lending, marketing technology, customer communications, or another regulated environment.
• Familiarity with Unity Catalog or comparable data-governance tooling.
• Experience with Java or Scala.
• Experience using AI tools such as ChatGPT, Copilot, or similar tools to improve engineering productivity, testing, documentation, troubleshooting, data quality, or operational effectiveness.
• Familiarity with using AI agents or automations to research, prototype, test, document, and deliver solutions across a broad technology stack with sound judgment for security, privacy, accuracy, and human review.

Details

• Remote role that must be performed from within India.