Overview
Senior Data Engineer on PayPal's Marketing Technology Platform, designing and developing scalable data pipelines and data services on GCP (BigQuery, Bigtable) using Python, Kafka, and SQL, and delivering agentic use cases built on Google Gemini, Claude, and other AI platforms.
What you'll do
- Lead the design and development of complex data pipelines for data collection and processing.
- Develop and maintain advanced data storage solutions.
- Ensure data quality and consistency through sophisticated validation and cleansing processes.
- Implement advanced data transformation techniques to prepare data for analysis.
- Collaborate with cross-functional teams to understand data requirements and provide innovative solutions.
- Optimize data engineering processes for performance, scalability, and reliability.
- Design and build highly scalable, high-throughput data pipelines in GCP using BigQuery and Bigtable, primarily batch with streaming integrations via Kafka.
- Write and maintain production-grade pipelines in Python and advanced SQL.
- Own BigQuery-backed datasets and pipelines that support complex Marketing use cases, with ingress and egress at scale.
- Build agentic services end to end on top of our data and pipelines, covering data access, orchestration, guardrails, evaluation, deployment, and monitoring, leveraging Google Gemini, Claude, and other AI platforms.
- Use AI-native development practices to accelerate design, coding, testing, and operations.
- Design data models and access patterns that are adaptable to evolving business needs.
- Optimize BigQuery performance, cost, and reliability using partitioning, clustering, and query tuning.
- Build and operate real-time and near-real-time pipelines using Pub/Sub, batch or streaming frameworks, and Java services.
- Model data and design schemas across OLTP and OLAP systems, balancing performance, cost, and usability.
- Schedule and orchestrate jobs using Airflow and UC4.
- Instrument pipelines for monitoring, data quality, and alerting, and implement data lineage so that data is observable and trustworthy.
- Independently own multiple data features end to end, proposing innovative solutions for new and existing functionality.
- Deliver complete solutions across all phases of the SDLC: design, implementation, testing, delivery, and operations.
- Operate with little day-to-day supervision, making sound technical decisions grounded in internal conventions and industry best practices.
- Troubleshoot pipeline and data issues and drive root-cause fixes.
- Advise management on project-level issues and risks.
- Actively participate in design and code reviews, giving and incorporating constructive feedback.
- Guide junior engineers and share your knowledge to help new team members onboard quickly.
What you'll need
- 3+ years relevant experience and a Bachelor's degree OR any equivalent combination of education and experience (minimum qualification).
- A bachelor's degree in computer science or an equivalent combination of technical education and work experience.
- 6+ years of data engineering experience, with expertise in ETL/ELT: extracting, transforming, and loading data from varied sources using advanced SQL and Python.
- An AI-native mindset, with hands-on experience building agentic services end to end using Google Gemini, Claude, and other LLM platforms and agent frameworks.
- Strong data fundamentals: data warehouses, data marts, data lakes, data modeling, and schema design.
- Fluency with both OLTP and OLAP systems and the trade-offs between them.
- Hands-on experience with event-driven architecture and messaging systems such as Kafka or Pulsar, and streaming integrations.
- Hands-on experience with databases, both SQL and NoSQL.
- Exposure to cloud platforms such as AWS, GCP, or Azure and Infrastructure as Code (Terraform).
- Experience designing and building highly scalable data pipelines that handle very high data volumes in GCP (BigQuery, Bigtable).
- 4+ years of hands-on experience on Google Cloud Platform, particularly BigQuery and Pub/Sub.
- Experience with workflow orchestration tools such as Airflow or UC4.
- Experience with pipeline instrumentation, data quality, and data lineage.
- Experience implementing CI/CD, automated testing, and observability to improve delivery quality and operational resilience, with fluency in Git and collaborative development workflows.
- Experience using Atlassian JIRA, ServiceNow, and Confluence.
- Experience delivering projects using Agile methodology.
Details
- Location: Bangalore, Karnataka, India; Chennai, Tamil Nadu, India.
- Hybrid work model: 3 days in the office and 2 days at the PayPal office or home workspace.
- Subsidiary: PayPal Travel.
Read the full description and apply on the company’s own careers page.