Data Engineer II

Overview

Lead the design, build and operation of data pipelines and data services for Disney's Advertising Data environment, providing engineering and architectural direction for the Audience Platform within Ad Platforms.

What you'll do

• Lead the team to design, develop, maintain, operate and support resilient data pipelines and data services in Scala, Python / Spark, and Java while maintaining strict uptime SLAs
• Collaborate with teams to extend functionality of current Operational Data platform data services, and build new integrations with APIs both internal and external to the Data organization
• Drive and provide guidance to teams on architecture, design, development, integrations, and technical work breakdown for complex applications and data flows, and align SLAs and data delivery to business strategies, principles, and practices
• Review architecture and architectural standards, proposing changes to improve and optimize them and automate where possible
• Lead the team to implement quality coding practices across batch and streaming data pipelines using Scala, Python, Databricks, and other key technologies
• Mentor, coach and educate a team of Data Engineers and Software Engineers on the best use of data capabilities and development best practices (fault tolerance, reliability, reuse, observability, etc.)
• Implement shared libraries in Scala and Python that abstract complex business logic for consistent functionality across all data pipelines
• Lead the resolution of critical incidents and proactively address data and pipeline issues; promote and expand the use of CI/CD pipelines
• Lead engineering solutioning sessions, contribute to development planning and implementation, and help the team understand and break down work
• Lead the development and documentation of internal and external standards and best practices for pipeline configurations, naming conventions, partitioning strategies, and more
• Ensure high operational efficiency and quality of data platform datasets so solutions meet SLAs and stakeholders (Engineering, Data Science, Operations, BI, and Analytics teams) receive reliable, accurate data
• Engage with customers to understand needs and prioritize new offerings and incremental platform improvements
• Drive standards for detailed documentation of work and changes to support data quality and data governance requirements
• Mentor engineers, conduct design and code reviews, and enforce data engineering best practices
• Collaborate with infra, product, and governance teams to align on priorities and safe adoption of AI
• Drive delivery of high-profile AI applications by balancing execution speed with system reliability

What you'll need

• Bachelor's degree in Computer Science, Information Systems, Electrical Engineering, or another comparable field of study
• 10+ years of experience in software development or software engineering in a large enterprise environment using modern languages, frameworks, and data platforms/tools
• At least 5 years of data engineering experience developing large data pipelines
• Strong algorithmic problem-solving expertise
• Demonstrable expertise with relevant data technologies such as MSK, Kinesis, S3, Flink, Spark, NiFi, SingleStore, Snowflake, AWS Glue, Terraform, K8s, EKS, Lambda, Airflow, DataDog and Grafana for observability
• Advanced Scala and Python programming skills
• Deep understanding of data architecture, data management, data governance, security, and automation tools
• Track record of championing quality engineering and writing code that can be tested with unit tests
• Hands-on production environment experience with distributed processing systems such as Spark and data pipeline orchestration systems such as Airflow
• Experience working with high-performing teams using Agile and Lean methodologies and frameworks
• Care about your craft and have opinions about the right way to do things with technology

Nice to have

• Master's degree in Computer Science, Software Engineering or related technical discipline (highly desirable)
• Previous work experience in Ad Serving or Ad Technology platforms
• Excellent leadership and communication skills
• Experience processing large amounts of data at petabyte level, with experience in machine learning and AI
• Strong curiosity about how Disney delivers the Magic and a desire to be part of it
• Experience with Python, Java, Databricks, LangChain, vector stores (Pinecone, Weaviate, FAISS, pgvector), SQL, and the AWS big data tech stack (such as S3, Glue, MWAA)

Details

• Full-time employment
• Location: Bangalore, India
• Part of Ad Platforms, Product Engineering team (PE - India Product & Technology, PE - India Ad & Data Platforms)