Senior Data Engineer II

Overview

Senior Data Engineer II on the Application Data Engineering team within Disney Entertainment and ESPN Product & Technology, building and maintaining the foundational data pipelines and datasets that support the streaming service.

What you'll do

• Lead and mentor other engineers in maintaining, updating, and expanding existing Application Data Engineering data pipelines in Scala, Python, and SQL using Spark while maintaining strict uptime SLAs.
• Become an SME in domain subject areas including Subscription, Application Performance, Experimentation, and Session data to support customers.
• Lead the SDLC surrounding modifications to both batch and streaming data pipelines using Scala, Databricks, and Airflow.
• Work with manager to create a comprehensive plan to improve observability, alerting, and maintain existing Spark jobs that power key business metrics.
• Collaborate with product managers, architects, and other engineers to drive the success of the Core Data platform.
• Contribute to debugging complex customer scenarios across different subject matter areas, determining if bug fixes and backfills are needed.
• Ensure high operational efficiency and quality of the Application Data platform datasets so solutions meet SLAs and project reliability and accuracy to stakeholders (Engineering, Data Science, Operations, and Analytics teams).
• Be an active participant and advocate of agile/scrum ceremonies to collaborate and improve processes for the team.
• Engage with and understand customers, forming relationships that allow the team to understand and prioritize new offerings and incremental platform improvements.
• Maintain detailed documentation of work and changes to support data quality and data governance requirements.

What you'll need

• 7+ years of data engineering experience developing large data pipelines.
• Strong algorithmic problem-solving expertise.
• Strong fundamental Scala and Python programming skills.
• Basic understanding of AWS or other cloud provider resources (S3).
• Strong SQL skills and ability to create queries to analyze complex datasets.
• Hands-on production environment experience with distributed processing systems such as Spark.
• Hands-on production experience with data pipeline orchestration systems such as Airflow for creating and maintaining data pipelines.
• Some scripting language experience.
• Willingness and ability to learn and pick up new skillsets.
• Self-starting problem solver with an eye for detail and excellent analytical and communication skills.
• Bachelor's or Master's Degree in Computer Science, Information Systems, or equivalent industry experience.

Nice to have

• Experience with at least one major Massively Parallel Processing (MPP) or cloud database technology (Snowflake, Redshift, BigQuery).
• Experience in developing APIs with GraphQL.
• Deep understanding of AWS or other cloud providers as well as infrastructure as code.
• Familiarity with Data Modeling techniques and Data Warehousing standard methodologies and practices.
• Familiar with Scrum and Agile methodologies.

Details

• Employment type: Full time.
• Location: Bangalore, India.