Senior Data Engineer

Overview

As a Senior Data Engineer, you will design and build data pipelines, model data for analysis, and keep data platforms dependable and well-governed. You will work hands-on with Databricks and AWS, mentor engineers, and act as an informal technical lead within the squad.

What you'll do

• Design, build and run reliable, scalable, distributed data pipelines on Databricks and AWS.
• Model data for analysis, keeping it accurate, well-governed and easy to use.
• Write secure, well-tested software to build, integrate and quality-assure data across bp pulse, meeting privacy and compliance needs.
• Follow and encourage good engineering practice, including technical design and review, testing, code review, source control and documentation.
• Build and improve CI/CD pipelines and manage infrastructure as code.
• Monitor and alert on the pipelines and services the team builds, provide business-hours support, and improve their reliability.
• Mentor other engineers and share knowledge to lift the quality of the team's work.
• Work with people across the business to understand what they need and turn it into dependable data products.

What you'll need

• Strong, hands-on experience building and running reliable, scalable, distributed data pipelines and data products in complex environments.
• Hands-on experience with Databricks, PySpark, Spark SQL, Delta Lake, Unity Catalog and Workflows, or similar lakehouse experience with a real appetite to grow on Databricks.
• Experience with AWS data and serverless services, including Lambda, Glue, S3, Redshift, Kinesis and SNS/SQS.
• Strong Python and advanced SQL.
• Good data modelling skills, applied in production.
• Experience with infrastructure as code and CI/CD.
• Experience mentoring engineers and helping raise the quality bar.
• Comfort working with people across the business and shaping work through technical influence.
• A curious, continuous-improvement mindset.
• A degree in computer science or a related field, or equivalent knowledge and experience.

Nice to have

• Experience with dbt or Apache Airflow.
• Familiarity with open table formats such as Delta, Apache Hudi or Apache Iceberg.
• Experience with Terraform or another infrastructure-as-code tool.

Details

• Location: Pune, India.
• This position is a hybrid of office/remote working.
• Business-hours support is required.
• No travel is expected with this role.
• This role is eligible for relocation within country.
• No prior experience in the energy industry is needed.