Data Engineer

Overview

Build and run data pipelines and platforms that bp pulse relies on, using Databricks and AWS. This hands-on role focuses on dependable data products, data accuracy and governance, and collaboration with senior engineers and people across the business.

What you'll do

• Build and maintain reliable, scalable data pipelines on Databricks and AWS.
• Implement data ingestion, validation and transformation.
• Write well-tested software to build, integrate and quality-assure data across bp pulse, meeting privacy and compliance needs.
• Follow good engineering and data practice, including technical design, testing, code review, source control, documentation, data modelling and observability.
• Help build and improve CI/CD pipelines and infrastructure as code.
• Run data quality checks to keep data accurate and reliable.
• Keep pipelines and services healthy through monitoring, alerting and business-hours support.
• Work with senior engineers and people across the business to turn needs into dependable data products.

What you'll need

• Hands-on experience building and maintaining reliable, scalable data pipelines and data products, often alongside more senior engineers.
• Experience with Databricks, PySpark, Spark SQL and Delta Lake, or a similar lakehouse platform, with a real appetite to grow on Databricks.
• Experience with AWS data and serverless services, such as Lambda, Glue, S3, Kinesis and SNS/SQS.
• Good Python and SQL skills.
• An understanding of data modelling and data quality fundamentals.
• Some experience with CI/CD and infrastructure as code.
• Strong verbal and written communication skills.
• A curious, continuous-improvement mindset.
• A degree in computer science or a related field, or equivalent knowledge and experience.

Nice to have

• Experience with dbt or Apache Airflow.
• Familiarity with data quality tools such as Great Expectations.
• Familiarity with open table formats such as Delta, Apache Hudi or Apache Iceberg.
• Experience with Terraform or another infrastructure-as-code tool.

Details

• Location: Pune, India.
• Hybrid of office/remote working.
• No travel is expected with this role.
• This role is eligible for relocation within country.
• Business-hours support is part of the role.