Overview
Design, build and operate scalable data solutions for engineering, energy, oil & gas and infrastructure programmes, turning engineering, project, commercial and operational data into trusted, reusable and AI-ready data products.
What you'll do
- Design, build and maintain scalable enterprise data pipelines using Azure Databricks.
- Ingest structured, semi-structured and unstructured data from complex enterprise and engineering systems.
- Implement Bronze, Silver and Gold/Medallion data architectures.
- Develop reusable ingestion and transformation frameworks rather than project-specific point solutions.
- Build batch, incremental and streaming pipelines appropriate to different data sources and latency requirements.
- Develop transformation logic that converts complex source data into governed enterprise and domain data models.
- Build robust data pipelines with appropriate monitoring, error handling, observability and recovery capabilities.
- Optimise pipelines and Databricks workloads for performance, reliability and cost.
- Provide hands-on engineering capability across Azure Databricks, Apache Spark / PySpark, SQL, Python, Delta Lake, Delta Live Tables / Lakeflow, Databricks Workflows, Unity Catalog, Databricks SQL, Structured Streaming, Change Data Capture, data sharing, CI/CD and Infrastructure as Code, data quality and observability, and performance optimisation.
What you'll need
- 7+ years of experience in data engineering, data platform engineering or an equivalent discipline.
- 3+ years of substantial Azure Databricks experience, ideally within enterprise-scale environments.
- 3+ years of applicable industry experience.
- Strong hands-on capability with Python, PySpark and SQL.
- Strong knowledge of Spark, Delta Lake and lakehouse architectures.
- Experience designing and implementing enterprise-grade data ingestion and transformation pipelines.
- Experience implementing Medallion/Bronze-Silver-Gold architectures.
- Experience working with complex data models and translating logical models into physical implementations.
- Experience with Unity Catalog, data lineage and data governance.
- Experience implementing production-grade CI/CD, testing, monitoring and deployment practices for data solutions.
- Experience handling large volumes of structured and unstructured data.
- Strong understanding of cloud architecture, preferably Microsoft Azure.
- Experience working alongside architects, data scientists, analysts and business/domain SMEs.
Nice to have
- Experience with one or more of: Microsoft Fabric, Microsoft Purview, Power BI semantic models, Azure Data Lake Storage, knowledge graphs and ontologies, digital twins, CFIHOS or similar industry information standards.
Details
- Location: Bangaluru (in-person).
- Level: Senior.
Read the full description and apply on the company’s own careers page.