Sr.Data Engineering

Overview

Sr. Data Engineering role focused on building and optimizing ETL/ELT pipelines and data solutions on the Azure stack.

What you'll do

• Design, build, and optimize scalable ETL/ELT data pipelines using Python and PySpark.
• Develop ingestion and transformation processes from databases, APIs, and streaming sources.
• Implement unit/integration tests and set up monitoring, alerting, and dashboards for data quality and pipeline health.
• Support data infrastructure setup and maintenance on OpenShift using tools like HELM.
• Drive CI/CD automation using GitHub Actions for testing, builds, and deployments to environments like OpenShift.
• Develop and optimize complex SQL queries and apply data modeling principles for SQL Server and other stores.

What you'll need

• 6–8 years of relevant experience.
• Strong proficiency in Python and PySpark for large-scale data processing and ETL.
• Expertise in SQL for complex querying, data manipulation, and schema design.
• Hands-on experience with Azure Databricks and Azure Analytics services (Azure Data Factory, Azure SQL Server, Azure Key Vault).
• Experience implementing CI/CD pipelines from GitHub (GitHub Actions) with automated unit testing, build, and deployment.
• Experience with OpenShift and HELM for deployment on Kubernetes/OpenShift.
• Experience configuring Grafana for data quality and pipeline health monitoring.

Nice to have

• Experience in SQL optimization and performance tuning.
• Bachelor’s or Master’s degree in a related quantitative field.
• Relevant Azure certifications (e.g., Azure Data Engineer Associate).
• Experience with real-time data processing frameworks (e.g., Kafka, Azure Event Hubs).
• Knowledge of data governance, data security, and compliance best practices.

Details

• Location: Bengaluru, India.