Sr.Data Engineering

Overview

Sr. Data Engineering role focused on designing and running scalable data pipelines and Azure-based data solutions.

What you'll do

• Design, build, and optimize ETL/ELT data pipelines using Python and PySpark.
• Ingest data from sources such as databases, APIs, and streaming, and transform it for downstream use.
• Implement unit and integration tests and set up monitoring/alerting/dashboarding for pipeline health and quality (e.g., Grafana).
• Contribute to setup, configuration, and maintenance of data infrastructure on OpenShift using tools like HELM.
• Drive CI/CD best practices with GitHub Actions, including automated testing, build, and deployment.
• Develop and optimize complex SQL queries and apply data modeling for SQL Server and other stores.

What you'll need

• 6–8 years of relevant experience.
• Strong proficiency in Python and PySpark for large-scale data processing and ETL development.
• Expertise in SQL for complex querying, data manipulation, and schema design.
• Hands-on experience with Azure Databricks.
• Proficiency with Azure analytics services including Azure Data Factory, Azure SQL Server, and Azure Key Vault.
• Experience implementing CI/CD pipelines using GitHub Actions with automated unit tests, build, and deployment.
• Experience setting up Grafana dashboards for data quality and pipeline health monitoring.