Overview
Design, build and optimize scalable data pipelines and data-processing solutions on Azure Databricks as part of Concentrix's Information Technology and Global Security team.
What you'll do
- Design, develop and maintain scalable ETL/ELT data pipelines using Azure Databricks, PySpark, Python and SQL.
- Build and optimize batch and large-scale data-processing solutions using Apache Spark.
- Develop data ingestion and transformation workflows for structured and semi-structured data.
- Implement reliable data-processing frameworks using Delta Lake capabilities.
- Perform data cleansing, validation, reconciliation and quality checks to ensure accuracy and consistency.
- Optimize Spark jobs, Databricks notebooks, SQL queries, clusters and data pipelines for performance and cost efficiency.
- Troubleshoot pipeline failures, data-quality issues, production incidents and performance bottlenecks.
- Implement incremental data processing, partitioning, caching, join optimization and file-management techniques.
- Develop reusable and configurable data-engineering components.
- Implement version control and automated deployment processes using CI/CD and DevOps practices.
- Collaborate with data architects, analysts, application teams and business stakeholders to translate requirements into technical solutions.
- Maintain technical documentation covering pipeline architecture, transformation logic, dependencies and operational procedures.
- Ensure solutions comply with organizational standards for security, governance, monitoring and data protection.
What you'll need
- 4–6 years of relevant experience in data engineering and cloud-based data platforms.
- Strong hands-on experience with Azure Databricks, PySpark, Apache Spark, Python and advanced SQL.
- Proven expertise in developing scalable ETL/ELT pipelines for batch and large-scale data processing.
- Solid understanding of Delta Lake, including Delta tables, ACID transactions, schema evolution, merge/upsert operations and table optimization.
- Experience with Azure data services such as Azure Data Lake Storage and Azure Data Factory.
- Strong knowledge of Spark and Databricks performance optimization, including partitioning, caching, join strategies, shuffle optimization, file sizing and cluster configuration.
- Experience with Git-based version control, CI/CD pipelines and DevOps practices for automated deployment across environments.
- Good understanding of data quality, security, monitoring, troubleshooting and production support.
Details
- Location: Bangalore, India (IND Bangalore - MTP C4, 3rd Floor).
- Employment type: Full time.
Read the full description and apply on the company’s own careers page.