Concentrix logo

Data Engineer

Concentrix
NewPosted today

LOCATION

Bangalore · Onsite

EXPERIENCE

4 - 6 Years

TYPE

FullTime

SKILLS REQUIRED

Data EngineeringApache SparkDelta LakeData PipelinesAzure Data Lake StoragePythonPySpark

Job description

Overview

Design, build and optimize scalable data pipelines and data-processing solutions on Azure Databricks as part of Concentrix's Information Technology and Global Security team.

What you'll do

  • Design, develop and maintain scalable ETL/ELT data pipelines using Azure Databricks, PySpark, Python and SQL.
  • Build and optimize batch and large-scale data-processing solutions using Apache Spark.
  • Develop data ingestion and transformation workflows for structured and semi-structured data.
  • Implement reliable data-processing frameworks using Delta Lake capabilities.
  • Perform data cleansing, validation, reconciliation and quality checks to ensure accuracy and consistency.
  • Optimize Spark jobs, Databricks notebooks, SQL queries, clusters and data pipelines for performance and cost efficiency.
  • Troubleshoot pipeline failures, data-quality issues, production incidents and performance bottlenecks.
  • Implement incremental data processing, partitioning, caching, join optimization and file-management techniques.
  • Develop reusable and configurable data-engineering components.
  • Implement version control and automated deployment processes using CI/CD and DevOps practices.
  • Collaborate with data architects, analysts, application teams and business stakeholders to translate requirements into technical solutions.
  • Maintain technical documentation covering pipeline architecture, transformation logic, dependencies and operational procedures.
  • Ensure solutions comply with organizational standards for security, governance, monitoring and data protection.

What you'll need

  • 4–6 years of relevant experience in data engineering and cloud-based data platforms.
  • Strong hands-on experience with Azure Databricks, PySpark, Apache Spark, Python and advanced SQL.
  • Proven expertise in developing scalable ETL/ELT pipelines for batch and large-scale data processing.
  • Solid understanding of Delta Lake, including Delta tables, ACID transactions, schema evolution, merge/upsert operations and table optimization.
  • Experience with Azure data services such as Azure Data Lake Storage and Azure Data Factory.
  • Strong knowledge of Spark and Databricks performance optimization, including partitioning, caching, join strategies, shuffle optimization, file sizing and cluster configuration.
  • Experience with Git-based version control, CI/CD pipelines and DevOps practices for automated deployment across environments.
  • Good understanding of data quality, security, monitoring, troubleshooting and production support.

Details

  • Location: Bangalore, India (IND Bangalore - MTP C4, 3rd Floor).
  • Employment type: Full time.

Read the full description and apply on the company’s own careers page.

Stay safe

Hiring on Abekus is free for applicants

We never charge a fee, and employers are prohibited from doing so. If a recruiter asks for payment, please report them right away.

Data Engineer