Sr Data Engineer

Overview

Senior Data Engineer role at Capco, a Wipro company, working on engaging transformation projects for banking, financial and energy clients.

What you'll do

• Build and support enterprise production data pipelines.
• Develop ETL pipelines and batch processing workloads.
• Design and develop Rest API integrations.
• Work with Delta Lakes and Lakehouse architecture.
• Apply Spark fundamentals including partitioning, caching, distributed processing and performance enhancement.
• Implement exception handling in data pipelines.
• Manage infrastructure using Terraform/IaC.
• Apply CI/CD practices to data engineering delivery.

What you'll need

• Python.
• PySpark.
• Spark SQL.
• Databricks.
• Azure.
• CI/CD.
• Rest API.
• ETL pipelines.
• Delta Lakes and Lakehouse architecture.
• Spark fundamentals: partitioning, caching, distributed processing, performance enhancement, exception handling.
• Batch processing.
• ADLS.
• Terraform/IaC.
• Enterprise production pipeline development and support.

Details

• Location: Pune, India.