Data Engineer

Overview

Design, develop and maintain data solutions for data generation, collection and processing, creating data pipelines, ensuring data quality and implementing ETL processes to migrate and deploy data across systems.

What you'll do

• Design, develop and maintain data solutions for data generation, collection and processing.
• Create data pipelines and ensure data quality.
• Implement ETL (extract, transform and load) processes to migrate and deploy data across systems.
• Collaborate with multiple teams to optimize data workflows and troubleshoot issues.
• Lead efforts to optimize data pipeline performance and scalability.
• Provide solutions to problems that apply across multiple teams.
• Mentor junior team members to support their professional growth and development.

What you'll need

• Minimum 12 years of experience is required.
• Minimum 7 years of experience in Microsoft Azure Databricks.
• Proficiency in Microsoft Azure Databricks.
• Proficiency in Python (Programming Language).
• Proficiency in PySpark.
• Strong knowledge of cloud-based data platforms and distributed computing.
• Experience in designing and implementing scalable ETL workflows.
• Familiarity with data storage solutions and data modeling techniques.
• Ability to troubleshoot and optimize complex data processing tasks.
• 15 years full time education is required.

Nice to have

• Experience with Python (Programming Language).
• Experience with PySpark.

Details

• Based at the PDC3B - SEZ location.