Azure Data Engineer

Overview

Design, build and optimize data pipelines and storage on the Azure platform, working with Databricks and Apache Spark to deliver scalable data solutions for analysts, data scientists and business stakeholders.

What you'll do

• Build data pipelines to ingest data from databases, APIs or streaming platforms.
• Integrate and transform data to ensure compatibility with the target data model or format.
• Design and optimize data storage architectures, including data lakes, data warehouses or distributed file systems.
• Implement partitioning, compression or indexing to optimize data storage and retrieval.
• Identify and resolve bottlenecks, tune queries and implement caching strategies to enhance data retrieval speed and system efficiency.
• Design and implement data models that support efficient data storage, retrieval and analysis.
• Write efficient and scalable code to process, transform and clean large volumes of structured and unstructured data.
• Design and implement data processing systems on the Azure platform.
• Collaborate with data scientists, analysts and business stakeholders to understand requirements and provide well-structured, optimized data and technical solutions.
• Communicate complex technical concepts to non-technical stakeholders in a clear and concise manner.
• Take independent responsibility for delivering a solution.
• Train and mentor junior data engineers, providing guidance and knowledge transfer.
• Work under Agile and Scrum development methodologies.

What you'll need

• Databricks, level 4 of 5.
• Apache Spark, level 4 of 5.
• Programming for data engineering, level 4 of 5.
• Software development for data engineering, level 4 of 5.
• Azure services, level 3 of 5.
• Technical leadership, level 3 of 5.
• Architecture, data processing, level 3 of 5.
• GenAI for data engineering, level 3 of 5.
• CI/CD automation, level 3 of 5.
• 4-6 years of professional experience in a similar role.
• Very good communication, including the ability to convey information clearly and specifically to co-workers and business stakeholders.
• Working experience in agile methodologies and supporting tools (JIRA, Azure DevOps).
• Hands-on experience designing and optimizing data storage (data lakes, data warehouses, distributed file systems).
• Independence and responsibility in delivering solutions.
• Ability to work under agile methodologies.

Nice to have

• Experience or familiarity with other cloud technologies, data warehouses, data governance and business analysis.