Senior Data Engineer

Overview

Build and maintain systems that collect, store, process and analyse data, including data pipelines, data warehouses and data lakes. As a Senior Data Engineer, deliver technology solutions and work on complex technical problems with engineers, business analysts and business stakeholders.

What you'll do

• Build and maintain data architectures and pipelines that transfer and process durable, complete and consistent data.
• Design and implement data warehouses and data lakes that manage appropriate data volumes and velocity and meet required security measures.
• Develop processing and analysis algorithms suited to intended data complexity and volumes.
• Collaborate with data scientists to build and deploy machine learning models.
• Perform prescribed activities in a timely manner and to a high standard while driving continuous improvement.
• Lead and supervise a team, guide and support professional development, allocate work requirements and coordinate team resources.
• Develop technical expertise in the work area and act as an advisor where appropriate.
• Partner with other functions and business areas.
• Take responsibility for the end results of a team's operational processing and activities.
• Escalate policy and procedure breaches appropriately.
• Embed new policies and procedures adopted due to risk mitigation.
• Advise and influence decision-making within the area of expertise.
• Manage risk and strengthen controls in relation to owned or contributed work.
• Deliver work and areas of responsibility in line with relevant rules, regulations and codes of conduct.
• Build understanding of how the sub-function integrates with the function and how areas coordinate to achieve organisational sub-function objectives.
• Make evaluative judgements based on factual information, paying attention to detail.
• Resolve problems by identifying and selecting solutions through acquired technical experience and precedents.
• Guide and persuade team members and communicate complex or sensitive information.
• Act as a contact point for stakeholders outside the immediate function and build a network of contacts outside the team and external to the organisation.

What you'll need

• Experience with AWS, hands-on with S3, EC2, Lambda, RDS, Glue, Athena, data pipelines and Step Functions, and knowledge of analytical services.
• Experience of distributed computing architecture.
• Knowledge of core Hadoop components: HDFS, Spark, YARN, Hive and Impala.
• Expertise in SQL and advanced SQL skills.
• Expertise in shell scripting and programming with Scala or Python.
• Knowledge of ETL technologies, preferably Ab Initio.
• Experience of technical design for ETL, Hadoop and AWS components.
• Knowledge of data warehousing.

Details

• Location: Pune, India.