Data Engineer II

Overview

Data Engineer II role supporting multiple Data Analysts and Data Scientists day-to-day and assisting various data-driven requests across the company, spanning coding, infrastructure, data analysis and database management.

What you'll do

• Build and maintain database technologies, environments, and applications, seeking opportunities for improvements and efficiencies
• Work with cloud data platforms such as Databricks and/or Cloudera, Iceberg/Delta tables
• Write SQL statement procedures to optimize SQL execution and query development
• Use technologies such as Spark (Scala, Python), Apache Kafka, Trino
• Manage the deployment of data solutions that are optimally standardized and database updates to meet project deliverables
• Lead database security posture, including proactively identifying security risks and implementing risk mitigation plans and control functions
• Support process improvement efforts to identify and test opportunities for automation and/or reduction in time to deployment
• Participate in the entire solution development cycle from requirement gathering to solution deployment
• Work with multiple stakeholders across teams to fulfill ad-hoc investigations, including large-scale data extraction, transformation, and analyses
• Support production code to produce comprehensive and accurate data artifacts
• Promote strategies to improve data modeling, quality, and architecture

What you'll need

• Experience working with Databricks and/or Cloudera platforms
• Skilled in Spark framework (Scala/Python), Apache Kafka, Spark Streaming
• Highly skilled in SQL and relational databases
• Experienced with developing and maintaining data pipelines for the extraction, transformation, and/or loading of large-scale data
• Experienced with working with large data sets, including implementing parallel processing and dealing with the limitations and challenges of terabytes of data
• Strong analytics skills related to working with unstructured datasets
• Experienced with performing root cause analysis on internal and external data and processes to answer specific business questions and identify improvement opportunities
• Experienced with database administration and query optimization across different databases
• Experienced with at least one programming language such as Python
• Capable of effectively communicating requirements and solutions to audiences with different skill sets
• Familiarity with the core requirements of data analysis and/or presentation
• Focused, innovative, and self-sufficient problem solver who can independently deep-dive into an issue and solve it

Nice to have

• Prior experience with AWS: S3, EMR, Ozone, Iceberg
• Prior experience with Python, Javascript, PLSQL, regex
• Prior experience with Jenkins, Datadog, Splunk, Dynatrace

Details

• Location: Pune, India