Data Engineering- SSEII

Overview

Data Engineering role focused on operationalizing a unified data platform and maintaining data pipelines using Azure Databricks and related data technologies.

What you'll do

• Develop and maintain a unified data platform using Azure Databricks.
• Design, develop, and maintain scalable data pipelines.
• Establish API integrations for efficient data transfer.
• Implement ETL processes and ensure data integrity and quality.
• Use Spark, Kafka, and Airflow to manage large-scale data processing.
• Test, monitor, and troubleshoot data systems to ensure smooth operation.
• Collaborate with data engineers, analysts, and scientists to optimize data workflows.

What you'll need

• Have 5+ years of development experience with Python, Scala, Java, or C#.
• Have 5+ years of experience with Databricks, Snowflake, BigQuery, Apache Spark, HIVE, Hadoop, Cloudera, or RedShift.
• Demonstrate proficiency in high-performance data pipeline design and pipeline guarantees (e.g., idempotency, exact-once, fault tolerance).
• Have experience developing in a containerized environment such as Docker, Rancher, or Kubernetes.
• Have experience with Apache Airflow (or similar) for orchestrating data processing jobs.
• Hold a Bachelor's degree in Computer Science (or related) or an equivalent education/experience combination.

Details

• Location: Bangalore, Karnataka, India.