Data Engineer

Overview

Data Engineer to build and optimize data infrastructure, including data pipelines and AWS-based data lakes/warehouses.

What you'll do

• Build and optimize data pipelines, warehouses, and data lakes on AWS and Databricks.
• Manage and maintain AWS and Databricks environments.
• Ensure data integrity, accuracy, and consistency via quality checks and monitoring.
• Maintain system uptime and optimal performance.
• Translate business requirements into technical solutions with cross-functional teams.
• Explore and implement new tools to enhance ETL platform performance.

What you'll need

• 5 to 9 years of experience.
• Proficient in SQL for extracting, transforming, and analyzing complex datasets.
• Proven ability to optimize query performance on big data platforms.
• Proficiency with Python, PySpark, and Airflow for scalable data ingestion, transformation, and loading.
• Strong problem-solving and analytical skills.
• Excellent communication and teamwork skills.

Nice to have

• Experience with SQL/NoSQL databases and vector databases for large language models.
• Experience with data modeling and performance tuning for OLAP and OLTP.
• Experience with Apache Spark and Apache Airflow.
• Experience with software engineering best practices including version control, CI/CD, automated unit testing, and DevOps.
• Experience with AWS, GCP, or Azure cloud services.
• AWS Certified Data Engineer certification and/or Databricks certification.

Details

• Department: OUS Contracts & Pricing Hub.
• Department/role category: Information Systems.