PySpark Data Engineer – Python, ETL & Data Warehousing

Overview

PySpark Data Engineer to design, develop, test, deploy, and support scalable ETL pipelines, data marts, and data warehousing solutions.

What you'll do

• Design, develop, test, deploy, and support scalable ETL pipelines and data marts using Python and PySpark.
• Build data processing solutions for structured, semi-structured, and unstructured data.
• Develop clean, maintainable, robust, and reusable Python and PySpark code.
• Develop and optimize SQL and Oracle queries for extraction, transformation, validation, and analysis.
• Integrate data from multiple sources, databases, files, and systems.
• Debug PySpark code and investigate pipeline failures and data quality/processing issues.
• Validate outputs, reconcile results, and ensure quality/business requirements are met.

What you'll need

• 7+ years of overall professional experience in data engineering, software development, or related roles.
• 5+ years of commercial experience in a data-driven role.
• Hands-on experience building data marts and ETL pipelines.
• Strong expertise in Python and PySpark for ETL scripting and testable code.
• Hands-on experience with Spark, PySpark, Hadoop, MapReduce, Hive, and Pandas.
• Strong knowledge of SQL and Oracle query development.
• Experience across the end-to-end SDLC including build, UAT, deployment, and post-production support.