Python with Spark Developer (5.1-7 years)-Chennai

Overview

Python + PySpark developer with 5–7 years of experience to design, develop, test, and maintain high-performance data processing pipelines.

What you'll do

• Design and develop ingestion, transformation, and enrichment pipelines using Python, PySpark, and SQL.
• Write and optimize complex SQL queries, analytical UDFs, and window functions for aggregation and reporting.
• Collaborate with data architects, data scientists, and business analysts to turn requirements into technical specs.
• Unit-test, integration-test, and review code.
• Maintain CI/CD pipelines using Git, Jenkins, and Docker for automated builds, tests, and deployments.
• Monitor production workloads and troubleshoot performance, memory issues, and job failures.

What you'll need

• 5–7 years of experience as a Python Developer with Spark.
• Strong Python, PySpark, and SQL skills for data-processing pipelines.
• Sound knowledge of Agile (Scrum/Kanban).
• Hands-on PySpark skills including DataFrames, Spark SQL, Structured Streaming, and performance tuning.
• Advanced SQL skills including complex queries, stored procedures, and query optimization.
• Comfort working with Linux/Unix, testing, documentation, and new framework.
• Bachelor’s degree or equivalent.

Details

• Location: Chennai, India.