Principal Data Engineer

Overview

Principal Data Engineer to design, develop, and optimize complex data pipelines, integration frameworks, and metadata-driven architectures for analytics and enterprise data management.

What you'll do

• Architect and maintain scalable data pipelines for batch and real-time processing.
• Design metadata-driven data engineering frameworks for reusable, consistent pipelines.
• Optimize performance via Spark tuning, resource utilization, scheduling, and query improvements.
• Implement ingestion, transformation, lineage tracking, and platform observability solutions.
• Define and uphold SLOs, monitoring standards, and data quality KPIs for production pipelines.
• Collaborate with platform, architecture, analytics, and governance teams to align enhancements with enterprise strategy.
• Mentor engineers and promote knowledge sharing and continuous improvement.

What you'll need

• 12 to 17 years of experience in Computer Science, IT, or related field.
• Hands-on experience with Databricks, PySpark, SparkSQL, Delta Lake, and Apache Spark.
• Working knowledge of AWS services and strong understanding of AWS-based data engineering.
• Proficiency in workflow orchestration and performance tuning on big data processing.
• Experience with Data Fabric, Data Mesh, or similar enterprise-wide data architectures.
• Strong problem-solving/analytical skills and excellent communication/teamwork skills.
• Experience with Scaled Agile Framework (SAFe), Agile delivery practices, and DevOps practices.

Details

• Location: Hyderabad, India.