Principal, Data Engineer

Overview

Principal Data Engineer role at Walmart Global Tech, designing, building and deploying large-scale data solutions and pipelines for Walmart's Corporate Tech teams.

What you'll do

• Design, build, test, and deploy cutting-edge data solutions at Walmart scale, impacting millions of customers worldwide.
• Develop high-performance, scalable Spark and Big Data solutions to extract, transform, and load large-scale datasets.
• Perform root cause analysis on data and processes to answer business questions and identify improvement opportunities.
• Build and optimize data pipelines, architectures, and datasets handling terabytes and petabytes of data.
• Collaborate with Walmart engineering teams across geographies to leverage expertise and contribute to the technology community.
• Partner with Product Management and business teams to drive priorities and deliver innovative product features.
• Work closely with Data Engineers to evaluate and implement open-source tools through research, POCs, and pilot programs.
• Support business and product teams by building data solutions and developing deep expertise in data analytics.
• Collaborate with team members to define best practices and meet client requirements for software development.
• Analyze and test programs and products before launch to ensure high-quality performance.
• Develop solutions and monitoring mechanisms to ensure software security and protect sensitive information.
• Continuously identify opportunities to enhance software performance, scalability, and effectiveness.
• Mentor, coach, and support team members to strengthen their confidence and proficiency in software applications.
• Contribute to a culture of innovation, collaboration, and continuous improvement across the engineering organization.

What you'll need

• Bachelor's or master's degree in computer science or a related technical field.
• 12 years of hands-on object-oriented programming experience in Python.
• 8+ years of hands-on experience in Database Fundamentals, SQL, PLSQL, Store Procedures.
• 7+ years of experience in designing and developing large-scale, fault-tolerant, highly scalable data pipelines using big data technologies (e.g., Apache Spark, Kafka, Cassandra, Hadoop, Hive, Presto, Airflow).
• Strong understanding and experience in system design, algorithms, distributed systems, and high-scale data processing.
• Deep understanding of ETL, Data Modeling and Data Warehousing concepts.
• Strong programming expertise to understand requirements, design high-level and low-level solutions (HLD/LLD), and implement design patterns with S.O.L.I.D principles.
• Experience in end-to-end product development, emphasizing configuration-driven, behavior-driven, and test-driven development (including unit test cases and DevOps best practices).
• Designing systems that gracefully handle failures, high concurrency, data inconsistencies, and edge cases.
• Ensuring zero data loss and 100% accuracy in data pipelines, even under high-volume traffic.
• Implementing and tuning systems that process petabytes of data with sub-second latency.
• Experience in mentoring junior resources.
• Platform driven approach.
• Implementing fine-grained access controls, encryption, and data masking techniques.
• Implementing real-time monitoring, logging, tracing, and alerting to ensure high availability and proactive issue resolution.
• Writing robust integration, regression, and performance test cases to detect defects early.
• Experience with Jenkins, GitHub Actions, or GitLab CI/CD to streamline release management.
• Ability to prioritize and make trade-offs between performance, cost, and reliability.
• Expertise in failure analysis, RCA (Root Cause Analysis), and rollback strategies.
• Experience in building large-scale distributed systems, ensuring scalability and fault tolerance.
• Strong exposure to cloud infrastructure (AWS, GCP, OpenStack, etc.).
• Experience working with metadata-driven development approaches.
• Passion for automation, optimization, and improving system efficiency.
• Strong problem-solving, critical thinking, and analytical skills.
• Excellent communication skills, with the ability to explain complex technical concepts effectively.
• Minimum qualifications, Option 1: Bachelor's degree in Computer Science and 5 years' experience in software engineering or related field.
• Minimum qualifications, Option 2: 7 years' experience in software engineering or related field.
• Minimum qualifications, Option 3: Master's degree in Computer Science and 3 years' experience in software engineering or related field.
• Minimum qualifications: 4 years' experience in data engineering, database engineering, business intelligence, or business analytics.

Details

• Location: Bangalore, Karnataka, India (Block-1, Prestige Tech Pacific Park, Kadubeesanahalli Village, Varthur Hobli).
• Work mode: based in the Bangalore office for daily work, with flexibility for associates to manage their personal lives.
• Experience range: 12 to 16 years.