Staff Software Development Engineer - Data Platforms

Overview

Staff Software Development Engineer on the Data Platforms team at JioStar, designing and operating large-scale data infrastructure that serves hundreds of millions of requests per minute, processes billions of daily events from Kafka, and manages petabytes of data in S3, including the data foundations for AI initiatives and an MCP server.

What you'll do

• Build POCs, create architecture designs, and evaluate trade-offs for new services and migrations
• Build and maintain highly scalable, accurate, clean and real-time events collection systems
• Design and build data pipelines to support the development and training of AI models and autonomous AI Agents, including robust data infrastructure for the MCP, ensuring data quality and availability for personalized learning paths and skill validation
• Detect, alert on, and fix data quality issues to build trust in the data
• Improve existing systems and services using latest tooling and engineering standards
• Improve unit economics across ingestion, storage, and processing components
• Own the entire development process from ideation to deployment
• Work closely with different teams to achieve common goals through efficient coordination
• Maintain up-to-date documentation with architecture diagrams for entire projects
• Take complete ownership of services within your workstream and drive continuous improvement
• Establish and maintain robust monitoring, alerting, and operational procedures for mission-critical data systems
• Write blogs and articles to help team members and the community
• Break down high-level architecture into manageable services and tasks
• Learn and share new trends in technology, engineering, and productivity
• Actively work towards improving team productivity and knowledge
• Guide and mentor junior team members
• Govern data assets across the entire company and audit data posture
• Reduce time to insight for data team stakeholders
• Assist and challenge business decisions using data insights
• Make data available to all stakeholders in an efficient, quick, queryable, and intuitive manner

What you'll need

• 8+ years of progressive professional experience in data engineering, with a proven track record of designing, building, and operating large-scale, high-throughput data platforms, ideally in a consumer-facing or streaming industry
• Demonstrated experience working with data systems that handle hundreds of millions of events/requests per minute, billions of daily events, and petabytes of storage (e.g., S3, HDFS)
• Strong proficiency in at least one major programming language such as Python, Java, Scala, or Go, with a focus on building robust, production-grade data applications
• Deep expertise and practical experience with distributed data processing frameworks like Apache Spark, Apache Flink, or equivalent technologies for batch and streaming workloads
• Specific experience in building data pipelines for Machine Learning model training, inference, and feature stores
• Familiarity with data needs for AI Agents or similar intelligent systems
• Extensive experience with high-throughput messaging queues and stream processing platforms like Apache Kafka, Kinesis, or equivalent
• Hands-on expertise with at least one major cloud platform (AWS, GCP, or Azure) and their data-related services
• Strong understanding and hands-on experience with modern data warehousing concepts and data lake architectures (e.g., Snowflake, BigQuery, Redshift, Delta Lake, Iceberg, Hudi)
• Expertise in building, optimizing, and orchestrating complex ETL/ELT pipelines using tools like Apache Airflow, Prefect, or other workflow management systems
• Strong SQL skills and experience with both relational (e.g., PostgreSQL, MySQL) and NoSQL databases (e.g., Cassandra, DynamoDB)
• Advanced data modeling skills (dimensional, Kimball, data vault) for large, complex datasets
• Exceptional analytical and problem-solving abilities, with a proven track record of debugging and optimizing complex distributed systems
• Excellent communication, collaboration, and interpersonal skills, with the ability to articulate complex technical concepts to diverse audiences
• Proven ability to mentor, guide, and technically lead senior and junior engineers

Nice to have

• Strong preference for AWS experience (S3, EMR, Kinesis, Glue, Redshift, etc.)

Details

• Location: Bengaluru
• Work mode: We Work (office-based)