Sr. Data Engineer-Hadoop,Hive,Spark,Trino, Star Rocks, Clickhouse,Data engineering-3+yrs AI exposure

Overview

Sr. Data Engineers collaborate with stakeholders to clarify data-driven feature requirements, design data architectures and pipelines, and present technical solutions to peers.

What you'll do

• Collaborate proactively with stakeholders to clarify requirements and identify key product stakeholders.
• Translate functional requirements into architecture designs for one or more components, leveraging existing architecture design patterns, and communicate these to Data Engineers.
• Implement extensible, maintainable, and reusable code using appropriate coding patterns, guidelines, and best practices; participate in code reviews to ensure standards are followed.
• Provide end-to-end ownership of data products or data pipelines, including mitigation of technical debt and documentation of errors or unexpected bugs.
• Identify and conduct required testing for moderately complex use cases, including unit testing, coding standards, and security scans.
• Build tools and standard automation processes to assist in transforming, managing, accessing, deploying, and monitoring data processes in batches and in real time.
• Write queries to extract and compile raw data across end-to-end pipelines, and implement orchestration techniques to automate data extraction logic.
• Ensure adherence to data management principles, governance, and tools to maintain data quality across multiple features.
• Develop and update troubleshooting guidance and procedures for reviewing, addressing, and fixing advanced problems flagged by testing.
• Manage software upgrades and server patches for security remediation where applicable.

What you'll need

• 3+ years of relevant work experience and a Bachelor's degree, OR 5+ years of relevant work experience.
• Experience in solving data problems using data technologies.
• Experience with Hadoop, Hive, HBase, Spark, Trino, or comparable distributed data systems.
• Experience in designing and implementing data pipelines and large-scale data processing jobs.
• Experience in writing, modifying, and reviewing high-quality, testable, and efficient code.
• Data pipelines, ETL/ELT, data-quality frameworks, Java or Python, SQL, Git, and production troubleshooting.
• Experience in developing unit, integration, and end-to-end tests with a focus on automation.
• Experience in building and maintaining CI/CD pipelines and deploying applications using containerization and orchestration tools.
• Experience in monitoring applications in production and troubleshooting issues using observability tools.
• Experience in integrating AI or ML services into applications, where applicable.
• Experience in collecting and analyzing metrics to guide optimizations and improvements.
• Experience in partnering with stakeholders to clarify requirements and ensure deliverables meet business needs.
• Experience in applying secure coding practices and complying with regulatory standards.

Nice to have

• 3 or more years of work experience with a bachelor's degree or more than 2 years of work experience with an Advanced Degree (e.g. Masters, MBA, JD, MD).
• Experience in solving data problems using data technologies.
• Experience with Hadoop, Hive, HBase, Spark, Trino, or comparable distributed data systems.
• Experience in contributing to open-source components.
• Use/build AI agents and agentic patterns to improve development, testing, operations, troubleshooting, automation, onboarding, and documentation.
• Data pipelines, ETL/ELT, data-quality frameworks, Java or Python, SQL, Git, and production troubleshooting.
• Experience in enterprise-class systems engineered for high availability, low latency, scalability, and supportability.
• Architecture depth across storage/compute separation, query and table formats, streaming, metadata, governance, and interoperability.
• Experience in building and pushing code into production.
• Experience in implementing and supporting real-time systems.

Details

• Location: Bengaluru, India.
• Work mode: at least 3 days in office, to be confirmed by the Hiring Manager.