Data Platform Engineer

Overview

Hands-on Data Platform Engineer role on NexusOne's product team in Pune, supporting, operating and maintaining customer data platform clusters across cloud and on-premises environments.

What you'll do

• Install, configure, and support NX1 product clusters in cloud (AWS, Azure, GCP) and on-premises environments, ensuring seamless integration and functionality using Kubernetes, Spark, NiFi, Kafka, Airflow, Trino, Hive, Iceberg, Jupyter, YARN, Impala, Ranger and HBase, ensuring robust data protection and compliance.
• Maintain, patch, and upgrade existing NX1 product setups in 24x7 environments, ensuring minimal downtime and adherence to best practices.
• Configure Datadog/Grafana agents and develop dashboards to monitor and alert operations teams.
• Manage daily operations with on-call rotations of NexusOne products and cloud-based technologies using open-source Linux-based data technologies.
• Evaluate and optimize complex distributed production deployments, identify bottlenecks, and recommend performance enhancements.
• Develop and maintain technical documentation, including administration of runbooks and knowledge base articles.
• Work in an agile team and participate in an on-call rotation, providing expert-level support and troubleshooting for critical issues.
• Lead and mentor junior team members, fostering a culture of continuous learning and improvement.
• Collaborate with cross-functional teams, including data engineers, solution architects, and DevOps, to design and implement scalable data solutions.
• Stay current with emerging technologies and industry trends, applying this knowledge to improve the CDP environment and drive innovation.

What you'll need

• Minimum of 5 years of experience installing and administering a Data Platform (Cloudera, Databricks, Snowflake) or using open-source tools, with a proven track record of managing large-scale deployments.
• Hands-on experience with Python/Terraform automation skills for open-source Linux-based data technologies.
• Proven ability to build and support solutions across major cloud platforms (AWS, Azure, GCP), leveraging cloud-native tools and open source for optimal performance.
• Experience integrating Azure Active Directory with FreeIPA and other directory services such as LDAP, enhancing security and user management.
• Up-to-date knowledge of the Hadoop Big Data ecosystem, staying current with the latest technologies and best practices.
• Excellent troubleshooting skills, with a thorough understanding of capacity planning, identifying bottlenecks, and optimizing memory utilization, CPU usage, OS performance, storage, and network configurations.
• In-depth understanding of both on-premise and cloud network architectures, ensuring seamless integration and efficient data flow.
• Strong analytical and problem-solving abilities, capable of diagnosing and resolving complex technical issues efficiently.

Nice to have

• Cloudera/Databricks/Snowflake Administrator certification.
• Prior experience as a support or development engineer with a focus on Data, Analytics, and AI.

Details

• Location: Pune HQ.
• Role requires participation in an on-call rotation and 24x7 environment support.