DevOps Engineer II - DataBricks

Overview

DevOps Engineer II focused on hands-on operation of Databricks, production databases, AWS and Kubernetes infrastructure at CommerceIQ.

What you'll do

• Manage and maintain Databricks infrastructure and production workloads.
• Manage clusters, jobs, compute resources, configurations, permissions, and integrations.
• Monitor Databricks workloads and troubleshoot performance and reliability issues.
• Drive Databricks cost optimization through cluster sizing, autoscaling, job optimization, scheduling, and resource utilization.
• Establish best practices around Databricks governance, security, access management, and operational efficiency.
• Perform database maintenance, monitoring, troubleshooting, performance tuning, backup/recovery, and capacity management.
• Troubleshoot database performance, connectivity, storage, replication, and availability issues.
• Identify opportunities for database cost and performance optimization.
• Manage and maintain production AWS infrastructure with a strong focus on reliability, security, scalability, and cost optimization.
• Troubleshoot complex infrastructure and networking issues.
• Deploy, troubleshoot, scale, and maintain Kubernetes workloads, networking, ingress, storage, and clusters.
• Build and maintain CI/CD pipelines and automate repetitive operational tasks.
• Implement Infrastructure as Code and follow best practices for infrastructure provisioning and management.
• Build dashboards, alerts, and operational metrics for production systems.
• Participate in incident management, root-cause analysis, and reliability improvements.
• Ensure production systems meet availability, performance, and operational requirements.

What you'll need

• 4–7+ years of DevOps / Cloud / Infrastructure experience.
• Strong hands-on experience with Databricks.
• Strong experience managing databases in production environments.
• Work with databases such as PostgreSQL, MySQL, Aurora, Redis, or similar technologies.
• Hands-on experience with AWS infrastructure and services, including EKS, EC2, RDS/Aurora, S3, IAM, CloudWatch, VPC, ECR, and related services.
• Strong hands-on experience with Kubernetes, preferably AWS EKS.
• Experience with Helm, ArgoCD, Terraform, GitHub Actions/Bitbucket Pipelines, or similar DevOps tools.
• Experience with monitoring and observability tools such as Prometheus, Grafana, CloudWatch, OpenTelemetry, or similar.
• Strong database administration and troubleshooting experience.
• Terraform / Infrastructure as Code.
• CI/CD and Git-based workflows.
• Helm and GitOps tools such as ArgoCD.
• Monitoring and observability.
• Strong Linux and networking fundamentals.
• Strong troubleshooting and problem-solving skills.
• Experience working with production environments at scale.

Details

• Location: Bengaluru, Karnataka, India.