Staff-Data Engineer

Overview

Staff Data Engineer to architect and optimize big data platforms with a strong focus on Databricks, driving the core design and performance of the lakehouse that enables analytics, AI/ML, and product innovation at scale.

What you'll do

• Lead the architecture, design, and evolution of highly scalable, cloud-native backend systems and APIs.
• Design, build, and own Python-based API services (e.g., FastAPI) that expose data platform capabilities to consumers.
• Design and optimize relational database schemas (PostgreSQL), including indexing, query tuning, and schema migration management.
• Drive technical strategy for reliability, scalability, performance, and cost efficiency across data-intensive workloads.
• Own complex backend initiatives end-to-end, from design through production rollout and operational maturity.
• Own and maintain CI/CD pipelines (GitHub Actions, Terraform), deploying your own work to production rather than handing off to a separate team.
• Debug across system boundaries, tracing issues from the API layer through the database to the data pipeline and back.
• Collaborate closely with data engineering, frontend, platform, and SRE teams to ensure cohesive system design and execution.
• Influence engineering standards and best practices around code quality, testing, CI/CD, observability, and operational excellence.
• Review and guide designs and implementations across teams, acting as a trusted technical advisor.
• Mentor senior and junior engineers, helping grow technical depth and leadership within the organization.
• Stay ahead of industry trends in backend systems, cloud platforms, and data engineering, applying them thoughtfully at scale.

What you'll need

• Bachelor's or master's degree in computer science, engineering, or equivalent practical experience.
• 8+ years of backend engineering experience, with a proven track record of building and scaling complex production systems.
• Hands-on Python application/service development, building production APIs and services, not limited to PySpark notebooks or data science scripts.
• Strong RDBMS experience (PostgreSQL preferred), including schema design, complex query authoring and optimization, migration tooling (Flyway), and query-plan-level performance debugging.
• API development ownership: designing, building, deploying, and operating RESTful APIs (FastAPI, Flask, or equivalent).
• CI/CD pipeline ownership: building and maintaining deployment automation (GitHub Actions, Azure DevOps, or similar), not relying on a separate platform team.
• Infrastructure-as-Code: Terraform, or similar; comfortable provisioning cloud resources yourself.
• Experience building large-scale data or analytics platforms.
• Strong background in observability, monitoring, and incident-driven learning, including distributed tracing, query profiling, and latency diagnosis (not just dashboard monitoring).
• Experience with security, compliance, and enterprise-grade SaaS requirements.
• Prior experience in FinOps, cloud cost optimization, or enterprise SaaS platforms.
• Experience with AWS (ECS/Fargate, RDS, S3, IAM, Secrets Manager).
• Experience building AI-powered backend systems, including LLM-based or agent-driven workflows.

Nice to have

• Experience with Power BI Direct Lake mode and semantic modeling for lakehouse-based analytics.
• Familiarity with Mosaic AI, Databricks GenAI capabilities, or Feature Store implementations.
• Contributions to open-source Spark, Databricks community content, or certifications (e.g., Databricks Certified Data Engineer Professional).

Details

• Location: Home-India (remote, India).
• Flexible work arrangements offered.