Staff Software Engineer

Overview

Staff Software Engineer responsible for setting the technical direction of Prophecy’s Runtime service for orchestrating and executing data workflows at enterprise scale.

What you'll do

• Architect and evolve the runtime service that turns agent- and human-authored workflows into reliable production jobs and interactive previews.
• Design distributed execution and task-coordination covering scheduling, concurrency, resource management, and streaming of sample data.
• Extend execution across multiple engines and warehouses (Databricks, Snowflake, BigQuery, Spark, dbt, Airflow) with connectors and consistent behavior.
• Build the execution path for agents to generate, run, validate, and refine pipelines in a tight loop.
• Own reliability and observability, including fault tolerance, graceful degradation, recovery, restarts, monitoring, and SLA-related robustness.
• Lead technically by setting architecture/standards, driving cross-team designs, and mentoring engineers.

What you'll need

• 5+ years of backend engineering experience building distributed systems or data-intensive services.
• Deep expertise in distributed systems fundamentals such as scheduling, coordination, concurrency, resource management, and fault tolerance.
• Strong proficiency in Go, Scala, or Java (runtime primarily Go and Python).
• Hands-on experience running data at scale with Spark and/or cloud data warehouses (Databricks, Snowflake, BigQuery).
• Ability to own reliability, performance, and observability of a production service.
• Experience leading technically across teams and raising engineering quality bar.
• B.Tech or above degree in CS or equivalent.

Nice to have

• Experience integrating with multiple execution backends or building multi-cloud data connectors.
• Familiarity with AI/agent execution paths, including running and validating LLM/agent-generated code.
• Experience building an execution engine, orchestrator, or query engine (examples include Airflow, Spark internals, query planners/optimizers like Apache Calcite, or dbt).
• Shipped software over multiple releases across systems where you drove delivery of the first version.

Details

• Location: Bengaluru, Karnataka, India.
• Work described as driving enterprise-scale execution against customer platforms (examples listed: Databricks, Snowflake, BigQuery and more).