Senior Software Engineer

Overview

Build and operate the data pipelines, retrieval systems and evaluation harnesses that make Clearwater's AI agents accurate, grounded and measurable, working on security and threat data at scale.

What you'll do

• Build and operate data pipelines (batch and streaming) that ingest, normalize, and enrich large volumes of security and telemetry data
• Design retrieval systems for agents: chunking, embeddings, hybrid search, reranking, vector and graph stores, and freshness and access control
• Build evaluation harnesses and datasets to measure agent and retrieval quality, regressions, and cost
• Build backend APIs and services in Python, and the full-stack features that surface data and AI insights to users
• Design data models and manage relational, analytical, and vector stores for performance and cost
• Add data quality checks, lineage, observability, and governance, including PII and sensitive-data handling
• Use AI agents to automate data engineering work such as pipeline generation, data validation, and documentation
• Containerize and deploy on Kubernetes and cloud with CI/CD, and mentor engineers on data and AI practices

What you'll need

• 4+ years of professional software engineering with strong data-engineering exposure and sound testing and CI/CD practice
• 1+ year building production systems involving LLMs, retrieval (RAG), or large-scale data pipelines
• Strong Python and SQL, including query optimization and data modeling
• Hands-on with pipeline or stream-processing tooling such as Airflow, Dagster, Spark, or Kafka, or an equivalent
• Experience with vector stores and retrieval design (pgvector, OpenSearch, or similar), and with evaluating retrieval quality
• Docker, working Kubernetes knowledge, and experience on AWS, Azure, or GCP

Nice to have

• Working JavaScript/TypeScript, enough to build and ship full-stack features
• Experience with security, threat intelligence, or SIEM/telemetry data
• Healthcare or compliance-driven data experience (HIPAA, SOC 2, HITRUST)
• Knowledge graphs or graph databases
• LLMOps and evaluation tooling experience
• Experience with a lakehouse or warehouse such as Snowflake, Databricks, or BigQuery
• Open-source contributions or public data or retrieval projects

Details

• Location: Pune (M.H)
• Fast-paced environment handling multiple demands; requires a high level of initiative and independence
• Excellent written and oral communication skills required
• Requires the ability to use a personal computer for extended periods of time