Applied Science - Senior Software Development Test Engineer ( SDET )

Overview

Senior SDET role within Tekion's Applied Science team, focused on automated quality assurance for generative AI, LLM-powered products, data pipelines and ML models on Tekion's cloud-native automotive platform.

What you'll do

• Build automated testing suites to detect hallucinations, bias, toxicity, and prompt injection vulnerabilities across LLM-powered products.
• Implement automated evaluations for RAG systems measuring context relevance, groundedness, and answer faithfulness using frameworks like RAGAS or DeepEval.
• Design test beds to validate multi-agent workflows, including tool-calling accuracy, multi-step reasoning, memory, and autonomous decision loops.
• Build and run automated conversation simulations, scripted and synthetic user journeys, to stress-test agent behaviour across intents, edge cases, and multi-turn dialog flows.
• Create prompt regression frameworks to assess how changes in system prompts, temperature, and sampling parameters impact output consistency.
• Statistically validate AI data outputs, including distributions, precision/recall, and error pattern analysis, to catch silent data quality failures before production.
• Programmatically audit data ingestion, transformation, and feature store pipelines for schema drift and data corruption.
• Validate vector DB indexing, embedding semantic similarity accuracy, and retrieval latency.
• Verify quality, diversity, and privacy compliance of synthetic datasets used for model training and evaluation.
• Maintain automated suites tracking ML metrics (Precision, Recall, F1, ROC-AUC) and deep learning loss curves across model versions.
• Implement continuous monitoring scripts to detect data and concept drift on live inference endpoints.
• Build and maintain scalable test automation frameworks for APIs, backend services, and model endpoints.
• Embed AI evaluation and data QA suites into MLOps and CI/CD pipelines so quality failures block releases automatically.
• Define and track AI quality KPIs and communicate release readiness to engineering and product teams.

What you'll need

• Experience of 5+ years in an SDET role.
• Python, expert level: test automation, eval pipelines, data analysis (Pandas, NumPy, Pytest).
• SQL for data output validation, ground truth querying, and pipeline data quality checks.
• LLM evaluation frameworks such as RAGAS, TruLens, DeepEval, or Promptflow for measuring faithfulness, hallucination rate, and task success.
• LangChain, LangSmith, or LlamaIndex for agent workflow testing, prompt tracing, and LLM response debugging.
• OpenAI, Anthropic, or Hugging Face APIs for direct LLM endpoint testing and output consistency validation.
• Vector DBs for retrieval quality testing, embedding validation, and latency benchmarking.
• Pandas, NumPy, or similar for statistical analysis, output validation, error pattern investigation, data profiling, schema validation, and pipeline integrity checks.
• Pytest for modular, reusable test frameworks for AI eval and automation suites.
• Postman, REST Assured, or Requests for API contract validation and service-level integration testing.
• MLflow for tracking model versions and eval runs to detect regressions across updates.
• Docker, GitHub Actions, or Jenkins for containerised test environments and deployment pipeline automation.
• Grafana, Kibana, or OpenTelemetry for monitoring AI system health, output drift, and distributed tracing across agent pipelines.

Nice to have

• Cloud AI services such as AWS Bedrock, Azure OpenAI, or GCP Vertex AI for testing managed model endpoints and cloud-deployed agents.
• MLOps platforms such as MLflow, Kubeflow, Weights & Biases, or Feast feature stores for experiment tracking and model governance.
• Familiarity with Scikit-learn, TensorFlow, or PyTorch for understanding model internals and validating training pipelines.
• Infrastructure as Code with Docker, Kubernetes, or Terraform for managing containerised test environments at scale.
• UI automation with Playwright or Cypress for end-to-end conversational AI application testing.
• Performance engineering with Locust or JMeter for load testing heavy AI inference endpoints under peak traffic.
• Statistical hypothesis testing (t-tests, confidence intervals, significance testing) to distinguish real quality signal from noise.
• Synthetic data generation using LLMs to create diverse test cases and evaluation datasets at scale.

Details

• Location: Bangalore HQ.