Senior AIOps Engineer

Overview

Senior AIOps Engineer I role at Porch Group focused on keeping AI and ML-powered systems reliable, observable, secure, and cost-efficient in production, across monitoring, deployment pipelines, incident management, and new AI infrastructure.

What you'll do

• Own production reliability for AI/ML services by monitoring availability and performance and defining SLOs and SLIs (latency, error rates, model performance) tied to user experience and business impact.
• Own recurring model refresh cycles (retraining, validation, and redeployment) to prevent staleness and drift.
• Design and implement monitoring, logging, and alerting for models and data pipelines, and integrate model and system metrics with existing observability stacks and dashboards (Datadog, Opik, and others).
• Streamline deployment workflows with data scientists and ML engineers (blue/green, canary, A/B, and shadow deployments), improve CI/CD pipelines and release processes for AI services, and support the productionization of image-based ML models, including batch prediction workflows.
• Provision, deploy, configure, and maintain new AI infrastructure as it is adopted, including AI gateways, RAG platforms, LLM observability tools, agentic workflows, and no-code agent builders, using tools such as Union Cloud, BentoML, Feast, Kubernetes, Terraform, and GCP services.
• Build automation and tooling to reduce manual operational work around model promotion, configuration, environment management, and Docker image maintenance, and support multi-BU infrastructure provisioning (separate environments, projects, roles, and CI/CD integrations).
• Own the operational health of Label Studio, our production data annotation platform, including supporting GCS buckets, BigQuery pipelines, and database and SSO dependencies, and supporting bi-weekly annotation project cycles.
• Stand up and operate RAG and vector database infrastructure, including embedding pipelines and context retrieval APIs, and optimize embedding storage and retrieval performance at scale.
• Monitor and optimize LLM token usage and costs across providers, maintain centralized cost tracking dashboards and alerting across business units, and recommend cost-efficient model routing and provider selection strategies.
• Participate in an on-call rotation for AI/ML systems, triage and resolve production incidents, and lead and contribute to post-incident reviews that drive lasting improvements.
• Develop production-grade services and tooling, primarily in Python, on Kubernetes, applying strong engineering practices (testing, code reviews, CI/CD, observability, security), and document standards, architecture diagrams, and operational playbooks for other teams.

What you'll need

• A curious and driven mindset, with a strong interest in how AI systems behave in production and how to make them more reliable and observable.
• 4+ years of applicable experience, including at least 3 years in commercial software engineering or production operations (SRE, DevOps, Platform, or ML Platform), operating services on Kubernetes and/or major cloud platforms.
• Experience operating production systems at scale, preferably distributed, microservices-based, or Kubernetes-based architectures.
• Hands-on experience with CI/CD tools and deployment automation (GitLab CI/CD, GitHub Actions, or similar), infrastructure-as-code tools (preferably Terraform), monitoring, logging, and alerting tools (Datadog, Prometheus, Grafana, ELK/EFK, or similar), and container management and the Docker image lifecycle.
• Experience operating and administering self-hosted open-source tools on Kubernetes, including installation, configuration, upgrades, security hardening, and backup and disaster recovery.
• Understanding of basic machine learning concepts and model lifecycles (training, evaluation, deployment, monitoring), and familiarity with common ML tooling.
• Strong computer science and programming fundamentals, with solid problem-solving skills and a proven ability to write high-quality software using Test-Driven Development (TDD), SOLID principles, clean code, and robust documentation.
• Ability to communicate effectively with data scientists, ML engineers, and product stakeholders in both business and technical contexts.
• Comfort participating in an on-call rotation and working in a trustful, respectful, and collaborative environment.

Nice to have

• Strong GCP experience, including hands-on work with GKE, BigQuery (data warehousing, SQL, schema management), Pub/Sub, Vertex AI (batch prediction, model deployment), Cloud SQL, GCS, and IAM.
• Familiarity with LLM ecosystem tooling (AI gateways, vector databases, RAG pipelines, embedding models, or LLM evaluation frameworks).
• Experience with Python-based data and ML stacks (NumPy, SciPy, Pandas, Scikit-Learn, PyTorch).
• Experience building or operating ML infrastructure (feature stores, orchestration platforms, model serving, annotation tools).

Details

• Location: United States, remote (Workplace Type: Remote). Porch Group does not consider applicants from Alaska, Delaware, Hawaii, Mississippi, Nebraska, Montana, New Hampshire, West Virginia, or the District of Columbia for Remote positions.
• Pay range: $135,000 to $189,000 annually, with eligibility for long-term incentive awards.
• Application window anticipated to close in 2 weeks (10 business days) from 11/2/2026, subject to change.