Overview
Lead a team of trainers developing high-quality training and evaluation tasks for frontier AI coding models, transforming real-world open-source vulnerabilities into Harbor-format audit-and-patch tasks across multiple programming languages.
What you'll do
- Plan task deliveries by batch, assign work across pod members, and track daily throughput against phase-specific targets.
- Monitor pod performance, identify bottlenecks, resolve delivery issues, and ensure tasks move through production stages on schedule.
- Lead weekly client reporting, communicate delivery status, and proactively escalate risks involving tooling issues, rate limits, safety blocks, and resource constraints.
- Collaborate with the client to clarify requirements and align on task scope, volume, complexity, and difficulty changes.
- Conduct final reviews of completed tasks to ensure they meet required quality, security, reproducibility, and evaluation standards.
- Train and calibrate Pod Leads and trainers on CWE classification, verifier robustness, task difficulty, technical requirements, and acceptance criteria.
- Use throughput and quality data to forecast staffing needs across project phases.
- Initiate hiring and onboarding waves approximately three weeks ahead of projected capacity requirements.
- Step in to research vulnerabilities, reproduce security issues, develop proof-of-vulnerability tests, and create or review benchmark tasks during periods of high workload.
- Maintain clear documentation of delivery plans, risks, quality findings, throughput metrics, staffing forecasts, and project decisions.
What you'll need
- 6+ years of professional software engineering experience.
- Expert-level proficiency in at least one of: Python, JavaScript / TypeScript, Java, C / C++, C#, Go, Rust, Ruby, PHP, Kotlin, Swift.
- Strong understanding of software security concepts, including CWE, CVE, and OWASP Top 10.
- Ability to reproduce software vulnerabilities and develop proof-of-vulnerability tests.
- Hands-on experience with security tooling, including sanitizers, fuzzers, and static-analysis tools such as CodeQL or Semgrep.
- Strong proficiency with Git, Linux, Bash, and Docker.
- Strong technical judgment when evaluating verifier robustness, security patches, and AI-generated code.
- Experience reviewing technical work and maintaining consistent quality standards across multiple contributors.
- Some experience mentoring engineers, trainers, or technical contributors.
- Excellent technical writing, documentation, and communication skills.
- Ability to work independently in a fully remote, research-oriented environment.
- Proven track record working on this project or equivalent software security dataset, benchmark development, or security evaluation work.
Details
- Contractor assignment, 3 months, 40 hours per week.
- Fully remote.
- Minimum 4 hours of overlap with PST.
- Expected to start next week.
Read the full description and apply on the company’s own careers page.