Overview
Develop Harbor-format audit-and-patch benchmark tasks from real-world software vulnerabilities to evaluate whether AI coding models can identify, reproduce and correctly fix security issues.
What you'll do
- Source and investigate real-world vulnerabilities from CVEs, security advisories, bug reports, and open-source repositories.
- Identify and classify vulnerabilities using the most specific applicable CWE and map them to relevant OWASP Top 10 categories where appropriate.
- Reproduce security issues and develop reliable proof-of-vulnerability tests that demonstrate the underlying weakness.
- Build Harbor-format audit-and-patch tasks across multiple programming languages and software ecosystems.
- Write realistic task instructions that describe the security issue without revealing vulnerable code, exploit details, or the expected remediation.
- Create reproducible Docker-based environments with pinned dependencies, clear setup instructions, and reliable execution procedures.
- Design and harden automated verifiers to prevent gaming, test manipulation, hardcoded solutions, and overfitting.
- Review and score AI-generated patches and technical reasoning using structured security and engineering rubrics.
- Identify tasks that can be solved through superficial pattern matching or copying known upstream patches without understanding the vulnerability.
- Document vulnerability context, reproduction steps, expected behavior, patch requirements, test methodology, and review decisions.
What you'll need
- 5+ years of professional software engineering experience.
- Expert-level proficiency in at least one of the following languages: Python, JavaScript / TypeScript, Java, C / C++, C#, Go, Rust, Ruby, PHP, Kotlin, Swift.
- Strong understanding of software security principles and common vulnerability classes.
- Practical knowledge of CWE, CVE, and OWASP Top 10.
- Ability to independently reproduce vulnerabilities and develop proof-of-vulnerability tests.
- Hands-on experience with security testing and analysis tools, including sanitizers, fuzzers, and static-analysis tools such as CodeQL or Semgrep.
- Strong proficiency with Git, Linux, Bash, and Docker.
- Ability to create automated tests and reproducible technical evaluation environments.
- Excellent technical writing skills, with the ability to explain complex security issues and remediation requirements clearly.
- Ability to work independently in a fully remote, research-oriented environment.
Details
- Contractor assignment, 3 months.
- Expected to start next week.
- Commitment: 40 hours per week.
- Fully remote work arrangement.
Read the full description and apply on the company’s own careers page.