AI Benchmark Engineer | Native Language Specialist – German (Switzerland)

Overview

Remote freelance AI Benchmark Engineer specializing in German (Switzerland), designing and validating Terminal-Bench multilingual evaluation tasks.

What you'll do

• Design and engineer benchmark tasks for Terminal-Bench coding agents.
• Create realistic task environments using native-language datasets and files.
• Identify model failure points by prompting and working within the native language.
• Develop reference implementations and write deterministic verifier scripts for benchmark judging.
• Analyze execution logs and calibrate task difficulty from Easy to Very Hard across model tiers.
• Participate in a 4-layer human quality control process with automated checks to ensure integrity.

What you'll need

• Have 1+ years of industry experience in software or prompt engineering.
• Have native or near-native fluency in German with deep grammar and phrasing knowledge.
• Maintain high English proficiency.
• Be proficient in Python, standard shell scripting, and data processing.
• Have extensive experience with Terminal/CLI-based development workflows and coding agents familiarity.
• Understand multilingual text processing pitfalls including encoding/Unicode normalization and locale-dependent conventions.

Details

• Location: Switzerland (Remote).
• Employment: freelance (independent contractor).