AI Benchmark Engineer | Native Language Specialist - German (Austria) - Remote

Overview

Remote freelance AI Benchmark Engineer (Native Language Specialist for German) to design, build, and validate a rigorous Terminal-Bench evaluation suite for multilingual coding agent tasks.

What you'll do

• Engineer and build Terminal-Bench tasks to evaluate coding agents.
• Create realistic task environments using native-language datasets and files.
• Identify model failure points by crafting prompts in your native language.
• Develop reference implementations and write deterministic verifier scripts.
• Analyze execution logs and calibrate benchmark difficulty (Easy to Very Hard) across model tiers.
• Participate in 4-layer human quality control and automated LLM checks to ensure benchmark integrity.

What you'll need

• 5+ years of industry experience in software engineering.
• Native or near-native fluency with deep knowledge of grammar, register, and phrasing.
• High English proficiency.
• Strong proficiency in Python, standard shell scripting, and data processing.
• Extensive experience with Terminal/CLI-based development workflows and familiarity with coding agents.
• Deep understanding of multilingual text processing pitfalls, including encoding/decoding and Unicode normalization.

Details

• Job listed as remote and freelance/independent contractor.