AI Benchmark Engineer | Native Language Specialist – Dutch

Overview

Design, build and validate Terminal-Bench evaluation tasks that test large language models on multilingual software challenges, using native-language assets and environments.

What you'll do

• Design and build realistic task environments using datasets and files kept in your native language.
• Find failure points where AI does not work, in your native language, through prompting and translation.
• Support the development of reference implementations and write highly reliable, deterministic verifier scripts (rubric-based judging only when strictly necessary).
• Analyze execution logs and calibrate task difficulty (Easy to Very Hard) using standard Terminal-Bench run configurations against model tiers (Haiku, Sonnet, Opus).
• Participate in the 4-layer human quality control process (creation, human review, calibration review, audit) alongside automated LLM-based checks.

What you'll need

• 5+ years of industry experience in software engineering.
• Proven track record at leading technology companies and/or graduation from top-tier engineering universities.
• Native or near-native fluency in Dutch, with a deep understanding of its grammar, register and phrasing rules.
• High English proficiency.
• Strong proficiency in Python, standard shell scripting and data processing.
• Extensive experience with Terminal/CLI-based development workflows and working familiarity with coding agents.
• Deep technical understanding of multilingual text processing pitfalls, including encoding/decoding robustness and Unicode normalization.
• Knowledge of locale-dependent conventions (collation, casing, non-Gregorian dates).
• Understanding of text I/O, toolchain interoperability and safe string operations.

Nice to have

• For specific languages: familiarity with bidirectional/RTL handling, font fallbacks, and rendering/typography in UI or artifacts.

Details

• Remote, freelance (independent contractor) engagement in the Netherlands.
• Work availability fluctuates with project demand; most tasks require a minimum of 2 hours per day or 15-20 hours per week.
• Not a full-time job or primary income source; no benefits such as health insurance, paid time off or retirement contributions as a 1099 contractor.
• Geographic restrictions may apply due to international embargo or sanctions.
• Application requires an English CV, a GenAI assessment, and onboarding/profile setup.