AI Benchmark Engineer | Native Language Specialist – French (France)

Overview

AI Benchmark Engineer native-language specialist (French, France). You will design, build, and verify Terminal-Bench evaluation tasks to test multilingual performance of large language models.

What you'll do

• Engineer benchmark tasks that evaluate coding agents.
• Create realistic task environments using native-language datasets and files.
• Identify model failure points in your native language.
• Develop reference implementations and write deterministic verifier scripts.
• Analyze execution logs and calibrate task difficulty using Terminal-Bench runs.
• Participate in a 4-layer human quality control process with automated LLM-based checks.

What you'll need

• 5+ years of industry software engineering experience.
• Native or near-native fluency in French with deep grammar/register knowledge.
• Strong proficiency in Python.
• Standard shell scripting and data processing proficiency.
• Experience with Terminal/CLI-based development workflows.
• Familiarity with coding agents.
• Technical knowledge of multilingual text processing pitfalls including encoding/Unicode normalization.

Details

• Remote (France) and described as a remote, freelance opportunity.