AI Benchmark Engineer | Native Language Specialist – French (Canada)

Overview

AI Benchmark Engineer (French - Canada) to design, build, and validate a rigorous Terminal-Bench evaluation suite for multilingual coding and terminal workflows.

What you'll do

• Design and engineer benchmark tasks evaluating coding agents.
• Create realistic task environments using native-language datasets and files.
• Identify model failure points through native-language prompting and translation work.
• Develop robust reference solutions and write deterministic verifier scripts.
• Analyze execution logs to calibrate task difficulty using Terminal-Bench configurations.
• Support multi-layer human quality control with automated checks for benchmark integrity.

What you'll need

• 5+ years of industry experience in software engineering.
• Native or near-native fluency in the target language with deep grammar/register knowledge.
• Strong proficiency in Python, standard shell scripting, and data processing.
• Extensive experience with Terminal/CLI development workflows and familiarity with coding agents.
• Knowledge of multilingual text processing pitfalls including encoding/decoding and Unicode normalization.
• Experience with locale-dependent conventions, text I/O, toolchain interoperability, and safe string operations.

Details

• Location: Canada (Remote).
• Engagement: remote, freelance / independent contractor (1099).