AI Benchmark Engineer | Native Language Specialist – Portuguese (Brazil)

Overview

Remote AI Benchmark Engineer and Native Language Specialist (Portuguese - Brazil) to design, build, and validate multilingual Terminal-Bench evaluation tasks.

What you'll do

• Engineer benchmark tasks for evaluating coding agents.
• Create realistic task environments using native-language datasets and files.
• Identify model failure points using prompts written in the native language.
• Develop reference implementations and write deterministic verifier scripts.
• Analyze execution logs and calibrate task difficulty using Terminal-Bench configurations.
• Participate in a multi-layer human quality control process with automated checks.

What you'll need

• 1+ years of industry experience in software or prompt engineering.
• Native or near-native fluency in Portuguese with strong English proficiency.
• Strong Python skills, standard shell scripting, and data processing.
• Experience with Terminal/CLI-based development workflows and coding agents.
• Expertise in multilingual text processing pitfalls (Unicode normalization, locale conventions, text I/O, safe string operations).
• Ability to support robust verifier logic using rubric-based judging only when strictly necessary.

Details

• Location: Brazil (Remote).
• Employment: remote freelance (independent contractor/1099 mentioned).