AI Benchmark Engineer | Native Language Specialist – Indonesian

Overview

Remote, freelance AI Benchmark Engineer (Native Language Specialist – Indonesian) to design, build, and validate rigorous Terminal-Bench tasks for multilingual LLM evaluation.

What you'll do

• Design and engineer benchmark tasks for evaluating coding agents.
• Create realistic native-language task environments using datasets and files.
• Identify failure points in the model's native-language output/prompting.
• Develop robust reference solutions and deterministic verifier scripts.
• Analyze execution logs and calibrate task difficulty using Terminal-Bench runs.
• Participate in a multi-layer human quality control process with automated checks.

What you'll need

• 5+ years of industry experience in software engineering.
• Native or near-native proficiency and deep knowledge of Indonesian grammar/register/phrasing.
• Strong proficiency in Python.
• Strong proficiency in standard shell scripting and data processing.
• Experience with Terminal/CLI-based development workflows.
• Working familiarity with coding agents.
• Ability to handle multilingual text processing pitfalls (e.g., Unicode normalization and locale conventions).

Details

• Location: Indonesia (Remote).
• Employment model: freelance (independent contractor, 1099).