LATAM Software Engineers: Coding Tasks for AI Evaluation

Overview

Participate in a paid study evaluating coding tasks used to test AI agents. Review programming environments and evaluation harnesses, validate their logic, and provide feedback on their realism, accuracy, structure, complexity, and application.

What you'll do

• Review realistic programming tasks designed for AI evaluation.
• Assess the accuracy and structure of various evaluation harnesses.
• Verify the logic of evaluation harnesses.
• Walk through the environments and point out potential flaws or improvements while sharing your screen.
• Walk through your thought process while checking code validity.
• Provide actionable feedback on how to improve task complexity and realism.
• Discuss technical architectures and evaluation frameworks.

What you'll need

• Professional experience as a software engineer or developer.
• Strong technical background.
• Direct experience building, reviewing, or testing evaluation harnesses and programming tasks.
• Familiarity with building or verifying coding tasks and test environments.
• Comfort discussing technical architectures and evaluation frameworks.

Details

• Located in Brazil or Argentina.
• Paid study.
• Compensation is $40 per hour.
• The session involves code review, technical discussion, direct feedback on task structures, and screen sharing.