Expert: Software Engineer – AI Evaluation (Task-based Contract, Remote)

Overview

Contract role designing realistic, multi-step coding and technical tasks for frontier AI models and judging how well those models handle them.

What you'll do

• Design realistic, multi-step coding and technical tasks that test an AI model's ability to reason, write code and follow precise instructions.
• Build tasks around realistic workspace materials such as codebases, technical specs, architecture docs and datasets.
• Evaluate AI-generated code and technical content for correctness, efficiency, clarity and alignment with real-world engineering practice.
• Grade model outputs against rubrics and ground-truth answers, and write clear, structured feedback.
• Help improve how AI understands programming, system design, data workflows and engineering best practices.
• Work independently and asynchronously, meet deadlines, and collaborate with the Patronus team to keep quality high.
• Test and refine content to reflect real-world complexity.
• Review other contributors' work for accuracy, clarity, and practical relevance.
• Incorporate feedback promptly and flag gaps.

What you'll need

• 2+ years of professional experience in software engineering (internships excluded).
• Experience with system design, data pipelines, ML or LLM-based applications.
• Prior work in code review, technical writing, AI training or model evaluation.
• Strong coding skills in at least one major language, such as Python, Java, C++, JavaScript, TypeScript or Go.
• Experience building, reviewing and debugging production-grade code.
• Ability to follow detailed technical guidelines, work independently, respond to feedback, and meet deadlines.
• Excellent written communication and close attention to detail.
• A bachelor's degree in computer science, engineering or a related field, or equivalent experience.

Details

• Location: Remote (India).
• Type: Hourly, project/task-based contract (not a full-time role).
• Pay: $50–$100 per hour; some tasks have a flat rate.
• Commitment: Flexible, typically 15–20 hours per week; can be done alongside a full-time job.
• All tasks, project files and solutions must be human-authored; LLM-generated content is not permitted, and tasks must be original.