We’re hiring a Code Reviewer with deep Python expertise to review evaluations completedby data annotators assessing AI-generated Python code responses. Your role is to ensurethat annotators follow strict quality guidelines related to instruction-following, factualcorrectness, and code functionality.
Responsibilities
- Review and audit annotator evaluations of AI-generated Python code.
- Assess if the Python code follows the prompt instructions, is functionally correct,
and secure.
- Validate code snippets using proof-of-work methodology.
- Identify inaccuracies in annotator ratings or explanations.
- Provide constructive feedback to maintain high annotation standards.
- Work within Project Atlas guidelines for evaluation integrity and consistency.
Required Qualifications
- 5–7+ years of experience in Python development, QA, or code review.
- Strong knowledge of Python syntax, debugging, edge cases, and testing.
- Comfortable using code execution environments and testing tools.
- Excellent written communication and documentation skills.
- Experience working with structured QA or annotation workflows.
- English proficiency at B2, C1, C2, or Native level.
Preferred Qualifications
- Experience in AI training, LLM evaluation, or model alignment.
- Familiarity with annotation platforms.
- Exposure to RLHF (Reinforcement Learning from Human Feedback) pipelines.
Skills: python,reinforcement learning from human feedback,debugging,llm evaluation,testing,communication,qa,docker,model alignment,testing tools,ai-generated python code,annotation workflows,c2,code review,ai training,b2,code execution environments,rlhf