Expert Rater – AI Model Evaluation & Code Quality

Expert Rater – AI Model Evaluation & Code Quality

Email- nancy.singh@truelancer.com

Fill up this form- https://forms.gle/RELmAdij16e9jyFF8

We’re expanding our elite rater team and looking for highly skilled technical experts who love reviewing code, debugging tricky problems, and shaping the next generation of AI systems.

If you enjoy analyzing AI-generated content, evaluating technical accuracy, and working with cutting-edge tools — this role is calling your name.

About the Role

As an Expert Rater, you will evaluate AI model responses across programming, machine learning, and general technical reasoning. Your feedback will directly influence the accuracy, safety, and performance of AI systems used worldwide.

This role is perfect for software engineers, ML/AI practitioners, tech reviewers, or anyone who loves deep technical problem-solving.

What You’ll Do

Model Quality Assessment

• Evaluate AI-generated responses involving code, ML/AI concepts, and technical workflows
• Identify errors, inefficiencies, vulnerabilities, and deviations from best practices
• Ensure outputs meet strict quality guidelines

Code Annotation & Labeling

• Annotate code snippets, algorithms, and documentation
• Apply project-specific labeling frameworks accurately

Review & Feedback

• Provide structured, actionable feedback to improve AI behavior
• Spot correctness issues, edge cases, and quality gaps

Comparative Analysis

• Compare multiple AI outputs and rank them based on correctness, clarity, and efficiency

Data Validation

• Validate, clean, and correct datasets used in training and evaluation

Collaboration

• Work alongside data scientists and engineers
• Contribute to refining rating and annotation guidelines

Required Qualifications

Technical Background

• Strong foundation in software engineering, computer science, ML/AI, or related fields
• Excellent attention to detail and passion for data quality

Programming Expertise

Must-have:

• Python

Plus at least one of the following:

• JavaScript, Rust, Node.js, TypeScript, C, C++, Shell

Bonus languages:

• Go, Ruby, Swift, PHP, Kotlin, Rust (extra points), Shell

Web Technologies

• APIs, web scraping
• HTML/CSS/JavaScript
• Web app development (Flask, Node.js, React, etc.)

ML & AI Knowledge

• Machine Learning fundamentals
• Deep Learning frameworks (TensorFlow, PyTorch, JAX, Keras)
• NLP, CV, RL concepts
• Understanding of transformer architectures, CNNs, RNNs, LSTMs

Data Science & Engineering

• Pandas, NumPy, visualization libraries
• SQL/NoSQL databases

Algorithms & Math

• Strong understanding of algorithms (e.g., A*, MCTS, optimization, combinatorics)

Software Engineering Practices

• Version control (Git/GitHub)
• Clean code principles
• Debugging & problem-solving
• Ability to follow complex multi-step instructions

Soft Skills

• Excellent written communication
• Analytical mindset
• Proactive, detail-oriented, and willing to challenge assumptions

Preferred Qualifications

• Experience with LLMs or AI-driven code generation
• Familiarity with multiple programming paradigms
• Prior experience reviewing code (professional or academic)
• Background in annotation, QA, or technical evaluation