Analyst

About the Role

Innodata is collaborating with a leading international conglomerate, to contract subject matter experts (SMEs) for a complex prompt data annotation project. SMEs will create complex prompts and responses, implement domain-specific evaluation criteria, and use cases, and provide evaluations and judgement of model responses. SMEs are an integral part in this impactful engagement that directly contributes to the helpfulness, reasoning, and performance of a frontier LLM model.

You will have a chance to collaborate with project managers, prompt engineers, and data scientists, to solve problems together. You will also utilize AI-driven processes to meet high standards for quality and reliability. Your work will directly contribute to improving our client's data annotation and evaluation processes, enabling them to scale more efficiently.

Key Responsibilities

• Create complex prompts in your subject domain according to detailed instructions and guidance.
• Design and implement evaluation rubrics on complex prompts of your creation, and corresponding model responses
• Provide evaluations and judgements of model responses, based on your domain knowledge
• Commit to consistent throughput and improvements over the course of the project

Technical & Required Skills

• Strong domain knowledge in one of the following fields:
• STEM (i.e., Chemistry, Biology, Physics, Mathematics)
• Humanities (i.e., History, Literature, Art, Journalism, Philosophy)
• Business (i.e., Economics, Finance, Accounting, Management)
• Eccentric (i.e., Games, Chess, Culinary Art)
• A collaborative mindset with the ability to solve complex technical challenges and work independently as needed.
• Exceptional attention to detail and a commitment to delivering high-quality, reliable AI solutions.
• Passion and curiosity in LLM improvement and data annotations
• Appreciation for issues of Diversity, Equity, and Inclusion in AI.

Preferred Skills and Experience

• Experience with data annotation, especially prompt creation and/or evaluation rubric creation

Minimum Education Requirements

• Master's degree or higher in STEM, Humanities, Business or a related field.