Senior Prompt Engineer - QA

Overview

We are seeking a Prompt Engineer to join a cross-functional Product-Oriented Delivery team and drive the creation, validation, and optimization of AI prompts and instruction sets powering proprietary product initiatives. You will focus on designing, developing, and refining effective prompts, alongside designing robust testing workflows, building validating procedures, and continuously evaluating the performance of large language models (LLMs) and related AI-driven systems.

You’ll collaborate closely with product managers, subject matter experts, and data scientists to ensure prompt quality, model accuracy, and reliable automation in alignment with both technical needs and business objectives. This role is ideal for someone who thrives on solving complex AI and data challenges, is passionate about automation, and is eager to build foundational QA processes for fast-moving product teams.

Key Responsibilities

• Design, develop, and refine AI prompts and instruction workflows to meet specific product requirements and optimize LLM performance.
• Design, execute, and refine automated and manual tests for AI prompt sets and instruction workflows in applied AI or NLP contexts.
• Build and maintain evaluation frameworks for LLM prompts.
• Develop and uphold test automation scripts using Python and related frameworks.
• Conduct data-driven analyses using Python libraries to assess prompt quality, model performance, and workflow accuracy.
• Perform API and web/mobile application testing when needed.
• Integrate automated tests into CI/CD pipelines, ensuring high quality at every deployment stage.
• Collaborate with cross-functional teams to prioritize tasks, document findings, and drive continuous improvement.
• Clearly communicate technical findings and recommendations to both technical and non-technical stakeholders.
• Ensure deliverables meet security, quality, and performance standards.

Required:

• Bachelor’s degree in Data Science, Computer Science, Statistics, Engineering, related field, or equivalent experience.
• Strong analytical skills and hands-on experience in Python.
• Programming proficiency in at least one test automation language (Typescript, JavaScript, Python, Java).
• Hands-on experience with web automation frameworks.
• Proficiency in API testing tools and visual regression testing.
• Excellent written communication skills for clear test case and bug report documentation.
• Strong problem-solving mindset and a drive to optimize and automate workflows.
• Enthusiasm for agile, cross-functional teamwork, and iterative product development.

• Fluent in English, with clear written and verbal communication.

Preferred:

• Demonstrated experience in prompt engineering, including designing and optimizing prompts for various use cases.
• Experience with LLM evaluation, prompt engineering workflows, and prompt evaluation tools.
• Familiarity with machine learning concepts, NLP workflows, and data pipeline tools.
• Familiarity with design tools (e.g., Figma) and basic security testing concepts, especially for cloud-based applications.