QA Consultant

As an AI QA Consultant at Atrium, you will play a pivotal role in ensuring the quality, functionality, and performance of our AI solutions. Your responsibilities will revolve around testing AI features on the Salesforce platform, focusing on validating AI-powered features, agentic workflows, and integrations with Large Language Models (LLMs). This unique position combines traditional Salesforce QA with modern AI testing principles, necessitating a deep understanding of Salesforce best practices, attention to detail, and technical proficiency in testing complex, non-deterministic systems. Your collaboration with AI developers, Salesforce administrators, and business stakeholders will be crucial in delivering robust and reliable AI-driven user experiences. Your key responsibilities will include conducting thorough manual and exploratory testing of AI features on the Salesforce platform. You will be responsible for identifying, documenting, and tracking defects in Jira, distinguishing between standard configuration bugs and nuanced, probabilistic issues inherent in AI model outputs. Ownership of data quality throughout the AI fine-tuning lifecycle will be a significant aspect of your role, ensuring the validation and preparation of high-quality, unbiased datasets essential for building reliable enterprise AI. Additionally, you will be tasked with developing detailed test plans, test cases, and test scripts tailored to evaluate the accuracy, relevance, safety, and performance of LLM responses and AI-driven logic. Collaboration with product owners and AI developers to understand requirements and design comprehensive test strategies for Salesforce solutions incorporating AI components will be essential. You will create scenarios to test for vulnerabilities such as prompt injection, model hallucinations, data grounding issues, and bias in AI outputs. Evaluating the performance, latency, and resource consumption of AI features to meet user experience benchmarks will also be part of your responsibilities. Furthermore, you will facilitate User Acceptance Testing (UAT) for AI-powered features, guiding business users on evaluating and providing feedback on intelligent systems. Staying abreast of the latest advancements in LLMs, prompt engineering techniques, and AI testing methodologies will be crucial, as you champion the adoption of best practices. To excel in this role, you should have a strong foundation with 2-3 years of dedicated QA experience on the Salesforce platform. An understanding of modern AI/NLP concepts, including Retrieval-Augmented Generation (RAG), model fine-tuning, and creating feedback loops for reinforcement learning is essential. Proficiency in Python for writing test scripts, data manipulation, and interacting with AI APIs is required. Hands-on experience with LLM APIs, frameworks like LangChain/LlamaIndex, and various Vector Databases will be beneficial. Knowledge of SQL, NoSQL, and vector databases is also important. Excellent analytical and communication skills, along with experience in thriving in an Agile/Scrum environment, will be key to success in this role. At Atrium, we value diversity in our workforce to align with our growth ambitions and foster inclusion across the business. We are an equal opportunity employer, and all qualified applicants will receive consideration for employment.,