QA Engineer AI Agent System

We're looking for a meticulous QA Engineer to lead testing efforts for our multi-agent AI system that automates crew member matching, onboarding, and job tracking. This is a cross-functional QA role where youll validate everything from AI-generated decisions to backend APIs, UI components, and third-party integrations (voice cloning, Twilio).

Youll create and run test plans for 8+ autonomous agents, including LLM-backed agents, and help ensure every part of the system behaves reliably and ethically before, during, and after deployment.

Key Responsibilities:

Manual & Exploratory Testing

• Understand agent workflows (Job Matcher, Onboarding Agent).
• Perform manual testing of workflows across:
• Web UI interfaces
• LLM responses and validation triggers
• Communications (email, SMS, voice)
• Backend API changes
• Identify issues in agent decision logic (false positives, incorrect matching, onboarding gaps, etc.).

Automated Testing

• Design and implement automated test scripts for:
• API endpoints
• Event/job creation flows
• Onboarding and contract handling
• SMS/Email/Voice triggers and responses
• Use tools like Postman, Selenium, Playwright, or PyTest for scripting.

Test Planning & Documentation

• Write and manage:
• Test cases and scenarios for each AI agent.
• Test Data and Synthetic data preparation (with support from ML team)
• Regression, functional, and smoke test plans.
• Edge-case tests for fallbacks, failed calls, no-shows, etc.
• Document known issues, test coverage, and agent-specific behavior.

QA Reporting

• Track bugs and raise detailed Jira tickets for:
• UI/UX inconsistencies
• API response mismatches
• Communication errors
• Model prediction issues (with support from ML team)
• Report test progress and bug severity regularly to the engineering team.

Collaboration & Feedback Loops

• Collaborate closely with:
• AI Engineers to validate LLM and ML behavior.
• Backend Devs for API consistency and error handling.
• Voice & Communication Engineers to validate cloned call quality and integrations.
• Help define agent success metrics from a QA lens (e.g., no incorrect crew matches, no duplicate invites, etc.).

Required Skills & Qualifications:

• 3-5+ years in QA Engineering or SDET roles
• Solid understanding of API and integration testing
• Experience with test automation tools:
• Postman / Swagger
• Selenium
• Python-based testing frameworks (PyTest preferred)
• Knowledge of version control (Git/Bitbucket) and issue tracking (Jira Issues)

Preferred (Bonus) Experience:

• Testing systems with AI/ML components or LLM-based applications
• Familiarity with voice API testing (e.g., Twilio Voice, ElevenLabs output validation)
• Worked on event-based or on-demand staffing platforms
• Understanding of CI/CD and QA automation pipelines

Tools You May Work With:

• Automation & API Tools: Postman, PyTest, Playwright, Selenium
• Bug Tracking: GitHub Projects, Jira
• Monitoring Feedback: Azure Monitor / Log Analytics
• Voice & Message Validation: Call transcripts, email/SMS message logs
• Other: Google Sheets (Test tracking), Teams (Internal team comms)