First Object logo

Qa Test Engineer

First Object

5 - 8 years

Bengaluru

Posted:18 hours ago

Job Description

JOB TITLE:

DEPARTMENT:

REPORTS TO:

NUMBER OF DIRECT REPORTS:

Location:

Summary:

Senior Automation QA Engineer

What You'll Do

  • Design and implement test automation strategies for backend APIs, agent orchestration logic, and Angular-based agent UIs.
  • Build and maintain Pytest-based test suites for Python backend services and agent workflow validation.
  • Implement Playwright automation for Angular-based UI, covering interaction flows, approval queues, and streaming response handling.
  • Validate multi-step agent workflows by asserting on intermediate states, tool call outputs, and handoff results, not just final API responses.
  • Use Langfuse traces to validate agent reasoning chains, confidence scores, and execution paths.
  • Build and maintain LLM evaluation harnesses covering prompt regression testing, RAG output validation, and model version change detection.
  • Integrate test suites into Azure DevOps CI/CD pipelines as automated quality gates.
  • Verify audit log integrity, including immutable log structure, correlation ID chains, and SOX-compliant event emission.

Who You Are

Test Automation - Primary Gate

  • 5+ years of QA automation experience for API and web-based applications.
  • Pytest as the primary test framework for Python backend and agent logic validation.
  • Playwright for UI automation targeting Angular-based interfaces.
  • REST API testing using Postman, HTTPX, or equivalent tools for contract and schema validation.
  • Python for test scripting, strongly preferred given the stack.
  • SQL/NoSQL for test data setup, teardown, and database state validation.
  • Azure DevOps CI/CD pipeline integration for automated quality gates.

LLM / Agent Testing Awareness - Required

  • Understands that LLM outputs are non-deterministic and knows why standard assertion patterns must be adapted, even without prior hands-on production experience.
  • Familiarity with multi-step agent graph testing, including asserting on intermediate states, tool calls, and handoff outputs rather than just end results.
  • Understanding of what Langfuse traces contain and how they can be used for agent validation. Hands-on production experience is a plus but not required.
  • Awareness of model version regression risks and how to detect quality degradation when model versions change.

Nice to Have

  • Hands-on Langfuse experience covering trace validation, eval pipeline setup, and confidence score assertion.
  • Experience with LLM evaluation frameworks such as Ragas or equivalent for RAG output quality.
  • SOX/GDPR audit trail validation, including verifying immutable log integrity.
  • Azure Monitor or Application Insights for observability-assisted debugging.

What You'll Learn & Gain

  • A front-row seat to defining what quality means for AI agent systems, working on test problems that don't yet have established industry playbooks.
  • Hands-on experience building LLM evaluation harnesses and agent validation frameworks in a production financial platform context.
  • Deep familiarity with Langfuse, agent trace analysis, and confidence-based quality gates, skills that are increasingly valuable across the AI engineering industry.
  • Collaboration with a cross-functional squad of backend, frontend, and AI/ML engineers where QA is treated as a first-class engineering discipline.

If you are a QA engineer ready to move beyond deterministic test automation and take on the challenge of validating AI agent behavior in a high-compliance enterprise environment, we encourage you to apply.

Skills Required

Automation Artificial Intelligence Python Software testing Pytest Automation testing Generation Quality assurance

Learn the skills this job asks for

See your skill gap for QA / Test Engineer →

Free courses, interview questions and practice for each skill.

Work Mode

Hybrid

Job Type

Full Time

Mock Interview

Practice Video Interview with JobPe AI

Start Artificial Intelligence Interview
cta

Start Your Job Search Today

Browse through a variety of job opportunities tailored to your skills and preferences. Filter by location, experience, salary, and more to find your perfect fit.

coding practice

Enhance Your Python Skills

Practice Python coding challenges to boost your skills

Start Practicing Python Now
First Object logo
First Object

RecommendedJobs for You

Clover Infotech logo

Qa Test Engineer

Clover Infotech

Mumbai, Navi mumbai

First Object logo

Qa Test Engineer

First Object

Bengaluru

Indihire Consultants logo

Qa Test Engineer

Indihire Consultants

Bengaluru

Anmeda Systems logo

Qa Test Engineer

Anmeda Systems

Bengaluru

Bct Consulting logo

Qa Test Engineer

Bct Consulting

Chennai

Clover Infotech logo

Qa Test Engineer

Clover Infotech

Mumbai, Navi mumbai

First Object logo

Qa Test Engineer

First Object

Bengaluru

Indihire Consultants logo

Qa Test Engineer

Indihire Consultants

Bengaluru

Anmeda Systems logo

Qa Test Engineer

Anmeda Systems

Bengaluru