Software Engineer in Test

Observance Solutions

0 years

0 Lacs

lucknow uttar pradesh india

Posted:2 days ago| Platform:

Apply

Skills Required

software test ai api consistency reasoning logic compliance automation design evaluation metrics latency graphs data analyze ml engineering testing simulation security

Work Mode

On-site

Job Type

Full Time

Job Description

The mandatory QA skill requirements are as follows:

#urgenthiring #immediatejoiners

Functional & Behavioral Testing:

Test AI agents across multi-step workflows, goal completion, and decision-making paths
Validate agent behavior under normal, edge, and failure scenarios
Verify correct tool calling, API usage, and response handling
Ensure consistency across retries, sessions, and memory states

Reasoning & Safety Validation:

Evaluate agent reasoning quality (logic, coherence, hallucination risk)
Test compliance with safety, policy, and ethical(HIPAA) constraints.
Identify prompt injection, jailbreaks, or unintended autonomous actions

Automation & Evaluation:

Design and maintain automated test suites for agent workflows.
Create evaluation metrics for accuracy, latency, success rate, and failure recovery
Use logs, traces, and conversation graphs to debug agent behavior

Data & Feedback Loops;

Curate and maintain test datasets, test cases and scenarios
Label and analyze failures to improve prompts, policies, and agent design
Collaborate with ML, product, and engineering teams to close gaps.
Experience testing AI/ML or LLM-based systems
Understanding of prompting and prompt chaining,tool/function calling,state, memory, and context management

Good to Have skills:

Experience with LLM frameworks (LangChain, AutoGen, CrewAI, Semantic Kernel, etc.)
Familiarity with evaluation tools (LLM evals, red teaming, simulation testing)
Knowledge of observability tools (traces, logs, replay tools)
Exposure to security or safety testing

hr@observancegroup.com

Mock Interview

Practice Video Interview with JobPe AI

Start Job-Specific Interview

Start Your Job Search Today

Browse through a variety of job opportunities tailored to your skills and preferences. Filter by location, experience, salary, and more to find your perfect fit.