Glossary
QA and AI testing glossary
Definitions we can stand behind. Each term is short on purpose. Longer write-ups live on Services, AI Quality, and Resources. This is not two hundred thin pages.
Acceptance criteria
The conditions that must be true for a story to be considered done — written so a tester and a developer can disagree in public.
Accessibility testing
Checking whether people can use the product with keyboard, zoom, assistive tech, and perceivable content — against a named standard you choose.
AI red teaming
Adversarial probing of an AI product for jailbreaks, policy failures, and harmful completions.
API contract testing
Checking that a provider and a consumer still agree on shape, status, and meaning — not only that a status code was 200.
Bug report
One issue, enough evidence to fix: steps, environment, expected vs actual, and severity versus priority.
Chatbot testing
QA for conversational products: multi-turn flows, handoff, and the answers that sound fine and are wrong.
Dedicated QA team
Named testers who sit in your ceremonies and keep product knowledge across sprints.
Definition of done
The team’s written bar for “this increment is finished” — tests, review, and evidence included.
Escaped defect
A defect found after the release it should have been caught in — by a customer, support, or the next environment.
ETL testing
Checking extract-transform-load: counts, keys, and transforms from source to target.
Eval set
A fixed collection of inputs (and expected traits) you rerun against a named AI build.
Exploratory testing
Simultaneous learning, design, and execution — chartered time on the product, not a script you already knew the answer to.
Flaky test
An automated check that fails and passes on the same build without a product change.
Functional testing
Checking that the product does what it is supposed to do for a user — permissions, data, and awkward paths included.
Golden set
A small, trusted eval set used as a gate — the cases you refuse to ship if they regress.
Groundedness
Whether an answer stays supported by the retrieved or provided source — the opposite of an unsupported fluent claim.
Guardrail testing
Checking the blocks, filters, and policies that are supposed to stop unsafe or out-of-policy model behaviour.
Hallucination (LLM)
A fluent output that is unsupported or false relative to the source or the world the product claims to use.
IoT testing
QA on connected-device journeys: pairing, telemetry, apps, and update paths you can put in front of a tester.
Jailbreak (LLM)
A technique to make a model produce output the product said it would refuse.
LLM as a judge
Using a model to score another model’s output against a written rubric.
LLM evaluation
Structured scoring of how your product uses a language model: answers, tools, retrieval, and safety — on a named build.
Load testing
Applying expected concurrent use to see whether response time and errors stay inside what you claimed.
Localization testing
Checking that translated and locale-specific builds still work: layout, formats, and the journeys that break when the string is longer.
Managed testing
An outside team runs the QA program — coverage, cadence, and reporting — not only a pile of tickets.
Offshore QA
Testing delivered from a team in another country — for Nexus Eclipse, Islamabad — with planned overlap to the buyer’s hours.
Performance testing
Checking speed, capacity, and stability under conditions you specify — load, soak, or a spike — not a Lighthouse screenshot as the whole job.
Prompt injection
An input that tries to override the product’s instructions so the model ignores policy or leaks context.
QA as a service
Burst testing coverage around a named release or a short window, not a standing department.
QA audit
A diagnosis of what is untested, who owns quality, and where the next defect will come from — before you buy a team.
QA metrics
Numbers that describe testing work: coverage of agreed risk, escaped defects, cycle time of a fix — not a vanity dashboard.
QA outsourcing
An external testing team designs, executes, and reports work against an agreed scope instead of you hiring that function in-house.
RAG evaluation
Checking retrieval-augmented generation: did we fetch the right chunks, and did the answer use them?
Regression testing
Re-checking journeys that used to work, after a change, so old defects do not return.
Requirements traceability matrix
A map from a requirement or risk to the cases that cover it, and to the result on a named build.
Salesforce testing
QA on a Salesforce org: flows, roles, and integrations you name — not an implementation partnership.
Sanity testing
A narrow check that a specific change did what it claimed, not a tour of the whole product.
Security testing (QA)
QA-shaped checks for auth, roles, and obvious exposure — not a pentest certificate or a SOC 2 stamp.
Severity vs priority
Severity is how bad the defect is for a user or the system. Priority is the order you fix it.
Shift-left testing
Moving test design, risk, and some checks earlier — into stories, reviews, and CI — instead of only after “dev done.”
Smoke testing
A short set of checks that the build is installable and the obvious paths are not on fire — before deeper testing.
Test automation
Checks that a machine can rerun on a named build — useful when a human will maintain them.
Test case
A repeatable check: preconditions, steps, and an expected result someone else can run.
Test coverage
How much of the agreed risk or code was exercised — only useful with a denominator you can defend.
Test plan
The written scope for one release or cycle: what will be tested, on which build, and what will be skipped.
Test pyramid
A shape for automated checks: more fast, isolated tests than slow end-to-end UI journeys.
Test strategy
How testing is supposed to work across releases: ownership, environments, automation shape, and what “done” means.
Tool calling
When an AI agent chooses and invokes a function, API, or tool as part of answering or acting.
Usability testing
Watching people try to complete a task and logging where the product fights them — not a visual design review dressed as research.
User acceptance testing
Checks by the people who will live with the product that it is acceptable to release — not a substitute for QA finding defects earlier.
WCAG
Web Content Accessibility Guidelines — a published standard. A product can be tested against a named version. That is not a legal certificate by itself.
Related: FAQs, Resources,AI Quality.
Talk to our QA team
A term on a real product?
Definitions are free. Coverage on your build is an engagement.