QA Metrics

QA metrics are numbers that change what you test next, or whether you ship. They are not a wall of charts that stay green while customers find the bug.

This page is for product QA. If the product is model-backed, also read LLM evaluation metrics. Those are a different set.

Metrics that usually earn their keep

Escaped defects

Bugs found in production (or by a customer) that a named test should have caught — or that no test covered. Pair with a short “why”: no case, wrong environment, skipped risk. This is how you improve the QA process, not how you blame a person.

Coverage you can explain

Not “87% of lines.” Requirement or journey coverage against an RTM, dated to a build. If you cannot name the journeys, you do not have coverage. You have a percentage.

Open defects by severity

A count is useless without severity vs priority. “47 bugs” includes typos and a broken checkout. Split them.

Automation flake rate

Failed runs that pass on retry with no product change. A high flake rate means the CI signal is theatre. Fix or delete those tests. See the test pyramid.

Cycle evidence, not “velocity”

Did this release leave a plan, a result, and a residual-risk note? A yes/no is a metric. Story points for QA usually are not.

Metrics that usually do not

• Test-case count as a KPI (it rewards junk cases)
• “Pass rate” when the suite is the wrong suite
• Hours logged as quality
• A single “quality score” with invented weights
• Benchmarks from a vendor case study we did not publish — we do not invent those here either

How to report without lying

Name the build, the date, the set, and what the number cannot see. A dashboard that hides skipped risk is how teams ship with confidence they did not earn.

A QA audit is the right start when you have metrics and still cannot say what a release covered.

FAQs

What is the one QA metric to start with?

Escaped defects plus a one-line cause. Then journey coverage on the next release. Do not start with a 20-chart pack.

Are more test cases better?

No. See test cases. Count the journeys that matter, not the rows.

How do we start?

Book a QA consultation. Bring last month’s production bugs and whatever dashboard you already have. We will say which numbers to keep.