Section 027 · Chapter 4, Statistical Tests for AI Quality
F-Scores, Precision, Recall, and AI Quality
F-scores help Confidence Engineers reason about the tradeoff between catching the right things and avoiding false alarms.
precisionrecallF-scoreconfidence engineerf scores precision recall quality
What to do
- Use F2 when missing cases is worse than false alarms.
- Use F0.5 when false alarms are more costly than misses.
- Report the threshold, calibration, confusion matrix, slice behavior, and confidence intervals around the measured rates.
- Use them as one metric inside a broader quality story.
Evidence to preserve
- Report the threshold, calibration, confusion matrix, slice behavior, and confidence intervals around the measured rates.
- Preserve the inputs, versions, configurations, raw outcomes, and results for precision, recall, F-score, confidence engineer needed to reproduce work on F-Scores, Precision, Recall, and AI Quality.
- Report results for precision, recall, F-score, confidence engineer by relevant slice, separate blocker failures from averages, state uncertainty and blind spots, and connect the result to a release decision.
Expert note
Decide whether F1 is the right F-score. F1 weights precision and recall equally. F-beta scores let you weight recall more heavily than precision, or precision more heavily than recall.
Continue the conversation
Apply this to your context.
Save your product context once, then open a focused conversation that combines it with this concept.
Cite this page
Jason Arbon. "F-Scores, Precision, Recall, and AI Quality." Testing AI Knowledge Edition, section 27.
https://jarbon.ai/testing-ai/knowledge/ch027-f-scores-precision-recall-quality.html