What are Precision, Recall, and F1 Score?
Precision, Recall, and F1 Score – Machine Learning Classification Metrics (Machine Learning): In-depth Guide
Definition of Precision, Recall, and F1 Score
Precision, recall, and F1 score are fundamental performance metrics for evaluating classification and object detection models. They are computed from a confusion matrix, a table that compares predicted labels against ground truth. These three metrics tell different stories about model performance, and the choice among them depends on the cost of different types of errors in your application.
Precision answers the question: “When the model flags a defect, how often is it correct?” Mathematically, precision = TP / (TP + FP), where TP is the count of true positive detections (correct detections) and FP is the count of false positive detections (incorrect detections). Precision measures the fraction of all positive predictions that are actually correct. A precision of 0.95 means that 95 percent of the defects the model detects are genuine.
Recall (also called sensitivity) answers: “Of all the actual defects present, how many does the model find?” The formula is recall = TP / (TP + FN), where FN is the count of false negative detections (defects the model missed). Recall measures the fraction of all actual defects that the model successfully detects. A recall of 0.90 means the model catches 90 percent of the real defects in the scene.
F1 Score is the harmonic mean of precision and recall: F1 = 2 * (Precision * Recall) / (Precision + Recall). This single number ranges from 0 to 1, with 1 being perfect detection. F1 provides a balanced metric when both false positives and false negatives are costly. It penalizes extreme imbalance between precision and recall, so a model cannot achieve a high F1 by sacrificing one metric in favor of the other.
The Confusion Matrix and Derived Metrics
All three metrics derive from a simple two-by-two table called the confusion matrix:
| Prediction | Actually Defect | Actually OK |
|---|---|---|
| Model says “Defect” | TP (True Positive) | FP (False Positive) |
| Model says “OK” | FN (False Negative) | TN (True Negative) |
From this matrix, precision and recall follow directly:
| Metric | Formula | What it means | Cost of being wrong |
|---|---|---|---|
| Precision | TP / (TP + FP) | Fraction of detections that are correct | False alarms waste inspector time |
| Recall | TP / (TP + FN) | Fraction of actual defects that are found | Missed defects endanger safety |
| F1 Score | 2 * P * R / (P + R) | Balanced harmonic mean of both | Penalizes extreme trade-offs |
A model with high precision but low recall catches only obvious defects and rarely raises false alarms. A model with high recall but low precision catches nearly every defect but flags many non-defects, drowning inspectors in manual verification work. The ideal model achieves both, but in practice the threshold (the decision boundary that separates positive from negative predictions) must be tuned to balance these competing demands.
The Precision-Recall Trade-off in Safety-Critical Inspection
In airport pavement and lighting inspection, the choice between precision and recall has real safety consequences. Consider two scenarios:
Scenario 1: Runway Foreign Object Debris (FOD) Detection
A FOD on an active runway is a critical hazard. An aircraft striking debris can suffer catastrophic damage. In this context, a missed defect (false negative) poses severe risk, so recall is paramount. An inspector might tolerate false alarms, spending extra time verifying that certain detected objects are indeed foreign material, because the cost of missing actual FOD is unacceptable. A recall of 0.98 is preferable to a recall of 0.80, even if precision drops from 0.95 to 0.85 in the process. The marginal false positives are a small operational burden compared to a missed safety hazard.
Scenario 2: Routine Pavement Distress Logging
For routine inspection of pavement surface distress (cracking, rutting, spalling), the failure mode changes. False positives mean the inspection report contains entries that maintenance crews must investigate and ultimately dismiss as non-issues. A system that flags 1,000 surface anomalies when only 500 are real defects forces expensive manual triage. Here, precision becomes more important relative to recall. A precision of 0.90 (90 percent of flagged distress is genuine) is more valuable than a recall of 0.99 with precision of 0.50.
This trade-off is not symmetrical across domains within the same airport. Lighting inspection (PAPI calibration, approach light status, intensity uniformity) may demand different precision-recall operating points than pavement surface scanning or obstacle clearance assessment.
Applications in Pavement and Infrastructure Inspection
In practice, computer vision models for airport inspection output detection confidence scores, not binary predictions. A model might flag a potential crack with confidence 0.72, another with 0.55, and another with 0.92. The decision threshold (e.g., “flag detections with confidence above 0.7”) directly controls the precision-recall trade-off.
Lowering the threshold from 0.7 to 0.5 catches more defects (higher recall) because marginal detections are now included. However, it also flags more false positives (lower precision) because borderline non-defects now cross the threshold. Raising the threshold to 0.9 does the opposite: fewer detections overall, but those that pass are more likely to be correct.
When evaluating a new defect detection model, practitioners often compute precision-recall across all possible thresholds and visualize the result as a precision-recall curve. The area under this curve (AUC-PR) is a summary metric. A model that achieves both high precision and high recall at most thresholds scores higher than one forced to choose between them.
When to Prioritize Precision vs. Recall
Operational context determines the optimal balance:
Prioritize Recall (High Sensitivity) when the cost of a missed defect is severe: FOD detection, critical structural damage, or lighting failures that affect approach navigation. An automated system with 0.95 recall and 0.70 precision is acceptable because missed defects are unacceptable.
Prioritize Precision (Low False Positive Rate) when false alarms are expensive: routine distress monitoring that requires manual verification, or when the inspection system feeds into a triage queue where false positives consume scarce expert time.
Balance with F1 (Medium Precision and Recall) when both false positives and false negatives carry significant cost, and you have no strong reason to weight one over the other. F1 score ensures the model does not excel at one metric while collapsing on the other.
Many airport inspection workflows operate in a two-stage process: an automated AI system with high recall (catching nearly all candidates) feeds a human review step that filters out false positives. The overall system achieves the safety benefits of high recall without the operational burden of excessive false alarms. The AI’s precision can be moderate. The human expert provides the final check.
Comparison and Summary
The following table contrasts how these metrics respond to model behavior:
| Scenario | Precision | Recall | F1 | Best Use Case |
|---|---|---|---|---|
| Detects everything, many false alarms | Low | High | Medium | High-risk scenarios where missing defects is unacceptable |
| Detects only certain defects, few false alarms | High | Low | Low | Low-urgency triage where false alarms waste time |
| Balanced, moderate on both | Medium | Medium | High | General-purpose assessment with no extreme cost asymmetry |
Conclusion
Precision, recall, and F1 score are not absolute measures of model quality. They are tools for understanding model behavior and making informed decisions about deployment. In airport lighting and pavement inspection, precision and recall carry different real-world costs. A high-recall, moderate-precision system may be the right choice for safety-critical defect detection, especially in the first stage of a two-stage inspection workflow. Conversely, a high-precision system may suit routine reporting where human review is abundant. The F1 score provides a balanced summary when no single cost dominates. Understanding these metrics and the confusion matrix they rest upon is essential for building and evaluating AI systems that enhance airport safety and operational efficiency.
Frequently Asked Questions
- What is the difference between precision and recall?
- Precision measures the fraction of detections that are truly correct (TP/(TP+FP)), reflecting false alarm rate. Recall measures the fraction of all actual defects that are found (TP/(TP+FN)), reflecting the miss rate. A high-precision model raises few false alarms but may miss defects, while a high-recall model catches most defects but produces more false positives.
- When should I prioritize recall over precision in defect detection?
- In safety-critical applications such as runway foreign object debris (FOD) detection or critical pavement distress that could endanger aircraft, recall is typically more important. Missing a defect (false negative) has severe safety consequences, so detecting most defects even with some false alarms is preferable.
- Why not just maximize recall to catch every possible defect?
- Maximizing recall creates excessive false positives, which in an automated inspection workflow forces manual review of numerous non-issues, consuming time and resources. This diminishes operational efficiency and can create alert fatigue, causing inspectors to miss real problems among the noise.
- What does F1 score measure?
- The F1 score is the harmonic mean of precision and recall, calculated as 2 * (Precision * Recall) / (Precision + Recall). It provides a single metric that balances both precision and recall, useful when you need both false positives and false negatives to be low.
- How do precision, recall, and F1 relate to the confusion matrix?
- All three metrics derive from the four cells of a confusion matrix: true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN). Precision uses TP and FP. Recall uses TP and FN. F1 is their combined harmonic mean.
- Can a model have high precision and high recall simultaneously?
- Ideally yes, but in practice there is often a trade-off. A model with a stricter decision threshold achieves high precision but lower recall (fewer false alarms but more misses), while a lower threshold increases recall but decreases precision (catches more defects but more false alarms). The best operating point depends on the application's tolerance for these errors.