Why 99% Accuracy Can Fail a Foundry AI System
Short answer: A foundry AI model can report 99% accuracy and still miss every rare defect. If only one casting in one hundred is defective, a model that predicts “good” for every cycle is 99% accurate—but its defect recall is zero. This is a modeled arithmetic example, not a customer result. For rare casting defects, ask which defects the model catches, what it misses, and what each error costs.
Why overall accuracy creates false confidence
Casting quality data are usually imbalanced: acceptable parts dominate, while individual defect modes are scarce. A single accuracy figure lets the majority “good” class overwhelm the failures that matter.
Consider a modeled set of 10,000 production cycles with 100 defective castings. A model that labels every cycle good produces 9,900 correct predictions and 99% accuracy, yet detects none of the 100 defects. The metric is numerically correct and operationally useless.
Casting research treats class imbalance as a central constraint. Studies on casting surface defect detection and imbalanced casting-defect recognition therefore report precision, recall and class-level measures alongside accuracy.
Translate the confusion matrix into foundry decisions
True positive: a defective casting is correctly held for inspection or rejection.
False negative: a defective casting is released as good—the escape risk.
False positive: a good casting is held or rejected, creating inspection, delay or scrap.
True negative: a good casting is correctly released.
The threshold moves false negatives and false positives in opposite directions. There is no universal setting. A safety-critical leak path, a cosmetic mark and a machining-allowance indication do not have the same consequence.
The minimum scorecard
Recall by defect mode
Recall asks how many actual defects the model catches. Report it separately for porosity, cold shut, misrun, crack, oxide inclusion or every label the plant can measure reliably.
Precision and operator workload
Precision asks how many alarms are real. Poor precision causes alarm fatigue and unnecessary holds. Research on continuous-casting crack prediction explains why precision–recall analysis is more informative than a ROC curve under high imbalance and why false-positive and false-negative costs should guide the threshold.
Precision–recall curves
One threshold hides the trade-off. A precision–recall curve shows how alarm quality changes as sensitivity rises. Average precision can summarize the curve, but the operating point still requires an engineering decision.
Rates in production units
Translate metrics into false alarms per 1,000 cycles, missed defects per inspection batch, or held castings per shift. These are modeled projections until verified at live production prevalence. Do not extrapolate directly from a balanced laboratory sample.
Calibration and uncertainty
A score of 0.8 is not automatically an 80% probability unless calibration has been tested. With weak confidence, the system should be able to abstain and refer the case to an operator or extra inspection.
A six-step validation gate
Audit the label system. Confirm repeatability and defect definitions first; use the measurement-system gate for casting labels.
Preserve production prevalence. Rebalancing may help training, but the final test set must represent the line.
Split by production boundary. Hold out later time periods, tools, cavities, alloys or product families—not neighboring rows. This reduces data leakage.
Report each defect separately. Include the confusion matrix, recall, precision and the number of positive test examples.
Choose thresholds from consequence. Document the risk or cost of an escape, false alarm and abstention.
Run a shadow period. Let the model advise without controlling disposition. Compare alerts with verified outcomes before operational use.
This follows the production-oriented principle in the NIST AI Risk Management Framework: evaluate in context, document limitations and involve domain experts.
Evidence boundaries
Image-based surface inspection and process-data prediction are not interchangeable. Results on one alloy, machine, die or lighting setup do not prove transfer to another. Augmentation cannot manufacture missing process physics or repair unreliable labels.
A 2026 study on generative anomaly detection for casting surfaces evaluates precision, recall and F1 because missed defects and false alarms both matter. That is the right mindset: use several decision-relevant measures and state the limits of the test.
Where Castella fits
Castella can structure evaluation around production decisions rather than a headline accuracy score: class prevalence, per-defect confusion matrices, threshold scenarios, time/tool/cavity holdouts, uncertainty and traceability can be reviewed together. The purpose is decision support—not automatic authority. Engineers remain responsible for the control plan and release decision.
After this gate, the next improvement question can be tested with design of experiments and parameter interactions or connected to HPDC and LPDC process optimization.
Frequently asked questions
Is accuracy useless in a foundry?
No. It can support evaluation when classes and error costs are similar. For rare defects, it is insufficient alone.
Should recall always be maximized?
Not blindly. Very high recall may create an impractical false-alarm load. Safety, quality, inspection capacity and cost determine the operating point.
Does oversampling solve class imbalance?
It may help learning, but it does not change real prevalence. Validate on a production-like distribution and untouched holdout.
When should the threshold be reviewed?
After material, die, machine, sensor or inspection changes, and whenever prevalence or calibration drifts.
Comments