When Automated Decisions Spoil Their Own Data

Key Takeaways

A lending model approves or declines applications throughout the day and learns from repayment outcomes. The problem is that it receives those outcomes only for approved applicants. Someone who has declined never takes the loan, so the bank never learns whether that person would have repaid. Each training cycle is therefore built from a population partly selected by the previous version of the model.

In lending, this problem is usually addressed through reject inference by estimating how declined applicants might have performed and incorporating those estimates into model development. It can reduce the blind spot, but it cannot replace an observed outcome. The system is still making assumptions about the people it chose not to serve.

Over time, the model becomes well-informed about the applicants it has approved and remains poorly informed about everyone else. That weakness can be difficult to see because the standard performance measures are calculated on approved loans. Accuracy, calibration, and default rates may all improve even as the system’s understanding of the broader applicant population worsens.

Consider a lender that raises its approval threshold. Fewer borderline applicants receive loans, defaults fall, and the approved portfolio performs better. The dashboard suggests that risk selection has improved. But the model may not have become any better at distinguishing good borrowers from bad ones. Management has simply removed more difficult cases from the population used to judge it.

The policy also changes the population the model encounters. A tighter lending rule may discourage some applicants from applying again. Stricter fraud controls may cause attackers to change their behavior. A recommendation system may alter what customers browse by repeatedly showing them the same categories. The model was trained on the population created by yesterday’s rules and is now operating on one shaped by today’s.

This creates a feedback loop. A fraud model blocks a segment, receives little evidence of legitimate transactions within it, and becomes more confident that the segment is risky. A pricing system stops presenting an offer to certain customers and then concludes that those customers have little interest in it. A next-best-action model repeatedly chooses the same action because alternatives no longer provide sufficient data to challenge that choice.

Most dashboards track outcomes that remain visible, such as repayment among approved borrowers, fraud among completed transactions, or conversion among displayed offers. They rarely show the outcomes that the policy prevented. The system can therefore look stable while the missing portion of the data grows.

The first control is complete decision logging. Every approval, decline, block, price, and recommendation should be stored with the model version, rule set, input data, score, threshold, and final action. This makes it possible to reconstruct which policy produced a training sample and determine whether a performance change came from a better model or a different population.

The second is a controlled way to keep learning near the decision boundary. Depending on the regulatory and operational setting, this could involve policy-approved testing, randomized offers, manual reviews, challenger models, delayed interventions, or other methods for collecting outcomes that the main policy would normally hide. The third is population monitoring by tracking approval rates, score distributions, applicant mix, channel mix, and movement between accepted and rejected groups, alongside the usual accuracy measures.

An automated decision system can weaken without generating an obvious failure. Its blind spots sit outside the evaluated population, its metrics improve as difficult cases disappear, and each retraining cycle gives greater weight to choices made by the previous one. The warning is often not a drop in accuracy but a steady change in who the system allows itself to observe.

The European edition of the Gurobi Decision Intelligence Summit returns to Prague, bringing together decision scientists, optimization practitioners, data scientists, and business leaders to explore how mathematical optimization and decision intelligence are being applied in complex real-world operations. The program covers customer use cases, technical training, expert panels, and one-on-one sessions with Gurobi specialists on specific models, constraints, and performance challenges. Details →

DGIQ and the AI Governance Conference will bring data and AI governance practitioners to Providence, Rhode Island, from November 16-19. The four-day program will cover data quality, metadata, master data management, AI ethics, algorithmic accountability, privacy, regulatory risk, semantic layers, and organizational governance. Sessions will include practitioner-led case studies, panels, workshops, tutorials, and seminars, with on-site CDMP and ADGP certification exams also available. Details →

BCG examines how AI agents can move supply-chain management beyond isolated copilots by coordinating decisions across operations, finance, and commercial teams. While 44% of companies already deploy AI in supply chains, most remain limited to narrow use cases. One consumer-goods deployment reduced administrative costs by 40% to 60%, while BCG estimates broader adoption could cut working capital by up to 30% and raise EBITDA by two to four percentage points. The report argues that these gains require connected data, auditable decisions, and end-to-end workflow redesign. Read →

Benjamin Baer, Founding Member of the DecideWise community, examines why the speed of AI-assisted software development does not automatically translate into business value. He points to an acquaintance who built a custom CRM with Claude in two days as an example of how vibe coding can give smaller businesses software tailored to their needs without a traditional development cycle. But the same approach carries greater risk in sectors such as finance, energy, insurance, manufacturing, and retail, where unreliable or poorly integrated systems can disrupt core operations. Baer argues that the real opportunity lies not in generating software faster, but in connecting AI to existing data, workflows, controls, and decision processes in a way that produces measurable returns. With a widely cited MIT study finding that 95% of enterprise generative AI pilots delivered no measurable financial impact, the advantage may go to smaller companies that use these tools to solve narrow industry problems more effectively than established vendors. Read More →

Dr. Lorien Pratt, co-founder of Quantellia, shares what decision intelligence means and how it differs from adjacent disciplines. The conversation covers the gap between data and decision-making, how decision engineering evolved into decision intelligence, the integration of operations research with DI methods, pricing optimization as a window into diffuse objectives and cross-silo complexity, the dangers of over-engineering models at the expense of stakeholder alignment, and why a shared blueprint for decisions matters more than a shared technology stack.