AI, Data Provenance, and Financial Analysis: Why Credit Unions Cannot Afford to Trust the Black Box
Guest Editorial by David Trainer, CEO of New Constructs
A new wave of AI-powered tools promises faster financial research and automated portfolio analysis, but recent evidence suggests many popular AI systems are failing in accuracy.
As financial professionals increasingly rely on AI-generated insights, questions are mounting around data quality, transparency, accountability, and regulatory compliance.
The Dangers of AI-Generated Financial Insights
The biggest danger in relying on AI-generated financial insights is that a confident-sounding recommendation and a reliable one are not the same. Opacity makes them indistinguishable. When a tool can’t show its source data or how it weighted competing inputs, there is no way to know whether a conclusion reflects sound analysis or a plausible-sounding guess.
Call it the 99% rule: you can’t rely on any of it if you don’t know which 1% is wrong, and that 1% is often exactly the input that would have changed the conclusion. A single overlooked liability or misread accounting adjustment can materially change an investment thesis, and once that error is embedded in an automated research or trading system, it doesn’t stay contained. It gets repeated, redistributed, and compounded across other tools and portfolios drawing on the same or similar sources. If you cannot see the data or source behind a recommendation, you are not meeting the standard fiduciaries are expected to meet—you are trusting a black box with other people’s capital.
Credit Unions Confront the Risks of General-Purpose AI
Credit unions face a version of this problem that is arguably more exposed than at a large broker-dealer, because a general-purpose, consumer-facing AI tool was never built to understand financial filings, member account data, or lending disclosures in the first place. Financial data, whether it’s a call report, a member’s financial history, or a company’s regulatory filing, is like crude oil: it cannot go straight into the engine. It has to be refined, verified, and organized before it becomes a reliable input, and most consumer AI tools skip that refinement step entirely, drawing instead on broad, unvetted internet content.
When a credit union uses one of these tools to analyze a loan applicant, screen an investment, or generate advice for a member without understanding what data shaped its answer, it inherits every bias, gap, and error in that underlying data. Unlike a large bank, many credit unions do not have in-house data science or compliance teams to audit AI outputs before they reach a member, which raises the stakes considerably. A biased or fabricated data point that a large institution’s controls would catch can move straight through to a lending decision or member recommendation, exposing the credit union to regulatory, fair-lending, and reputational risk it may not even know it is carrying.
How Credit Unions Can Evaluate the Reliability of AI-Generated Financial Insights
Before letting an AI-generated insight influence a lending decision, an investment recommendation, or advice given to a member, a credit union should be able to answer the same questions a broker-dealer would ask of any investment tool.
Can the conclusion be traced to verifiable data, whether that’s a member’s actual financial documentation or a company’s regulatory filings, rather than a generalized summary?
Is the underlying methodology transparent and applied consistently across similar members and applications? This matters not just for reliability but for fair-lending compliance, since inconsistent, unexplainable criteria are exactly what invites regulatory scrutiny.
Can a loan officer or advisor independently validate the result rather than simply passing it along?
Does the system flag missing or uncertain information instead of quietly filling the gap with an assumption?
And critically, could this output and the reasoning behind it hold up if a member, an examiner, or an NCUA auditor asked the credit union to explain it?
If the answer to any of those is no, the tool is not ready to sit upstream of a decision that affects a member’s finances. AI can meaningfully expand what a credit union’s team can review and how quickly, but the institution, not the AI, remains accountable for every recommendation that reaches a member.
Why Explainability Is Essential for Trusted AI-Driven Financial Decisions
Explainability is and always has been important to sophisticated researchers. They have, since the beginning of time, been held to standards requiring that they explain how they arrive at their conclusions. Their peers would reject anything less.
Currently, for AI, explainability is becoming critical because laypeople are realizing that AI is not reliable, and they know that confident answers no longer satisfy the burden of proof financial decisions require. Research from Sage found that 71% of finance leaders would reject an AI system that was 99% accurate if it could not explain how it reached its conclusions, and more than half said they would pay a premium for a system that could. That is not a preference for transparency as a nice-to-have; it reflects the reality that compliance teams, regulators, investment committees, and clients all need to see the evidence and reasoning behind a recommendation, not just the recommendation itself.
When explainability is absent, a firm cannot defend a decision after the fact, cannot identify where an error entered the process, and cannot demonstrate to an examiner or a client that a recommendation reflected sound judgment rather than a fluent guess.
An answer that cannot be defended is difficult to trust, and depending on the context, it might expose the firm to real regulatory and legal risk.
As AI becomes a permanent part of investment workflows, the firms with the advantage will not necessarily be the ones with the fastest models, but the ones whose systems can prove, not just produce, their conclusions.
About David Trainer
David Trainer is a Wall Street veteran and corporate finance expert with more than 25 years of experience in fundamental analysis, valuation, and financial statement research. As Founder and CEO of New Constructs, he has spent more than two decades challenging traditional investment research by combining forensic accounting with artificial intelligence to uncover the true economics behind public companies. His work focuses on helping investors look beyond accounting distortions, market narratives, and headline metrics to evaluate business performance based on economic reality.
Before founding New Constructs, David spent more than six years on Wall Street, including roles at Credit Suisse First Boston and Epoch Partners, where he developed proprietary valuation frameworks and led initiatives to apply economic earnings analysis across industries. A former member of the Financial Accounting Standards Board (FASB) Investor Advisory Committee, he is also the author of Modern Tools for Valuation (Wiley Finance). His research has been recognized by Harvard Business School, MIT Sloan, and Ernst & Young, and he is a frequent commentator on market trends, valuation, AI in finance, and investment risk across leading financial media.
