A year ago, the question was whether to explore AI at all. Today it’s which tool to buy and when to implement it. That shift is outpacing the harder question: can your data infrastructure actually support what you’re evaluating?
AI vendors are good at demos. Their tools look clean, capable, and purpose-built for the problems on your whiteboard. What those demos almost never show is what the model produces when run against five years of real behavioral health EHR data, with its template drift, multi-site inconsistency, incomplete screening fields, and free-text workarounds. Most executives see a vendor pitch before they see their own data. That sequence is where AI investments go wrong.
Behavioral Health Data Has a Structural Problem AI Cannot Fix
The figure cited most often in health technology discussions, roughly 80% of clinical EHR data lives in unstructured free text, understates the challenge for behavioral health specifically. Research published in the Journal of Medical Internet Research confirms that unstructured data provides significantly more clinical information than structured counterparts, and that the ratio of free-text to coded data varies meaningfully by care context, with psychiatric care among the settings most reliant on narrative documentation (JMIR, 2025). Progress notes, treatment narratives, clinical observations, and session summaries are the primary record here. The structured fields anchoring other specialties, lab values, imaging results, and procedure codes, play a smaller role.
The instruments designed to generate structured behavioral health data compound this problem in practice. PHQ-9s and GAD-7s are theoretically discrete, computable fields. In real organizations, they are entered as free text, duplicated across templates, scored inconsistently across clinicians and sites, and left incomplete at rates that would alarm any data scientist preparing a training dataset. Research in the Journal of Primary Care and Community Health found that even in primary care, a more highly regularized documentation environment, the gap between physician knowledge of standardized screening tools and consistent utilization was substantial, with willingness to use GAD-7 and PHQ-9 running well below awareness rates (Waheed et al., 2024). In behavioral health settings, where clinical narrative carries more weight and documentation norms vary more widely across programs and clinicians, that gap is wider. The ONC’s 2024 data brief on EHR adoption among substance use and mental health facilities documented persistent gaps in interoperability, data exchange, and structured data capture across the sector (ONC/SAMHSA, 2026).
The gap between “we administer PHQ-9s” and “our PHQ-9 scores are clean, consistent, and structured across every site” is the crux of that readiness evaluation. A vendor demo runs on vendor data. The only data that matters for your implementation is yours.
What Goes Into the Model Comes Out in the Output
AI models do not filter bad data. They learn from it. When a model ingests five years of behavioral health EHR records, with their inconsistent PHQ-9 entries, incomplete screening fields, and documentation patterns that vary by clinician and site, it does not flag those anomalies. It encodes them as normal. The output it produces afterward, whether a risk score, a clinical summary, or an outcomes prediction, reflects a version of reality that was never accurate to begin with. And the model will deliver that output with the same confidence it would if the underlying data were clean.
It has no way to tell the difference. You do.
NIST’s AI Risk Management Framework identifies valid and reliable performance as the foundational trustworthiness characteristic for AI systems, treating data quality as a prerequisite condition for any meaningful evaluation of model output (NIST, 2023). The framework does not treat data quality as a technical footnote. It treats it as the condition under which all other trustworthiness claims are either supported or invalidated.
Recent peer-reviewed analysis published in PLOS Digital Health goes further, identifying inconsistencies in de-identified EHR data as a foundational risk for AI systems trained on clinical records, and calling for rigorous data curation before any model deployment in healthcare settings (Sabet et al., 2025). The clinical and operational consequences of acting on unreliable AI output, in risk stratification, clinical documentation, or outcomes reporting, are not theoretical. They are the predictable result of skipping the step that vendors have no incentive to require of you.
The question worth asking before any demo is whether your data will produce output you can actually act on.
The Data Audit Belongs on the Executive Agenda
Following the standard AI adoption sequence without first assessing data quality is a significant bet on a foundation you have not inspected. That sequence runs from use case identification to vendor evaluation to contract negotiation to implementation. Most organizations complete all four steps before anyone has looked closely at the data those tools will actually process.
The sequence that produces reliable output starts earlier. Assess current data quality. Identify the gaps that would undermine the specific use case under consideration. Address those gaps at the EHR configuration and workflow level. Then evaluate vendors with a realistic picture of what your data will actually deliver. ONC has recognized data quality as a prerequisite for responsible AI development in healthcare, funding specific initiatives to improve healthcare data quality in support of AI tools, a signal that federal policy already treats the data infrastructure question as upstream of the technology question (ONC, 2025).
Buying AI before auditing your data is committing budget to a system before understanding whether the inputs will support the outputs you need.
This is a governance decision that belongs at the executive level, not an IT project to be delegated after the contract is signed. The organizations positioned to extract real value from AI tools in the next cycle are the ones whose leadership demanded a data infrastructure assessment before the vendor conversation reached the budget stage. Xpio Analytics is built on that premise, surfacing what your EHR data actually contains so that decisions about optimization, reporting, and tool adoption start from an accurate picture of current state.
The practical question is what that assessment looks like in operational terms: which data categories to audit, what “good enough” means for the use cases you are considering, and how to document findings in a format useful for both vendor evaluation and internal improvement planning. That is the subject of the companion post in this series.
If you are not sure where to start, Xpio Health works with behavioral health organizations to evaluate data infrastructure and EHR configuration ahead of AI tool decisions. Contact us when you are ready to look at the data before the demo.
#BehavioralHealth #PeopleFirst #XpioHealth #AIGovernance #EHROptimization #BehavioralHealthLeadership
References
- Ye C, et al. Using Structured Codes and Free-Text Notes to Measure Information Complementarity in Electronic Health Records: Feasibility and Validation Study. Journal of Medical Internet Research. 2025. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11887999/
- Waheed A, et al. Knowledge and Behavior of Primary Care Physicians Regarding Utilization of Standardized Tools in Screening and Assessment of Anxiety, Depression, and Mood Disorders at a Large Integrated Health System. Journal of Primary Care and Community Health. 2024. https://journals.sagepub.com/doi/10.1177/21501319231224711
- Office of the National Coordinator for Health Information Technology. Electronic Health Record Adoption and Exchange Capabilities Among Substance Use and Mental Health Treatment Facilities, 2024. ONC Health IT Data Brief. 2026. https://healthit.gov/data/data-briefs/electronic-health-record-adoption-and-exchange-capabilities-among-substance-use-and-mental-health-treatment-facilities-2024/
- National Institute of Standards and Technology. Artificial Intelligence Risk Management Framework (AI RMF 1.0). NIST AI 100-1. 2023. https://nvlpubs.nist.gov/nistpubs/ai/nist.ai.100-1.pdf
- Sabet CJ, et al. Regulating Medical AI Before Midnight Strikes: Addressing Bias, Data Fidelity, and Implementation Challenges. PLOS Digital Health. 2025. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12360511/
- Office of the National Coordinator for Health Information Technology. Funding Announcements: Improving Healthcare Data Quality to Support Responsible Development of Artificial Intelligence. 2025. https://healthit.gov/about/funding-announcements/
