Leadership has asked about AI. A vendor demo is probably already on the calendar. And somewhere between the executive conversation about AI readiness and the actual contract decision, someone is going to ask a question your team is uniquely positioned to answer: what does our EHR data actually look like right now?
That question gets answered not in the boardroom, but by the people who configure your templates, manage your structured fields, and pull your reports. This post gives those people a framework for finding out, in a format that travels directly into the vendor evaluation conversation happening one level up.
Four audit categories, with specific things to look for in each. That’s a findings structure that connects IT knowledge to executive decision-making.
What the Audit Is Actually Measuring
The question is not whether your EHR contains data. It does. The question is whether that data is consistent, structured, and complete enough to produce reliable output for the specific AI use case under consideration.
That is a narrower question than it sounds. An organization running ambient documentation support needs different data conditions than one evaluating a risk-flagging tool, which needs different conditions than one building population-level outcome reports. The audit findings tell you which use cases your current data can support, which require remediation first, and which require infrastructure decisions before any vendor contract makes sense.
ONC’s 2025 SAFER Guides, updated this year to include AI-specific guidance, identify structured data configuration and validation as core EHR safety and reliability practices (ONC SAFER Guides, 2025). The same practices that determine whether your EHR is producing safe, reliable clinical output also determine whether it will produce reliable AI output. The audit is not a new exercise invented for AI readiness. It is a standard EHR integrity assessment applied to a new question.
The audit does not block AI adoption. It tells you which AI adoption is ready to succeed.
Four Categories Worth Examining Now
Structured field completion rates. For each data element relevant to your intended AI use case (diagnosis codes, screening scores, treatment plan fields, discharge disposition), pull the completion rate: what percentage of records have the field populated with a structured value versus left blank or filled with a free-text workaround. Do this across clinicians and across sites. Cross-site variation is as informative as overall rates. A 70% completion rate that is consistent across your organization tells a different story than a 70% rate masking one site at 95% and another at 30%. The aggregate number is not the finding. The distribution is.
Free-text versus coded data ratios for screening instruments. PHQ-9s and GAD-7s administered as structured instruments produce computable scores. The same instruments transcribed into a progress note produce prose. Run a query to determine what percentage of your screening administrations exist as discrete structured scores versus narrative documentation. That gap is your AI reliability gap for any use case involving mental health severity, treatment response, or population risk stratification. NIST’s AI Risk Management Framework identifies valid and reliable data as the foundational trustworthiness requirement for any AI system, treating data quality as the prerequisite condition under which all other trustworthiness claims are evaluated (NIST, 2023). The ONC SAFER Guides reinforce this at the EHR level, treating structured field validation as a baseline safety and reliability practice. Research confirms this gap exists even in primary care settings with standardized instruments and clear administrative expectations. In behavioral health, where documentation norms vary more widely across programs, clinicians, and sites, the gap is characteristically wider (Waheed et al., 2024).
Template consistency across clinicians and sites. Map the templates actively in use across your system. Identify fields that exist in some templates but not others, fields where different data types are capturing the same clinical concept, and templates that have drifted from their original configuration without a corresponding update to downstream reporting logic. Template inconsistency is the most common reason aggregate reporting produces unreliable results across a multi-site organization, and it is the category most likely to be invisible until someone runs a cross-site query and gets output that cannot be reconciled. The ONC’s 2024 behavioral health EHR data brief found that less than half of behavioral health facilities were using their EHRs consistently for key clinical workflows, and template drift is the operational mechanism that produces that statistic (ONC/SAMHSA, 2026).
Screening instrument capture discipline. This category goes one level deeper than the free-text ratio check. It asks whether your organization has defined and enforced a standard for when screening instruments are administered, which template captures them, and which staff role is responsible for documentation. Without that standard, the same patient may have PHQ-9 scores scattered across three different fields from multiple encounters, or none at all despite active treatment, because each clinician developed a local workaround that made sense to them and no one else. The data exists somewhere. The question is whether it exists in a location and format a model can find and trust.
Turning Findings into a Document That Travels
The audit produces value only if findings reach the people making AI vendor decisions. Structure the output in two parts.
The first is a data readiness summary by use case. For each AI application under consideration, summarize the current state across the four categories and identify what would need to change before that use case could produce reliable output. This document does not need technical depth. It needs a clear answer to one question: can this data support what we are evaluating? An IT team that hands leadership a one-page readiness summary by use case has done more to advance a sound AI decision than any vendor demo will.
The second is a remediation priority list. Separate the gaps into three buckets: configuration fixes your team can execute in days, documentation discipline and training gaps that require a coordinated workflow change, and structural EHR issues that require vendor involvement or architectural decisions. The distinction matters because the timeline and resource requirements are different, and leadership needs to understand the difference before committing a budget and an implementation schedule.
A data readiness summary is not a technical document. It is the translation layer between what IT knows and what leadership needs to decide.
Xpio Analytics is built to surface exactly this picture, pulling the structured and unstructured data layers apart so that IT and clinical informatics teams can see field completion rates, template variation, and screening capture patterns in one view, without running manual queries across multiple reports. The goal is to get your team to a findings summary faster, and to get that summary into the vendor conversation before the contract is drafted.
Xpio Health works with behavioral health IT and clinical informatics teams to run exactly this kind of data infrastructure assessment. If you want to know what your data looks like before the demo, we can help you find out.
#BehavioralHealth #PeopleFirst #XpioHealth #AIGovernance #EHROptimization #HealthIT
References
- Office of the National Coordinator for Health Information Technology. SAFER Guides. 2025. https://healthit.gov/clinical-quality-and-safety/safer-guides/
- National Institute of Standards and Technology. Artificial Intelligence Risk Management Framework (AI RMF 1.0). NIST AI 100-1. 2023. https://nvlpubs.nist.gov/nistpubs/ai/nist.ai.100-1.pdf
- Waheed A, et al. Knowledge and Behavior of Primary Care Physicians Regarding Utilization of Standardized Tools in Screening and Assessment of Anxiety, Depression, and Mood Disorders at a Large Integrated Health System. Journal of Primary Care and Community Health. 2024. https://journals.sagepub.com/doi/10.1177/21501319231224711
- Office of the National Coordinator for Health Information Technology. Electronic Health Record Adoption and Exchange Capabilities Among Substance Use and Mental Health Treatment Facilities, 2024. ONC Health IT Data Brief. 2026. https://healthit.gov/data/data-briefs/electronic-health-record-adoption-and-exchange-capabilities-among-substance-use-and-mental-health-treatment-facilities-2024
