An AI system that checked a health plan's policies against a state compliance questionnaire and returned a sourced decision on each requirement for an auditor to review.
Healthcare · Audit Support · Document QA · Evidence Retrieval
Challenge
The challenge was completing a compliance audit faster, where every requirement in a state questionnaire had to be checked against a large body of policy documents.
System
The system parsed the audit questionnaire, retrieved the relevant policies for each requirement, decided whether the requirement was met and returned that with citations to the source, and presented each decision to an auditor to approve or change before it went into the submission.
Outcome
Completing an audit fell from about 35 hours to about 7, around 28 hours saved each time. Every decision was backed by a citation, and audit documents were processed under a HIPAA agreement.
Background
The client conducted compliance audits for health plans in Medi-Cal, California's Medicaid program, overseen by the state's Department of Health Care Services (DHCS). These audits checked a health plan's policy and procedure documents against the state's requirements.
For each audit, DHCS issued a standardized questionnaire, and every requirement in it had to be supported by the plan's own policies, with a decision recorded on whether the requirement was met. The audit in focus here was the hospice services questionnaire, APL 25-008.
Problem
A single questionnaire carried dozens of requirements, and answering one meant searching hundreds of policy documents for the passage that addressed it, judging whether the requirement was met, and recording the citation.
By hand, an audit took about 35 hours. The evidence for any requirement could sit anywhere across the policies, so most of the time went to finding it.
How it was solved
The system read the audit questionnaire and separated it into its individual requirements, each one a question to evaluate.
For each requirement, it searched the health plan's policies for the passages most likely to address it. The documents were split into smaller chunks and indexed with an embedding model built for legal and regulatory language, which matched the wording of a requirement to the wording of a policy more accurately on this kind of text.
The search combined vector similarity with keyword matching, so a policy that phrased something differently than the questionnaire was still found.
With the relevant passages retrieved, the system asked Claude to decide whether the requirement was met and to return a yes or no with citations to the policy text behind it. Each decision pointed back to the exact passage it rested on.
An auditor reviewed each decision before it entered the audit. The interface showed the retrieved passages and a confidence indicator beside the system's determination, and the auditor could change the determination or add notes.
The system saved progress so a long audit could be paused and resumed, focused the search on the policies named by the questionnaire's APL reference, and exported the finished audit into the state's submission form for filing.
Audit documentation contains protected health information, so the system ran under a HIPAA agreement with Anthropic that permitted this data to be processed by the model.
Impact
5x faster audits. Completing an audit fell from about 35 hours to about 7, around 28 hours saved each time.
Every decision backed by a citation. Each met-or-not determination linked to the exact policy passage behind it, so the auditor could verify it against the source.
Handled protected health information under a HIPAA agreement. The audit documents were processed through the model within the terms that data requires.
Completing an audit ran about 5x faster, from roughly 35 hours by hand to about 7, around 28 hours saved on each audit. The auditor reviewed determinations the system had already drafted and sourced, then approved or changed each one.
Every determination carried a citation to the policy passage behind it, so the auditor could confirm it against the source and kept final say over the submission.
The system ran under a HIPAA agreement with Anthropic, which permitted the protected health information in the audit documents to be processed by the model.
