When AI Tools Touch PHI
Any AI system that receives, processes, or generates individually identifiable health information on behalf of a covered entity or business associate is handling PHI under HIPAA. This is not limited to systems explicitly designed for healthcare — a general-purpose language model used to interpret beneficiary intake messages, generate enrollment recommendations, or respond to health-related questions is processing PHI if those messages contain identifiable health information. The application of HIPAA to AI follows existing business associate rules; there is no AI exception.
HIPAA's definition of protected health information (PHI) is functional, not format-based: it is any individually identifiable health information transmitted or maintained in any form that relates to an individual's health condition, healthcare, or payment for healthcare (45 CFR § 160.103). The fact that this information passes through an AI model does not change its status as PHI.
Which AI use cases create PHI exposure
- Natural language intake systems that interpret beneficiary messages containing health status, medication, or eligibility information
- AI-generated responses to beneficiary questions about their coverage or health condition
- AI systems that generate plan recommendations based on individual health information
- Automated summarization tools that process clinical notes, enrollment records, or call transcripts
- AI-powered audit tools that review claims data or enrollment records
Which AI use cases are generally outside HIPAA scope
- AI systems used exclusively for de-identified data — where the data has been de-identified under 45 CFR § 164.514(b) before the AI processes it
- AI tools used internally for operational purposes that have no access to PHI datasets
- General business productivity AI (email drafting, document formatting) that is not given access to PHI
Business Associate Agreement Requirements for AI Vendors
If an AI vendor processes PHI on behalf of a covered entity or another business associate, a signed HIPAA Business Associate Agreement (BAA) is required before PHI can be shared with the vendor (45 CFR § 164.308(b)(1)). This is non-negotiable — sharing PHI with a vendor that lacks a BAA is a HIPAA violation regardless of whether any harm results. Many commercial AI providers offer BAAs for enterprise customers but not for standard API access; organizations must verify BAA availability and review its terms before using AI tools with PHI.
What a HIPAA-compliant AI BAA must include
- 1Permitted uses and disclosures — Specifies exactly what PHI the vendor may access and for what purposes. Must not permit use for vendor's own purposes (including model training) without explicit authorization.
- 2Prohibition on model training with PHI — Explicit prohibition on using customer PHI to train, fine-tune, or evaluate foundation models unless separately authorized in writing.
- 3Security safeguards — Requires the vendor to implement appropriate administrative, physical, and technical safeguards under 45 CFR § 164.314.
- 4Breach notification — Requires the vendor to report any breach of PHI within 60 days, matching HIPAA's Breach Notification Rule timeline.
- 5Subcontractor chain — Requires the vendor to execute BAAs with any subprocessors (sub-BAs) that handle PHI, and to flow down BAA obligations throughout the chain.
- 6Return or destruction of PHI — Upon contract termination, PHI must be returned to the covered entity or destroyed, including in backups and prompt logs.
- 7Audit access — Must permit the covered entity or its designee to audit the vendor's PHI handling practices upon reasonable notice.
Common BAA gaps in AI vendor agreements
Many AI vendor BAAs are written primarily for the vendor's protection, not the covered entity's. Common gaps include: no explicit prohibition on PHI use for model training (relying instead on vague "we don't use your data for training" marketing language), unclear subcontractor disclosure requirements, breach notification timelines that exceed HIPAA's 60-day requirement, and vague PHI destruction provisions at contract end. Review BAA terms against HIPAA's required elements in 45 CFR § 164.314(a)(2)(i)-(ii) before signing.
Prompt Logging and Data Retention Risk
AI systems generate logs of inputs (prompts) and outputs by default. When those prompts contain PHI — a beneficiary's name, health condition, insurance status, or any identifiable combination — those logs are themselves PHI and must be secured, retained, and disposed of according to HIPAA requirements. Many organizations deploy AI tools without considering that prompt logs create a new PHI data store that may persist on vendor infrastructure indefinitely without appropriate controls.
Where prompt logs live and why it matters
Prompt logs may exist in: the AI vendor's infrastructure (API request logs), the calling application's own logging layer, cloud provider logging services, and monitoring tools integrated with the AI pipeline. Each of these is a potential PHI repository. Organizations must map all prompt log destinations and ensure that each is covered by appropriate HIPAA safeguards — encryption at rest, access controls, retention limits, and audit capabilities.
Retention schedule for prompt logs containing PHI
HIPAA does not specify a single universal retention period for all PHI — retention requirements depend on the type of record and state law. However, HIPAA Security Rule documentation must be retained for 6 years from creation or last effective date, and state medical record laws impose their own requirements. Organizations should establish a prompt log retention policy that satisfies applicable requirements and includes automated deletion of logs containing PHI after the required retention period expires.
Designing to minimize PHI in prompts
The most effective way to manage prompt log risk is to design AI prompts that minimize PHI exposure. Techniques include: passing anonymized or tokenized identifiers in prompts rather than names, keeping PHI in a separate data store accessed only by reference (e.g., "beneficiary ID: 12345" rather than including full records), and using system-level context injection that limits what PHI the model sees for any given task.
The Minimum Necessary Standard for AI Systems
HIPAA's minimum necessary standard (45 CFR § 164.502(b)) requires covered entities and business associates to limit PHI use and disclosure to the minimum necessary to accomplish the intended purpose. For AI systems, this principle requires active configuration: prompts should not include more PHI than the model needs, model outputs should not expose PHI that was not required for the response, and AI system access to PHI datasets should be scoped to the specific function being performed.
Applying minimum necessary to AI prompt design
Common minimum necessary violations in AI deployments include: sending entire patient records to an AI model when only a single field is needed for the task, including full beneficiary identifiers in prompts when a tokenized reference would suffice, and allowing AI systems broad read access to PHI databases rather than scoped, function-specific access. Each of these represents a potential HIPAA violation independent of any actual breach.
Minimum necessary for AI outputs
AI-generated outputs must also comply with the minimum necessary standard. A model that is given access to a broad range of PHI and asked a narrow question must not include unrequested PHI in its output. System prompts should instruct the model on scope constraints, and output filtering should verify that responses do not include PHI beyond what was required to answer the query.
TPMO Crossover: When AI Platforms Become TPMOs
An AI platform that presents Medicare Advantage or Part D plan options to beneficiaries, generates plan comparisons, facilitates enrollment, or materially assists a sales transaction under a written arrangement with a plan sponsor is likely a TPMO under CMS rules — regardless of whether it is staffed by humans. CMS applies a functional test based on what the platform does to the beneficiary interaction, not how the platform is technically structured. AI-powered plan recommendation engines face the same classification analysis as human-operated ones.
| AI Function | TPMO Risk | Analysis |
|---|---|---|
| Interprets beneficiary intake, routes to licensed agent | Lower risk | Intake-only function; no plan presentation to beneficiary |
| Generates plan comparisons or cost estimates | High risk | Presenting plan information to beneficiary — likely TPMO function |
| Recommends a specific plan based on beneficiary inputs | High risk | Sales presentation function; CMS treats recommendation as marketing |
| Processes enrollment forms submitted by beneficiaries | High risk | Enrollment facilitation is explicitly a TPMO function |
| Responds to general Medicare questions (educational) | Medium risk | Depends on specificity; plan-specific content triggers TPMO analysis |
| Eligibility verification only (Part A/B status) | Lower risk | Administrative function without plan presentation |
Organizations deploying AI in Medicare programs should document the functional analysis of whether each AI component crosses the TPMO line — and ensure that TPMO disclaimer delivery, recording, and SOA requirements are built into any component that does. See the CMS TPMO Compliance guide for a full breakdown of TPMO obligations.
AI Vendor Evaluation: Compliance Questions to Ask
Before deploying any AI vendor in a Medicare program that handles PHI, compliance teams should verify BAA availability, data use policies (specifically model training), encryption standards, audit log access, breach notification processes, and subcontractor disclosure. Vendors that cannot answer these questions clearly — or that provide answers only in marketing materials rather than contractual terms — are unlikely to be adequately prepared for HIPAA obligations.
HIPAA Security Rule Requirements for AI Systems
The HIPAA Security Rule (45 CFR Part 164, Subpart C) applies to all electronic PHI (ePHI), which includes data processed by AI systems, prompt logs, model outputs containing PHI, and API calls transmitting beneficiary data. The three categories of required safeguards — administrative, physical, and technical — must all be addressed for any AI system that handles ePHI. HIPAA's Security Rule is not waived because the ePHI passes through an AI model rather than a traditional database.
Technical safeguards applicable to AI systems
- Access controls (45 CFR § 164.312(a)) — Unique user IDs, emergency access procedures, automatic logoff, and encryption for AI system access. API keys and service accounts must be managed with least-privilege principles.
- Audit controls (45 CFR § 164.312(b)) — Hardware, software, and procedural mechanisms to record and examine access to ePHI. AI systems must generate audit logs of what PHI was accessed, by which component, and when.
- Integrity controls (45 CFR § 164.312(c)) — Mechanisms to confirm ePHI is not improperly altered or destroyed. For AI systems, this includes verifying that model outputs have not been tampered with before they are acted upon.
- Transmission security (45 CFR § 164.312(e)) — Encryption of ePHI in transit. All API calls transmitting PHI to or from AI models must use TLS 1.2 or higher.
Risk analysis obligation
The HIPAA Security Rule requires covered entities and business associates to conduct and document a risk analysis of the potential risks and vulnerabilities to ePHI (45 CFR § 164.308(a)(1)(ii)(A)). The introduction of AI systems that process ePHI constitutes a material change to the technical environment and must trigger an update to the organization's risk analysis — including identification of new threat vectors (model inversion attacks, prompt injection, training data leakage) specific to AI systems.
Primary Sources
ecfr.gov → 45 CFR § 164.308
ecfr.gov → 45 CFR § 164.502(b)
ecfr.gov → HIPAA Security Rule
hhs.gov → OCR Tracking Technology Guidance
ecfr.gov → 42 CFR § 422.2268