Building HIPAA-Compliant AI Pipelines: An Engineering Reference

Engineering Reference April 26, 2026 14 min read By Veklom Engineering

This is the architecture you actually need. Not a checklist. Not a marketing white paper. The specific design decisions, the regulatory citations they answer to, and the code that implements them.

The Regulation, Briefly

HIPAA's enforcement target is not "AI" or "the cloud." It is electronic protected health information — ePHI — under 45 CFR § 160.103. Anything that creates, receives, maintains, or transmits ePHI is subject to the Security Rule (§ 164.302–318) and the Privacy Rule (§ 164.500–534). An AI pipeline that touches ePHI is therefore subject to the same controls as a billing system, a chart-review tool, or an EHR integration.

The relevant Security Rule sub-parts for an AI pipeline are:

• § 164.308(a)(1)(ii)(D) — Information system activity review (your audit log)
• § 164.312(a)(1) — Access control (your authn/authz)
• § 164.312(b) — Audit controls (mechanism, not just policy)
• § 164.312(c)(1) — Integrity (tamper-evidence)
• § 164.312(d) — Authentication (the user is who they say they are)
• § 164.312(e)(1) — Transmission security (encryption in transit)
• § 164.514(a)–(c) — De-identification (Safe Harbor and Expert Determination)

Every architectural decision below answers to one or more of these. The point of writing the code this way is so that when an OCR auditor walks in, the engineer can point at the line and the line of the regulation in the same sentence.

Step 1: PHI Tokenization at the Boundary

The single most consequential decision is whether the LLM ever sees raw PHI. If it does, every component the call traverses is in HIPAA scope and must be operated under a BAA. If the PHI is tokenized before the gateway and de-tokenized after, the LLM operates on synthetic data and is potentially out of scope — depending on your covered-entity determination and the strength of the de-identification.

The Safe Harbor method (§ 164.514(b)(2)) lists 18 identifiers that must be removed. Names, addresses (smaller than state), dates more granular than year, telephone numbers, fax, email, SSN, MRN, account numbers, certificate/license numbers, vehicle identifiers, device identifiers, URLs, IPs, biometric identifiers, full-face photos, and any other unique identifying number or code.

A reference Python implementation of the boundary tokenizer:

from veklom.phi import Tokenizer, SafeHarborPolicy

tok = Tokenizer(policy=SafeHarborPolicy())

# At ingress: replace every identifier with a stable, opaque token
tokenized, vault = tok.scrub(prompt)
#  -> "Pt_5af3 presented with chest pain at Loc_a91d on Date_38f2"
#  -> vault: { "Pt_5af3": "John Doe", "Loc_a91d": "...", ... }

# Send tokenized text to LLM (out of HIPAA scope, by Safe Harbor)
response = gateway.complete(tokenized, model="...")

# At egress: rehydrate tokens for clinical display
final = tok.rehydrate(response, vault)

The vault is the only PHI-bearing artifact and lives in your encrypted store with workspace-scoped key access. The model never sees the names. The audit log records the token-set, not the identifiers.

Step 2: The Audit Log is Not a Log File

§ 164.312(b) requires "hardware, software, and/or procedural mechanisms that record and examine activity." The single biggest failure we see in healthcare AI pipelines is treating this as a request to write to app.log. That is not an audit control. That is a debug artifact.

The audit log under HIPAA must be:

1. Tamper-evident. Append-only, hash-chained, with a daily hash anchor written to a separate system. If a row is modified, the chain breaks visibly.
2. Per-event addressable. Every prompt, every response, every model decision, every routing decision, every error — one row each, with a stable event ID.
3. Tenant-scoped at storage. The audit table for Workspace A is logically separate from Workspace B at the row-security level. A bug in one application cannot expose the other's audit trail.
4. Retained for six years. Per § 164.316(b)(2)(i), the documentation retention requirement.

The Veklom audit schema implements this with row-level security in Postgres and a per-day SHA-256 chain anchor written to S3 Object Lock with WORM retention.

Step 3: BAA Scope & the Provider Stack

Every component handling ePHI — database, object store, queue, container runtime, observability stack — needs to be covered by a Business Associate Agreement, either with you (if you operate it) or with a vendor who has signed a BAA with you.

Three tiers of provider posture, ordered by ease of compliance:

1. You operate everything inside your own perimeter. No BAA needed externally; your internal data-handling policies cover it. This is the Veklom default deployment posture.
2. You operate inside a hyperscaler that signs a BAA for the listed services. AWS, Azure, GCP all have published HIPAA-eligible service lists. You stay inside the list. The BAA is a single document.
3. You use third-party services that each require their own BAA. Logging, monitoring, error tracking, embeddings provider — each needs a signed BAA in place before any ePHI flows. Plan for procurement to take 6–12 weeks per vendor.

The single most expensive mistake we see is teams beginning to flow ePHI before all three tiers of BAA are signed. Even a minor breach in this state escalates to a Notice of Privacy Practices violation and a corrective-action plan from OCR, regardless of whether any data was actually exposed.

Step 4: Encryption Boundaries

§ 164.312(a)(2)(iv) and § 164.312(e)(2)(ii) require encryption "to a level commensurate with the risk." NIST SP 800-66r2 interprets this as AES-128 minimum for at-rest and TLS 1.2+ for in-transit. In practice we use AES-256-GCM at rest and TLS 1.3 in transit, with mTLS between every internal service.

The boundary diagram (Fig. 1, top of post) shows the three encryption transitions: client → gateway, gateway → model, gateway → storage. Each is a separately-managed key, each is rotated quarterly under § 164.308(a)(5)(ii)(C), and each rotation event is itself an audit event.

Step 5: Access Control & Authentication

§ 164.312(a)(1) and § 164.312(d) require role-based access and verifiable authentication. The implementation pattern is:

• OIDC-only authentication against the customer's identity provider. No vendor-managed user accounts. The covered entity owns the user lifecycle.
• RBAC with workspace scoping. Every request carries a JWT with workspace claims. Authorization is enforced at the gateway, the data layer, and the audit layer.
• Session timeout under § 164.312(a)(2)(iii). Default 15 minutes idle; configurable to the covered entity's policy.
• Emergency access procedure under § 164.312(a)(2)(ii). A documented break-glass workflow that grants temporary elevated access, generates a high-priority audit event, and notifies the covered entity's privacy officer.

Step 6: De-identification, Re-identification, & Expert Determination

Safe Harbor (Step 1) is the simpler path. Expert Determination (§ 164.514(b)(1)) is the more powerful one: a qualified statistician determines that the risk of re-identification is "very small" and documents the methodology. This permits retention of more clinically-useful data — including dates more granular than year — at the cost of a paid statistical review and an annual re-evaluation.

For most AI pipelines, Safe Harbor is sufficient. For research pipelines or longitudinal cohort analyses, Expert Determination unlocks workloads that Safe Harbor cannot support. The pipeline should support both modes and tag every PHI element with the de-identification method used so the audit trail is complete.

The Reference Deployment

The full reference deployment ships with Veklom under the Sovereign Pro tier. It includes:

• Boundary tokenizer with Safe Harbor and configurable Expert Determination policies
• Tamper-evident audit table with row-level security and S3 Object Lock anchoring
• Pre-configured RBAC policies for the eleven HIPAA control families
• OIDC integration shims for the major identity providers
• Quarterly key-rotation runbook with audit-event emission
• Six-year retention policies as Postgres partitions with a documented archive flow
• OCR-ready evidence packet generator (point-in-time export of policies, configurations, and a 30-day audit sample)

The architecture is the same one we deploy for hospital systems and academic medical centers. The cost-engineering complement is in a separate piece. The procurement-realities companion is here.