HIPAA-Compliant Local LLM for Medical Records: On-Prem Extraction Blueprint
Extract and summarize medical records with a local LLM so PHI never reaches a third-party API. Architecture, VRAM sizing and a Docker Compose setup.
AI Biz Gate Editorial · Local AI infrastructure, private RAG and compliance research
Published
Architecture
Recommended model
Qwen3-32B (Q4_K_M)
Parser / tooling
Docling (OCR + layout analysis)
VRAM required
27.5 GB
Q4_K_M, 32k tokens
Compliance standard
HIPAA
Clinical documents are parsed by Docling on an isolated host, then passed to a locally served Qwen3-32B for structured extraction of conditions, medications and encounter dates. PHI never leaves the hospital network: the inference server has no outbound internet route, intake storage is encrypted, every prompt and response is written to an access-controlled audit log, and outputs land in your EHR integration layer for clinician review.
01
Ingest
Scanned PDFs and faxes land in an encrypted intake share.
02
Parse
Docling runs OCR and layout analysis, emitting structured JSON per page.
03
Extract
Qwen3-32B returns FHIR-aligned JSON (conditions, medications, encounters) against a fixed schema.
04
Review & log
A clinician approves each output; prompts, responses and user IDs are written to the audit log.
Performance & privacy profile
| Metric | Value | Basis |
|---|---|---|
| VRAM required | 27.5 GB | Weights + FP16 KV cache + 1 GB runtime overhead |
| Decode speed | ~33–51 tok/s | Memory-bandwidth estimate on RTX 5090 32GB, single user |
| Table extraction accuracy | Pending lab test | Will be measured on our reference document set |
| Privacy score | 9/10 | Editorial rubric, see breakdown below |
- On-prem inference, no third-party AI processor(implemented)+3
- No outbound network route from AI hosts(implemented)+2
- Encryption at rest for documents and indexes(implemented)+1
- Per-request audit logging(implemented)+2
- Role- or matter-based access control(implemented)+1
- Data minimization before storage (redaction or hashing)(not included)+1
Deployment config
Starting point only. Pin image versions and review network policy before production.
# docker-compose.yml: PHI stays on an internal-only network.
# Models must be pulled on a connected staging host first (ollama pull qwen3:32b),
# then copy the ollama volume across; the internal network has no internet route.
services:
ollama:
image: ollama/ollama:latest # pin a specific tag in production
volumes:
- ollama:/root/.ollama
networks: [phi-internal]
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: all
capabilities: [gpu]
docling:
image: quay.io/docling-project/docling-serve:latest
networks: [phi-internal]
# Your EHR integration service joins phi-internal and calls
# http://docling:5001 and http://ollama:11434
networks:
phi-internal:
internal: true # no route to the internet
volumes:
ollama:Size it for your hardware
Local LLM VRAM Calculator
Total VRAM required
27.5GB
- Weights
- 18.5 GB
- KV cache
- 8.0 GB
- Overhead
- 1.0 GB
Recommended local hardware
- RTX 5090 32GB~33–51 tok/s
- Mac mini M4 Pro 64GB~5–8 tok/s
Estimated single-user decode speed from memory bandwidth. Real throughput depends on runtime, drivers and prompt length.
Frequently asked questions
Does running an LLM locally make it HIPAA compliant?
No software is HIPAA compliant on its own: HIPAA applies to covered entities and business associates and how they handle protected health information. Running the model on infrastructure you control removes an AI vendor from the data flow, which can remove the need for a Business Associate Agreement with that vendor. You still need a documented risk analysis, access controls, audit controls (45 CFR 164.312(b)), encryption and workforce policies that cover the system.
How much GPU memory do I need to process medical records locally?
For this blueprint (Qwen3-32B at Q4_K_M with a 32k-token context) we estimate about 27.5 GB of VRAM, which fits a single 32 GB card such as the RTX 5090. A shorter context or a smaller model reduces this; use the calculator on this page to size alternatives for your hardware.
Should a clinician review the extracted data?
Yes. LLM extraction can miss or misread details, especially on low-quality scans or handwritten notes. Treat outputs as drafts: route them to a clinician or medical coder for approval before they enter the record, and log who approved each output so the audit trail is complete.