How do you implement an LLM on-premise in a hospital?

The failure pattern in hospital AI is not technical. It is starting with the case that excites the clinical directorate, discovering eighteen months of governance work underneath it, and losing the budget before anything reaches a ward.

The sequence that works starts where the documents are dense, the errors are visible and the data is not health data: internal protocols and procedures, regional and national circulars, tenders and purchasing rules, accreditation documentation, HR and administrative policy. These corpora are large, badly searchable, constantly superseded, and cost real staff time every week. A retrieval assistant over them pays for itself and, more importantly, builds the platform — identity, ingestion, audit, retention, operations — that a clinical use case would otherwise have to build from zero.

The same reasoning is why the reference deployment on this site is a regulatory assistant for pharmacies rather than a clinical tool: documentary currency is a real, expensive problem, and it can be solved without touching a patient record.

Health data is special-category data under the GDPR, and the cheapest way to hold that obligation is to keep it out of the index entirely. Exclusion at ingestion is stronger than suppression at answer time in every respect that matters: it is verifiable, it survives a prompt injection, it leaves nothing to purge, and it is far easier to explain to a data protection officer than a filter that mostly works.

Where a use case genuinely requires patient data, treat it as a different system with a different lawful basis, a different retention policy and a different approval path — not as an expansion of scope on the existing one. The architectural boundary should match the legal one, otherwise the first will erode until it no longer supports the second.

Two questions decide the rest: does personal data ever reach the generation step, and does it ever leave the perimeter. In a sovereign hybrid deployment the honest answer to the second is that retrieved context does leave, which is precisely why the first answer has to be no.

A hospital deployment is not a model with a web page in front of it. The components that determine whether it survives its first audit are the unglamorous ones:

• —Identity and access. Integration with the hospital directory, role segregation that mirrors clinical and administrative separation, and least privilege enforced at retrieval — not only in the interface.
• —Curated ingestion. A pipeline that knows document lineage, versions and validity intervals, so answers can state what is in force. Regulatory corpora are superseded constantly, and similarity search has no opinion about that.
• —Audit and reconstructability. A tamper-evident record of question, retrieval set, versions, model and disposition, with a retention period someone owns.
• —Human oversight. Explicit approval gates wherever an output has consequences, with the approver recorded. This is a workflow requirement, not a disclaimer in the footer.
• —Operations. Backup, monitoring, GPU capacity planning and a named on-call. An unavailable system in a 24-hour environment is a clinical risk of its own kind.

On hardware, the useful correction is that a documentary assistant does not need a research cluster. Production deployments of this shape run on modest GPU capacity because the expensive part is retrieval quality, not parameter count — the Federfarma system runs its local model, privacy filter and embeddings on a single 8 GB GPU while serving over a thousand pharmacies. Size the hardware after the retrieval design, not before it.

Two regimes decide how heavy the project becomes. Software intended for diagnosis, prevention, monitoring, prediction or treatment can qualify as a medical device under Regulation (EU) 2017/745, with the conformity assessment route that follows. Separately, the AI Act attaches obligations by intended purpose and role, and a system that is a safety component of a regulated product sits in its most demanding class.

A documentary assistant used by staff to find and interpret administrative and regulatory texts is a different object from a tool that proposes a diagnosis, and the distinction turns on intended purpose as declared and as actually used. Which means scope creep is a regulatory event: the moment a “procedure assistant” starts answering clinical questions, the classification conversation has to reopen. Build the boundary into the system, not into the training material.

Italian hospitals have a further layer — national AI legislation and the sector rules on secondary use of health data — discussed in the analysis of Legge 132/2025. And for organisations in scope of NIS2, an AI platform is part of the entity's own security posture and incident duties, not a project running beside them.