Sovereign AI Architecture Patterns
A practitioner reference for designing, deploying, and assuring large language model systems when data residency, retrieval integrity, and audit evidence are non-negotiable — on-premises, private VPC, and air-gapped.
Why this paper exists
Most “AI platform” diagrams stop at a chatbot box connected to a public model API. That pattern fails board, regulator, and CISO review in banking, government, healthcare, and defence. Sovereign AI is not a slogan — it is an architecture contract answering four questions for every use case:
- Where does inference run? Facility GPU, private VPC endpoint, or air-gapped lab — never an unnamed third-party by default.
- Which corpora may be retrieved? Allow-listed knowledge bases with classification labels, retention, and access control.
- When must the system refuse? Insufficient citation coverage, policy gate failure, or missing human oversight hooks.
- What evidence survives an audit? Query lineage, model/version cards, retrieval traces, and signed export packs.
Audience & intended use
Written for enterprise architects, AI governance leads, CISOs, and programme sponsors who must approve production designs. Patterns are drawn from WAIG Foundation field work and the Sovereign LLM Workbench reference implementation. Educational references to ISO, NIST, and EU instruments are for literacy; proprietary UAAF operational criteria require written approval before operational use.
Strategic findings
- Pilots fail at the boundary. Teams that skip Retrieval Guard and evidence export rediscover hallucinations as legal and reputational risk after go-live.
- Topology is a control. On-prem, private VPC, and air-gap are different residual-risk postures — not interchangeable “deployment options.”
- Governance must wrap inference. Policy gates, consent, and logging belong in the request path, not in a weekly spreadsheet.
- Hybrid RAG without a guard is incomplete. Dense + sparse retrieval improves recall; the guard decides whether an answer may leave the system.
How to read this whitepaper
Section 02 defines the five-layer stack and data flow. Section 03 maps those layers onto three sovereign topologies with capacity and control implications. Section 04 deep-dives Retrieval Guard thresholds, refusal behaviour, and citation manifests. Section 05 closes with EU AI Act / ISO 42001 / NIST AI RMF mapping and evidence-pack contents for assurance programmes.
- Has the use case been risk-classified (unacceptable / high / limited / minimal literacy)?
- Is the inference boundary named in an architecture decision record (ADR)?
- Are corpora allow-listed with owners, classification, and refresh cadence?
- Is insufficient-context refusal enabled and logged?
- Can you export a signed evidence pack for the last 30 days of production queries?
Five-Layer Sovereign Stack
Experience → Governance → Inference → Knowledge → Infrastructure — each layer owns controls, artefacts, and failure modes.
Sovereign stacks fail when responsibilities blur: the UI invents policy, the model invents facts, or infrastructure silently phones home. The five-layer model keeps operator experience, policy enforcement, model execution, corpus truth, and platform substrate separable for design reviews and assurance mapping.
Web workbench, IDE/MCP plugins, admin console, report builder, and human-in-the-loop queues. Operators never need to leave the perimeter to draft, review, escalate, or dual-control a high-impact action.
- Components: chat/workbench UI, role-aware navigation, HITL approval inbox, session continuity, accessibility baseline.
- Artefacts: decision logs, dual-control records, operator identity (SSO/IdP), UI telemetry (non-PII where possible).
- Controls: RBAC/ABAC, step-up auth for privileged actions, watermarked exports, disable copy for classified sessions.
- Failure modes: shadow IT chatbots; unlogged “helpful” side channels; screenshots of sensitive completions.
The request envelope: policy gates, consent/purpose binding, Retrieval Guard, risk labels, and evidence exporter. Every completion is either allowed with evidence or refused with reason codes.
- Components: policy engine, Retrieval Guard, consent manager, prompt/response filters, audit bus, evidence packager.
- Artefacts: control maps, escalation trees, refusal logs, signed evidence packs, model/use-case register.
- Controls: allow/deny by use-case ID, PII redaction hooks, rate limits, mandatory citation coverage, human oversight flags for high-risk paths.
- Failure modes: policy only in documentation; logging after the fact; “trust the model” without refusal paths.
Local or VPC-hosted model serving: vLLM, LM Studio, or equivalent adapters; model router; prompt/tool orchestration. Cloud APIs are opt-in exceptions with ADR justification — never the silent default.
- Components: model server, router (size/latency/risk), structured output schemas, tool/function sandbox, embedding service (often co-located).
- Artefacts: model cards, residency attestations, quantization notes, eval harness results, version pins.
- Controls: network deny-by-default, secrets in HSM/KMS, prompt injection hardening, tool allow-lists, temperature ceilings for regulated tasks.
- Failure modes: untracked model swaps; egress via “temporary” debug proxies; floating
latesttags in production.
Document ingest, chunking, hybrid vector + sparse index, and corpus governance. This layer is the system’s claim to truth — not the parametric memory of the LLM.
- Components: parsers (PDF/DOCX/HTML), chunkers, embedding pipeline, hybrid retriever, ACL-aware filters, corpus versioning.
- Artefacts: source manifests, chunk IDs, classification tags, retention schedules, retrieval traces.
- Controls: allow-listed sources only, DLP on ingest, per-chunk ACL inheritance, stale-document quarantine.
- Failure modes: open-web crawl into the index; ACL bypass via embeddings; citations that do not support claims.
Compute, network, identity, observability, and update channels — from laptop Compose stacks to air-gapped Kubernetes with offline model delivery.
- Components: GPU pools, container orchestration, private DNS, SIEM/syslog, backup/DR, offline update media.
- Artefacts: SBOM, access reviews, network diagrams, change tickets, capacity plans.
- Controls: microsegmentation, egress deny-all, privileged access management, immutable audit stores.
- Failure modes: shared tenancy bleed; telemetry to public SaaS; unpatched GPU hosts; USB sprawl without media control.
Canonical data flow
Ingest and index prepare knowledge. Retrieve applies hybrid search + ACL. Generate only proceeds after Governance (including Retrieval Guard) authorises. Evidence packages the lineage for later assurance — not as an optional afterthought.
Layer RACI (simplified)
| Concern | Experience | Governance | Inference | Knowledge | Infra |
|---|---|---|---|---|---|
| User identity | A | C | I | I | R |
| Refusal & citations | C | A/R | C | C | I |
| Model version pin | I | C | A/R | I | C |
| Corpus ACL | C | C | I | A/R | C |
| Egress deny | I | C | C | I | A/R |
R = Responsible · A = Accountable · C = Consulted · I = Informed
Three Sovereign Modes
Choose topology by residual risk, not by cloud preference. Each mode reshapes the same five layers.
Comparison at a glance
| Dimension | On-prem GPU | Private VPC (no egress) | Air-gapped lab |
|---|---|---|---|
| Data leaving facility | No (by design) | No external internet; cloud region residency | Physically impossible without media |
| Elasticity | CapEx / capacity plan | High within tenancy | Fixed rack |
| Model updates | Controlled download → internal mirror | Private registry / private endpoints | Sneaker-net / approved media |
| Best fit | Banks, hospitals, large enterprises | Digital-native regulated orgs | Defence, intel, classified R&D |
| Primary residual risk | Ops maturity, patch lag | Sub-processor & shared fabric | Update latency, media control |
Dedicated GPU nodes behind the corporate firewall. Inference via vLLM or LM Studio; identity via LDAP/AD/Entra; knowledge from internal DMS. Prefer when prompts, embeddings, and logs must never leave the facility.
- Reference capacity: 2–8× A100/H100 (or equivalent), 5–20 TB corpus, <200 concurrent interactive users; batch eval on separate queues.
- Layer emphasis: Infrastructure (power, cooling, spare GPUs) and Inference (pin versions, canary eval before promote).
- Controls: DMZ-free design, jump hosts, SIEM forwarders, offline package mirrors for CUDA/containers.
- Anti-patterns: “temporary” NAT for Hugging Face; USB model drops without malware scan; unsegmented training and production nets.
Isolated cloud tenancy with private endpoints only. Hybrid RAG over SharePoint/Confluence/object stores via private links. Governance events to SIEM over private syslog/HTTPS.
- Reference profile: MeitY-empanelled or equivalent region; single-region DR; customer-managed keys where available.
- Layer emphasis: Governance (sub-processor register) and Knowledge (connector ACLs mirroring source systems).
- Controls: VPC endpoints, egress firewall default-deny, private container registry, continuous config drift detection.
- Anti-patterns: public load balancers “just for health checks”; shared logging SaaS without residency review; blind trust in provider AI APIs.
Physically isolated environment. Zero external API calls. HSM-backed keys. Model and OS updates via approved media with two-person integrity checks.
- Reference profile: single-site GPU rack; classified corpus only; offline evaluation suites co-located.
- Layer emphasis: Infrastructure (media control) and Experience (no external help links; local docs).
- Controls: Faraday / network diode patterns where required; write-once audit media; courier procedures for model weights.
- Anti-patterns: Wi-Fi “exceptions”; personal phones in the lab; undocumented USB tools.
Topology selection guide
- If the threat model includes nation-state exfiltration of prompts → prefer on-prem or air-gap.
- If the organisation already runs regulated workloads in a named cloud region with private connectivity → private VPC can be sovereign if egress is deny-all and sub-processors are listed.
- If the corpus is classified → air-gap; do not “simulate” with VPC alone.
- Always document the choice in an ADR linked to the AI use-case register.
Defensible RAG
Hybrid retrieval improves recall. The Retrieval Guard decides whether an answer may be released — with citations — or must be refused.
Problem statement
Parametric LLM knowledge is not an auditable source of truth for policy Q&A, clinical guidance, or regulatory interpretation. Unguarded RAG systems still hallucinate when chunks are weak, contradictory, or missing. Fluent wrong answers are worse than silence for regulated enterprises.
Pipeline (production)
Model fills gaps from weights. Auditors cannot verify provenance. Policy Q&A becomes liability; “confident wrong” answers ship to customers.
Every released claim is traceable to corpus chunks. Refusals are first-class events. Citation manifests travel with the evidence pack.
Threshold & policy knobs
- min_chunk_score — default 0.72; raise for legal/compliance Q&A; lower only with HITL review for exploratory research.
- min_citation_coverage — fraction of answer sentences that must map to a chunk (e.g. 0.9 for policy assistants).
- max_chunks — cap context to reduce prompt injection surface and cost.
- corpus_allowlist — hard deny retrieval outside registered knowledge bases.
- contradiction_policy — refuse or escalate when top chunks conflict on material facts.
Operational metrics
- Refusal rate by use case (sudden drops may mean threshold regression).
- Citation coverage distribution; orphan claims (should be ~0 in production).
- Time-to-evidence for a sampled query (SLA for assurance teams).
- False refusal rate from gold-set eval (balance safety vs. usability).
Implementation notes
Keep the guard in the Governance layer so Experience and Inference cannot bypass it. Expose reason codes to operators (“no chunks above threshold for corpus X”) so humans can fix knowledge gaps instead of disabling the guard. Pair with offline eval suites that include insufficient-context cases — not only happy-path demos.
Framework Mapping & Audit Export
Wrap every inference with policy, oversight, and exportable evidence — mapped to EU AI Act literacy, ISO/IEC 42001, and NIST AI RMF.
Governance that lives only in slide decks will not survive an examination. The sovereign stack treats governance as runtime controls plus durable evidence. This section maps platform capabilities to common frameworks and defines what an evidence pack must contain.
Control themes → platform hooks
Use-case risk class, transparency records, human oversight flags, logging for high-risk paths, prohibited-use screens.
AIMS roles, continual improvement loops, documented objectives, supplier/sub-processor controls, performance evaluation via metrics.
Govern / Map / Measure / Manage — register, risk maps, eval measures, incident/manage playbooks tied to platform telemetry.
Banking (model risk), public sector (residency), health (clinical safety), India DPDP (consent & purpose limitation).
Evidence pack contents (minimum)
- Use-case card — purpose, risk class, owners, approved corpora, model pin, topology ADR reference.
- Query lineage — anonymised or tokenised operator id, timestamp, use-case id, latency.
- Retrieval trace — chunk IDs, scores, threshold, allowlist version, refusal reason if any.
- Generation record — model name/version/quantization, decoding params, tool calls (if any).
- Citation manifest — mapping from claims/sentences to sources (document, page/section, chunk hash).
- Integrity — hash chain or signature over the pack; retention aligned to legal hold policy.
Human oversight patterns
- Always-on logging for limited-risk assistants; sampled HITL review.
- Mandatory HITL before external customer communication for high-risk classifications.
- Dual control for policy corpus publishes and threshold changes.
- Kill switch — disable generation globally or per use-case without redeploying the stack.
90-day assurance roadmap
- Days 1–30: Inventory use cases; pick topology; stand up five-layer reference with deny-all egress; enable Retrieval Guard defaults.
- Days 31–60: Gold-set evals (including refusal cases); wire SIEM; first signed evidence pack dry-run with internal audit.
- Days 61–90: Threshold tuning with HITL; board dashboard (refusal rate, coverage, incidents); schedule ISO 42001 / RMF control mapping workshop.
Request a tailored architecture workshop, or continue with WAIG assurance tools and Academy pathways.
Educational references to ISO, NIST, and EU materials are for literacy only. WAIG Foundation does not claim ownership of third-party standards. Proprietary UAAF operational criteria require written approval before operational use.