Skip to main content

AI architecture

AI touches almost every worksheet. To avoid fifteen places each making their own call, everything runs through one gateway.

One gateway for all generation​

Every AI call goes through a single server-side gateway. It handles:

  • Two gates before every call. The first checks whether the user may write on this canvas: a reader cannot start an AI function, not even from outside the screen, and the attempt costs nothing. The second checks whether the organisation still has budget. Both look at the same canvas, and a call that works on a canvas cannot quietly fall back on the organisation's own credit.
  • Prompt resolution on the server. Calling code never passes prompt text, only a key from a fixed list. The matching prompt is looked up on the server: a tenant-specific variant if there is one, otherwise the default version. A client therefore cannot give the model instructions of its own.
  • Model choice per type of task. Heavy reasoning tasks get the strongest model; mechanical work such as deduplicating, clustering and labelling gets a faster, cheaper model. This assignment is held centrally, not spread across the code, and can be set in administration per class and per AI button: model, thinking depth and time budget. For a new model, the administrator adds it to the list with its price, compares it on the test environment against the old model with a test set, and switches the class or button over; if the new model returns an error, that button falls back to the previous one. A nightly check compares the model list with the provider's.
  • Two routes, one setting. A call goes directly to the model provider or through the Vercel AI gateway; this is a platform setting. Alongside the route there is a region setting: worldwide or EU. With EU, the call goes through the gateway only, to providers that process in the EU, and the platform refuses the call if that cannot be honoured. Route and region are recorded with every call in the usage log.
  • Resilience. Timeouts per type of task, and retries only for temporary failures on the provider's side, never after a timeout, because that only guarantees a second timeout.
  • Fallback on refusal. If a model refuses a call before any output exists, or is overloaded, exactly one new attempt follows on another model that is active, only if there is still time in the call's budget. After a timeout or a cut-off no second attempt follows, and never on a model that has been phased out.
  • Budget control. Before every call, the tenant's usage budget is checked.
  • Usage logging. Model, tokens and cost per call are recorded for insight and cost allocation.

The grounding chain​

This is how a substantiated AI statement comes about:

  1. Chunking. An uploaded document is split into coherent blocks of about 400 tokens, roughly 300 Dutch words, respecting sentence and heading boundaries. Smaller fragments were used earlier; they proved too fragmented for useful synthesis.
  2. Capturing meaning. Each block gets a numerical representation of its meaning, stored in the same database.
  3. Search, limited by count. For an AI question, the eight best-matching blocks are retrieved by default, supplemented by a light search branch for names and figures. For extraction tasks the count grows with a character budget (forty thousand characters for extraction). A document the consultant has labelled for a worksheet goes along in full for such a task, as long as it fits in the budget.
  4. Source material only. Searching takes place over a shielded view that contains only supplied source material. Previously generated AI output can never come back as evidence, enforced by a dedicated contract test.
  5. Verified reference. The model must refer to the fragments it actually received. The server checks every reference against that set. A reference that was not in it is rejected or flagged.
  6. Recording at the statement. File name, page and fragment are stored along with the statement, so the source reference stays readable when the document is re-indexed later.

Documents have versions. When a new version is uploaded, the user sees what changed, and for each part that still rests on the previous version the AI checks whether it still holds and proposes updated text, which the consultant approves.

Step 5 is the difference between "ask the model for sources" and a verifiable evidence chain.

Latency and reliability​

Serverless functions have a hard upper limit of 60 seconds. When we switched to a stronger but slower model for quality reasons, heavy generations hit that limit: the function was aborted and the user got an incomprehensible error.

This was solved structurally rather than by going back to a weaker model:

  • Streaming. The answer arrives while it is being written. This removes the timeout wall and lets the user see progress.
  • Line-by-line output. Generated lists arrive line by line and are stored per item; the user can accept, reject or edit a proposal while the rest is still coming in. If a generation is cut off anyway, the result is a usable partial result instead of an unusable half message, and what had already been written counts toward the cost. On the IT worksheet, the generation stops on the server as soon as the user leaves the section.
  • Background tasks. Work that does not fit in one function runs as a task with its own status and progress. The user can navigate away and come back; an analysis of the capability model keeps running on the server and is ready, with a red dot, on return.
  • Model choice by task weight and reuse of fixed prompt parts, which saves both time and cost.

Budget control stays active in all these paths, even when calls run in parallel. When the credit is used up, the user reads that in the language of the canvas, instead of a technical error message.