Guide: running local AI in the company

Local AI in the company: three ways.

Cloud or your own server? There is a third way — and for each an honest answer as to when it fits.

Anyone who wants to use company knowledge with AI soon faces the question of where the answers are computed. This page sets the three ways of operating side by side: who operates it, where the data lives, what your own IT carries.

THREE WAYS SIDE BY SIDE
WHO OPERATES WHAT
WHEN EACH ONE FITS

AT A GLANCE

Local AI: the essentials first.

KEY POINTS

Local AI means the language model and search compute on your own network instead of in a provider's data centre.
There are three ways to run it: a provider's EU cloud, a self-run stack of open models, or a device on your own network that the provider runs.
The model is only one part. Search, connecting the sources, permissions, updates, backup and a plan for outages belong to it too.
Local does not automatically mean compliant with data protection law: purpose, permissions, deletion and co-determination remain the company's job.
The biggest costs of a self-run stack are rarely on the hardware invoice — they are in operation.

THE QUESTION

Can a company use the cloud — or does it have to run an open AI stack itself?

The question is often asked as either-or: either the data goes into a cloud, or your own IT builds a stack of open models and runs it itself. Both work — and both have a price: with the cloud, the data leaves the building; with your own stack, everything rests on your own IT, from the hardware and the model choice to the security hole at the weekend.

In between lies a third way: a device in your own network on which the language model, search and knowledge run locally, but which the vendor operates — with updates, maintenance and replacement. The data stays in the building, the work stays with the vendor.

Which way fits depends less on technology than on three questions: may the data leave the building? Is there anyone in-house who wants to and can run AI? And who answers for it when something fails?

TERMS

What is local AI — and what does on-premise mean?

Local AI is AI that computes on your own network: on a server or device in the building instead of in a provider's data centre. Several terms come up for this, and they do not all mean quite the same.

LOCAL AIOn your own networkThe language model and search run on hardware in the building. The term does not say who runs it.

ON-PREMISEOn your own premisesThe classic IT term: the software sits with the customer. It usually also implies that your own IT runs it.

LOCAL LLMA language model on your own hardwareJust the model — no search across company knowledge, no permissions, no upkeep. Good for trying out, not yet a solution for a team.

SOVEREIGN AIUnder your own controlAbout dependency: who controls data, model and operation, and can you switch? Local is one way there, not the only one.

THE THREE WAYS

Who operates it, where the data lives, what your IT carries.

The same four questions for each way.

WAY 1A vendor's EU cloudThe vendor runs the language model and search in its data centre in the EU.

✓Operation: the vendor
✓Data: in the vendor's data centre in the EU
✓Your IT: accounts, permissions, integration
✓Fits when: cloud services are allowed and you want to start quickly

WAY 2Self-operated with open modelsYour IT builds a stack of open models on its own hardware and runs it.

✓Operation: your IT
✓Data: in your own network
✓Your IT: hardware, model choice, updates, security, failure plan
✓Fits when: you have your own AI team and want every component in your own hands

WAY 3Device in your own network, operated by the vendorLanguage model, search and knowledge run locally on a device the vendor operates.

✓Operation: the vendor — updates, maintenance, replacement
✓Data: in your own network, backed up to your storage
✓Your IT: rack space, power, one outbound connection
✓Fits when: the data should stay in the building without an AI team of your own

Explicio offers way 1 — the EU cloud on hardware operated by Explicio in Germany — and way 3, the Explicio Box.

WHAT IT TAKES

A language model on its own is not yet AI for the company.

What else has to run so a team can work with company knowledge — whoever runs it.

LANGUAGE MODELWriting the answerThe model writes the answer. Which model it is decides language, quality and hardware needs.

SEARCHFinding the right knowledgeBefore the model answers, the right document has to be found. Without good search, a model is more likely to make things up than to cite.

SOURCESConnecting repositoriesManuals, drives, wikis, mailboxes: read in and kept current without anyone filing everything again.

PERMISSIONSOnly see what you mayAn answer may only come from documents the person asking would be allowed to open.

OPERATIONUpdates, security, monitoringClosing vulnerabilities, switching models, keeping an eye on storage and load — continuously, not once.

BACKUPA plan for outagesIf the device fails, you need a backup and a fallback. Otherwise the knowledge is gone or the team stands still.

HARDWARE

What hardware does local AI need?

For a trial on a single workstation, a current computer with plenty of memory is enough: small open models run there, slowly but well enough to try out. For a team, that is too little. Several questions at once, long documents and images need a machine with a graphics processor and large, fast memory.

Model size drives the requirement: the larger the model, the more memory it needs — and the better its answers in specialist German or English usually are. Add power, cooling, a place on the network and spare parts for when something fails.

FOR A TRIALA workstationOne small model, one person, no connection to company data. Shows what local AI feels like — not how it runs in operation.

FOR THE TEAMAn AI server or a managed deviceGraphics processor, large memory, backup, monitoring. If you do not want to build that yourself, have it run for you — that is way 3.

MODELS

Local AI models: what open models mean for companies.

Open models publish their weights for download; you can run them on your own hardware. That is what makes local AI possible — and it brings decisions in-house: which model, in which size, under which licence.

Three things are worth checking before a model is used: the licence terms for commercial use, the quality in your own language and terminology, and who maintains the model when a better version comes out. Anyone who retrains a model themselves should also check whether this makes them subject to the obligations of a model provider under the EU AI Act.

LICENCECheck commercial useOpen does not mean free of conditions. Some licences restrict the use or the size of the company.

SWITCHINGModels age quicklyEvery few months a better model appears. If you run it yourself, you test and switch yourself; at Explicio we choose and name the model in the transparency sheet.

DATA PROTECTION AND CO-DETERMINATION

Local does not automatically mean compliant.

Local AI settles one question — where the data is. Others remain, wherever the computing happens.

Local settles

Computing happens on your own network, not at a provider
No third-party data centre if your own IT runs it
Network boundaries and access stay in your hands

Still to settle

Purpose and legal basis of the processing
Who may see which answers
Access and deletion for personal data
Co-determination by the works council
A data processing agreement if a provider runs the device

What a works agreement on AI should cover is on our page for works councils and data protection.

COST

What does local AI cost?

With a self-run stack, only part of the cost appears on the hardware invoice.

On the invoice

Server or device with a graphics processor
Storage for the backup
Power and cooling
Licences for the surrounding software

On no invoice

Working time for setup, updates and security
Testing when a new model comes out
Standby when it fails at the weekend
Replacement and recovery after a defect

With the Explicio Box, operation, updates, replacement in case of a defect and usage are part of the Box service — no base fee, no usage charges; prices can be checked on the pricing page.

CHECK

Three questions that decide every way.

Ask them of every vendor — and of your own IT if you are considering way 2.

QUESTION 101Who installs updates?With your own stack, your IT; otherwise the vendor. With the Explicio Box, updates are signed and can be rolled back if something does not fit.
QUESTION 202Who chooses the model?With your own stack, you do — with all the consequences for maintenance and review. At Explicio we choose it and name it in the transparency sheet for customers.
QUESTION 303What happens when it fails?With your own stack, your own plan applies. With the Explicio Box we replace a defective device as part of the Box service; the replacement restores your knowledge from your own backup.

WAY 3 IN DETAIL

What the Explicio Box is — and what it is not.

A device from Explicio in the customer's network, operated as part of the Box service.

The Box is …

a device from Explicio in your network that we operate
local: language model, image understanding, search and knowledge graph run on the Box, without falling back to a cloud
backed up: encrypted to your own network storage, with your key
reachable for us only for remote maintenance that you approve

The Box is not …

a server your IT has to maintain itself
a measuring or sensor device on the machine
an actuator — it controls no machine
a device you buy and operate on your own

Accounts, contracts and invoices stay in the cloud; bringing in knowledge, searching and answering run on the Box.

HONESTLY

When way 1 or way 2 is the right one.

The third way is not the best for everyone. These cases speak for one of the others.

Way 1 (EU cloud) fits if …

cloud services are allowed in your company
you want to start right away without a device and setup
several sites without a shared network work together

Way 2 (self-operated) fits if …

you have your own AI team and want to keep it
every component, including the model, has to be in your own hands
you want to carry updates, security and failures yourself

Way 3 fits if the data must not leave the building and yet nobody in-house should run AI.

QUESTIONS

What people often ask.

What is local AI?

AI where the language model and search run in your own network — on a server or device in the building instead of in a vendor's data centre. The data does not leave the building for it.

Do I need my own AI department for local AI?

With a self-operated stack, yes: hardware, model choice, updates and security then lie with your IT. With a device the vendor operates, no — with the Explicio Box, your IT provides rack space, power and an outbound connection.

Does the Explicio Box run without the internet?

Bringing in knowledge, searching and answering run completely locally. The Box needs an outbound connection for signed updates, operating data, approvals and the messenger message channel; without it, no remote maintenance is possible.

Source: Box page

Which model runs on the Box?

Everything Explicio needs for answers, image understanding and search runs on the Box. Which language model is used, we name in the transparency sheet for customers.

Can I move from the cloud to the Box?

Yes, from the EU cloud to the Box and back. The fees for setup and the move are on the pricing page.

Source: pricing page

What happens if the Box is defective?

We replace it, as part of the Box service. A pre-configured replacement takes over your settings, and your knowledge comes back from your own backup on your network storage.

Source: Box page

On data protection, hardware and private cloud.

Is local AI GDPR-compliant?

Run locally, computing happens on your own network rather than at a provider — that makes a lot easier. Purpose, legal basis, permissions, deletion and works council co-determination still need to be settled, even with a device on your own network.

More: page for works councils and data protection

What hardware do I need for local AI?

For a trial, a computer with plenty of memory and a small model. For a team, a server with a graphics processor and large memory, plus backup and monitoring — or a device the provider runs, such as the Explicio Box.

What is the difference between local AI and a private cloud?

With local AI, the hardware sits on your own network. A private cloud is a separate area in a provider's data centre, reserved for one company — the data then lies there, not in the building.

What happens after the 48 months?

Well before the end of the term we come to you with an offer for a new Box. The switch is plug and play: the new Box arrives preconfigured, takes over your settings and restores your knowledge from your own backup. If you do not want a new Box, you move to the EU cloud or have the data deleted.

Box page

NEXT STEP

Settle the right way in one conversation.

Test for 30 days in the EU cloud first — or request a quote for the Box straight away. We will also tell you if way 2 suits you better.

Request a Box quote Request trial access

Box: device, operation, backup
Checklist: eight questions for any platform
Pricing: you can recalculate it