What Hardware Do You Need for Private AI Search?

One of the first questions we hear is some version of "do we need to buy a supercomputer for this?" The honest answer is almost always no. Private AI search has a few moving parts, and only one of them is demanding. This guide explains what each part needs and the deployment options available, so you can have a sensible conversation with your IT team before talking to any vendor.

The three jobs the hardware does

Every private AI search system, Athena included, does three things.

1. Indexing. It reads your documents, splits them into passages and records what each passage means so it can be found later. This is heavy the first time and light afterwards, because only new or changed documents need processing.
2. Searching. When someone asks a question, it finds the most relevant passages, ranks them and filters out anything that person is not allowed to see. This is quick and light.
3. Answering. A language model reads the best passages and writes an answer with references to the sources. This is the part that benefits most from a graphics card.

Storage, a database and the web interface sit alongside these, and they run comfortably on ordinary server hardware.

Do you need a GPU?

For smaller document collections, a standard server works. Answers will be slower than with a graphics card, but for a pilot or a small team that is often acceptable.

For larger deployments, or when many people will ask questions at once, we recommend an NVIDIA GPU with 8 GB or more of video memory. Athena detects the hardware it is given and selects the model, context length and batch size that suit it automatically, so you do not need a machine learning specialist to tune it.

The useful mental model is that the GPU sets how quickly answers arrive and how many people can be served at the same moment, while the size of your document collection mostly affects storage and the one-off indexing time.

Where it can run

There are three common shapes, and all of them keep your data inside an environment you control.

Your own Azure subscription. Our engineers install Athena into a subscription held in your name, using our fleet installer. Installs are typically live within an hour once your Entra app registration and GPU quota are ready. The data stays in your tenancy, not ours.

Your own servers. Athena can be self-hosted with Docker Compose, or with the Windows installer, on your own Linux or Windows servers with an NVIDIA GPU. This suits organisations with existing server rooms or strict on-premise policies.

Fully disconnected. For air-gapped environments, the Enterprise plan includes offline deployment and on-site installation. Once models are on the machine, no internet connection is needed.

What else to plan for

• Storage. Allow room for the search index and logs alongside your documents. Documents themselves stay where they are; Athena indexes them in place rather than copying them into a new store.
• Identity. Athena uses Microsoft Entra ID or your existing OIDC single sign-on, so you will need an app registration or equivalent.
• Network access to sources. The system needs read access to the shares, sites, mailboxes and repositories you want searchable, using connectors for platforms such as SharePoint, OneDrive, Teams, Confluence and Git.
• Monitoring. Monitoring, alerting and health checks are built in, so your team can see the system's state without building dashboards from scratch.

A practical way to size it

Start smaller than you think. Choose one team and one representative collection of documents, deploy on a single server with a suitable GPU, and measure how quickly answers arrive for your real questions. That tells you far more than a specification sheet, and it gives you evidence for the decision to expand.

We publish how we measure Athena, including answer quality and answer time, in the measurement note on the Athena page. Your own hardware and documents will give different results, which is exactly why a pilot is worth doing.

If you would like help sizing a deployment, book a demo and bring a rough idea of your document volume and how many people will use it. We will give you a straight answer.