How to choose a local AI server
Last updated
Choose a local AI server by the largest model you need, how many people use it at once and how much setup you want to do. For models up to about 32B and a small team, a 24 GB CUDA box like Olares One fits.
Olares One → Local AI server → Ollama · vLLM · Open WebUI → Private AI on your own network
Criteria
- Memory for models: 24 GB of VRAM holds dense models up to about 32B at 4-bit; a 70B model needs about 40 GB (approximate rule).
- Concurrent users: compare per-request speed with several requests running, not single-request peaks.
- Software stack: CUDA runs vLLM, Ollama and most open-source image and video tools without porting.
- Server features: per-person accounts, remote access, backups and app installs.
- Noise, size and power for the room the server will live in.
- Total cost: hardware, drive, OS, electricity and setup time.
Machines compared
| Machine | Price | Memory for models | Memory type | GPU | CUDA |
|---|---|---|---|---|---|
| Olares One | $4,999 · €4,599.95 | 24 GB | VRAM | RTX 5090 Mobile | yes |
| NVIDIA DGX Spark | $6,950 | 128 GB | Unified, 273 GB/s | NVIDIA GB10 | yes |
| ASUS Ascent GX10 | $5,999 | 128 GB | Unified | NVIDIA GB10 | yes |
| Apple Mac Studio (M5 Ultra) | $5,499 · €6,599 | 96 GB | Unified, 1.2 TB/s | M5 Ultra, 64-core GPU | no |
| Apple Mac Studio (M5 Max) | $2,499 · €2,999 | 36 GB | Unified, 460 GB/s | M5 Max, 32-core GPU | no |
| Apple Mac mini (M5 Pro, 64GB) | $3,199 · €3,649 | 64 GB | Unified, 307 GB/s | M5 Pro, 20-core GPU | no |
| Minisforum MS-S1 Max | $3,959 | 128 GB | Shared (unified) | Radeon 8060S | no |
| GMKtec EVO-X2 (Ryzen AI Max+ 395, 128GB) | $3,499.99 | 128 GB | Shared (unified) | Radeon 8060S | no |
| Framework Desktop (Ryzen AI Max+ 395, 128GB) | $3,449 | 128 GB | Shared (unified) | Radeon 8060S | no |
| Beelink GTR9 Pro (Ryzen AI Max+ 395, 128GB) | $4,199 | 128 GB | Shared (unified) | Radeon 8060S | no |
| Used RTX 3090 tower build | $3,800 | 24 GB | GDDR6X VRAM, 936 GB/s | RTX 3090 (used) | yes |
| NVIDIA GeForce RTX 5090 (desktop card) | $1,999 | 32 GB | GDDR7 VRAM | RTX 5090 desktop | yes |
Prices and specs read on the makers' pages on 12-13 November 2026; each name links to its source.
How much memory do your models need?
Start from the largest model you will run every day. By an approximate rule, a 4-bit model needs roughly 0.5-0.6 GB per billion parameters plus a few GB for context. That puts 32B at about 19-20 GB, which fits a 24 GB GPU, and 70B at about 40 GB, which does not.
| Model (4-bit) | Approximate memory | Fits on |
|---|---|---|
| 8B | ~5 GB | Any GPU server |
| 14B | ~9 GB | Any GPU server |
| 32B | ~19-20 GB | 24 GB GPU or larger |
| 70B | ~40 GB | 128 GB unified memory, Mac Studio M5 Ultra, multi-GPU tower |
| 120B | ~65 GB | 128 GB unified memory or more |
How many people will use it at once?
Count the people who send requests at the same time, not the headcount. One person can live with any machine that fits the model. A team needs separate accounts, a model runner that batches parallel requests, and someone who owns updates and backups.
That is why we favor a ready-made system for offices without IT staff. If someone on the team enjoys running Linux, a self-built tower costs less in parts and more in hours. The cost guide sets the two side by side.
Questions & answers
Should I buy for today's model or next year's?
Buy for the largest model you will run daily, with room for context. If you expect to move to 70B models, a 24 GB server will not get you there.
Is a gaming PC good enough as a local AI server?
Often, for one person. A gaming PC with a 24 GB card runs models up to about 32B at 4-bit by the approximate rule. For a shared server, add accounts, a VPN and backups, or choose a system that includes them.
Sources for this entry
- Olares One specifications (Olares docs) (Company-published)
- NVIDIA DGX Spark: maker's page (Maker-published)
- ASUS Ascent GX10: maker's page (Maker-published)
- Apple Mac Studio (M5 Ultra): maker's page (Maker-published)
- Apple Mac mini (M5 Pro, 64GB): maker's page (Maker-published)
- Minisforum MS-S1 Max: maker's page (Maker-published)
- GMKtec EVO-X2 (Ryzen AI Max+ 395, 128GB): maker's page (Maker-published)
- Framework Desktop (Ryzen AI Max+ 395, 128GB): maker's page (Maker-published)
- Beelink GTR9 Pro (Ryzen AI Max+ 395, 128GB): maker's page (Maker-published)
- Used RTX 3090 tower build: maker's page (Maker-published)
- NVIDIA GeForce RTX 5090 (desktop card): maker's page (Maker-published)
Continue this reading path
How the related websites connect
Cite this entry
Best Local AI Server. “How to choose a local AI server.” Content reviewed 13 November 2026.
https://bestlocalaiserver.com/choosing/