Custom built AI Server Setup

Building Raymond

Raymond is our local AI assistant — but it doesn't run on the server itself. It's Ollama split across two GPU gaming rigs, fronted by Open WebUI and a wake-on-demand dispatcher on the headless server that ties it all together. Here's how it's built and how it works.

The hardware

All three machines share the same board and CPU (ASUS ROG STRIX X870-A GAMING WIFI, Ryzen 5 7600X, 32GB DDR5, Intel I226-V networking) — the difference is what each one is for.

Why the server doesn't do the thinking

The server has no dedicated GPU, so it was never a candidate for running inference itself. Both desktops have RTX 5060 Ti cards sitting idle most of the day, so the split is deliberate: inference happens on the workstations, and the server only coordinates — waking a workstation, routing the request to it, and serving the chat interface.

Sharding one model across both GPUs as a single cluster was considered and passed over in favor of two fully independent Ollama instances, load-balanced by the dispatcher. Workstation use has no fixed schedule, so a wake-on-demand proxy fits better than any fixed sleep/wake timer.

Architecture

[ Browser ]  --ask Raymond-->  [ Open WebUI  :3000 ]
                                        |
                                        v
                               [ Dispatcher  :11500 ]
                                 /michael      /suzanna
                                    |               |
                     (WoL if asleep)|               |(WoL if asleep)
                                    v               v
                      [ michael-desktop ]   [ suzanna-desktop ]
                        Ollama  :11434         Ollama  :11434
                       RTX 5060 Ti 16GB       RTX 5060 Ti 16GB

A request comes into Open WebUI, which is just the chat front end. It hands off to the dispatcher, which checks whether the target workstation is awake, sends a Wake-on-LAN magic packet if it isn't, and proxies the request through once it's up.

Ports at a glance

Build log

1. Ollama installed on michael-desktop — first workstation stood up, using its RTX 5060 Ti. Tested with qwen2.5:14b before anything else was wired up.

2. Open WebUI deployed on the server — Docker container brought up at :3000, initially pointed straight at michael-desktop's Ollama instance.

3. Fixed a network-binding snag — Open WebUI couldn't reach Ollama. Cause was Ollama listening on localhost only; fixed with OLLAMA_HOST=0.0.0.0 via a systemd override. (ufw was checked and confirmed inactive — not the culprit.)

4. Grew the model lineup — michael-desktop's Ollama now also carries qwen3.5:9b, qwen3.5:4b, and qwen3.5:2b alongside the original 14b.

5. Confirmed suzanna-desktop as a matching second node — same RTX 5060 Ti, 16GB VRAM, built identically to michael-desktop.

6. Built the dispatcher and load-balanced routing — the server got a dispatcher on :11500 exposing /michael and /suzanna, each waking that workstation over Wake-on-LAN if asleep and proxying to its Ollama instance.

7. Named it — Raymond — voice replies go out through Open WebUI's built-in text-to-speech, using the browser's Web Speech API.

Not built yet

Obsidian integration is still just an idea: either feed notes into Open WebUI's Knowledge/RAG feature, or use an Obsidian plugin that queries Ollama directly. No decision made yet on which way to go.

Software used

• Ollama — local model runtime, loads and serves the models on each workstation's GPU. (source)

• Open WebUI — self-hosted chat front end for Raymond, running in Docker on the server; also drives the TTS voice. (docs / source)

• Docker — container runtime hosting Open WebUI.

• Wake-on-LAN — protocol + wakeonlan tool the dispatcher uses to wake a sleeping workstation. (Debian Wiki)

• systemd — used for the OLLAMA_HOST override that got Ollama listening on the network.

• UFW — firewall checked (and ruled out) while debugging the WebUI ↔ Ollama connection.

• Web Speech API — browser TTS engine behind Raymond's voice output.

Build notes — will update this thread as the setup grows.

Example run of Raymond while discussing my cancer story and creation of Titanium