There's something satisfying about running a language model on your own hardware. No API keys, no subscriptions, no data leaving your machine. Just you and the model.

Not long ago, running a frontier-class model locally meant compromises: small models, weak answers, or handing your data to a cloud service. That's over. Qwen 3.8 27B is the current state of the art among open-weight models you can run on consumer hardware — a dense ~27-billion-parameter model with vision and reasoning, a 256K-token context window, and an Apache 2.0 license. It doesn't need a data center, just one GPU in your own PC.

This is a follow-up to my earlier post, Running Local Models with llama.cpp on Windows & ROCm. That one got you running fast with AMD's pre-built binaries — but those binaries have a hidden cost: they're an older llama.cpp, compiled once for a whole family of GPUs, and they leave performance on the table. On my own benchmarks, building the latest llama.cpp from source, tuned to my exact GPU architecture, delivered noticeably higher tokens per second (TPS). This guide takes the from-source path.

Why build from source? (vs. AMD's pre-built binaries)

AMD ships ready-made llama.cpp binaries that "just work" — that's the route my previous post took. The catch:

• They're an older llama.cpp build — every release adds faster kernels and better memory handling, and you miss all of it.
• They're compiled once for a range of GPUs (e.g. gfx110X-gfx115X-gfx120X) instead of your exact architecture, so the device code can't be fully tuned to your card.
• Building the latest source with GPU_TARGETS set to your GPU's exact gcnArchName uses newer kernels compiled specifically for your GPU.
• My before/after benchmarks: the from-source build beat the pre-built binaries on tokens per second — most visible on generation-heavy workloads.

The cost is a few extra minutes and one build. Worth it for the most out of your GPU.

Why Qwen 3.8 27B?

• 100% local — your conversations never leave your machine
• State of the art — the dense open-weight model to beat at ~30B parameters
• Free & open — Apache 2.0 license, official GGUF releases from day one
• Huge context — a 256K-token window for long documents and deep conversations
• One consumer GPU — roughly 16–17 GB of VRAM at 4-bit quantization
• Drop-in successor — same size and architecture as Qwen 3.6 27B, so existing GGUF tooling and configuration just work

What you'll have when you're done

• The latest llama.cpp, built from source for your exact GPU — higher tokens per second than AMD's pre-built binaries, and your GPU doing the math instead of your CPU
• A local server speaking the OpenAI API — point any tool at http://localhost:8080
• The tools, source code, and model downloaded, ready to rebuild or swap anytime

The journey at a glance

• Step 1 — Install the tools (driver, ROCm, Visual Studio, Git, CMake, Ninja). You.
• Step 2 — Download the llama.cpp source code. You.
• Step 3 — Download the Qwen 3.8 27B model. You.
• Step 4 — Build, verify, and fix any issues. Your AI assistant — this part has a few sharp edges (GPU architecture, multi-GPU quirks), so it deserves its own detailed checklist: BUILD_HIP_WINDOWS.md.
• Step 5 — Start the server and talk to the model. Both of you.

This guide covers steps 1–3 (the downloads). Step 4 is a technical job with a few sharp edges, so it has its own detailed document that your AI assistant will work through — more on that in Part 4.

Part 1: Install the tools

None of these are optional — each one is needed for the build. Do them in order.

1.1 AMD graphics driver

Download from amd.com/support (pick your GPU). Install, then restart your PC.

1.2 ROCm (AMD's GPU compute kit)

Go to amd.com/products/software/rocm and download the Windows installer (it's big — several GB). Run the installer. You only need ROCm Runtime, HIP, and rocBLAS — you can skip PyTorch, TensorFlow, profiling tools, MIOpen, and everything else. It lands in C:\Program Files\AMD\ROCm\<version> — remember the version number.

Note: ROCm installs a driver component. If your antivirus blocks it, add an exception for C:\Program Files\AMD\ROCm and restart.

1.3 Visual Studio (Community is free)

Download from visualstudio.microsoft.com. In the installer's workload list, tick "Desktop development with C++". You don't need anything else.

1.4 Git

Download from git-scm.com and install with the default options.

1.5 CMake

Download the Windows x64 Installer from cmake.org. During install, tick "Add CMake to the system PATH".

1.6 Ninja

Download ninja-win.zip from github.com/ninja-build/ninja/releases (get the newest release), or run winget install Ninja-build.Ninja. If you downloaded the zip: unzip it, put ninja.exe in a folder that's on your PATH (e.g. C:\Ninja), and add that folder to your PATH in System Properties → Environment Variables.

Part 2: Get the source code

llama.cpp is the engine that runs GGUF models like ours. Open PowerShell (or Git Bash) and run:

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp

The build itself happens later, with the AI's help — for now, downloading is enough.

Part 3: Get the model: Qwen 3.8 27B

1. Download the 4-bit GGUF from either of these (both are official community quants):
   • lmstudio-community/Qwen3.8-27B-GGUF — grab the file Qwen3.8-27B-Q4_K_M.gguf (LM Studio community, built with llama.cpp)
   • or unsloth/Qwen3.8-27B-GGUF — Unsloth's dynamic 4-bit GGUFs
2. Check your VRAM first (Task Manager → Performance → GPU). A 4-bit 27B needs 16–17 GB — a 16 GB+ GPU is required. Less VRAM? Download Q3_K_M instead, or pick a smaller model; the steps stay the same.
3. Download the .gguf file to a folder you'll remember, e.g. D:\models\. Keep the full path handy — you'll need it later.

Part 4: Hand over to the AI

Your PC is now ready. Point your AI assistant at the technical checklist — BUILD_HIP_WINDOWS.md — it will:

• Find your ROCm install and check your GPU's exact architecture
• Configure and build llama.cpp with the ROCm backend
• Verify the GPU is actually being used
• Fix anything that goes wrong

One thing to tell the AI before it starts: does your PC also have integrated graphics? (Windows shows a second GPU like "AMD Radeon(TM) Graphics" next to your real one.) If yes, say so — it changes how the setup is done and avoids a very common startup crash.

Wrapping Up

Running a frontier-class model locally isn't just practical — with the from-source build, it's the fastest option available on your hardware. No cloud, no subscriptions, no data leaving your machine; just the latest kernels, tuned to your exact GPU.

If you followed the previous post with AMD's pre-built binaries, you already know how far this goes. Now imagine the same setup with noticeably more tokens per second. That's what building from source buys you.

The model we used is just the start. The GGUF ecosystem has hundreds of models on Hugging Face — pick one that fits your VRAM and start experimenting. Welcome to local AI.