llama.cpp runs GGUF models directly on your AMD GPU when built against ROCm. On Linux you often want a source build so you can match your exact GPUs and rebuild on a whim. This guide walks through the whole thing in order, from installing the ROCm pieces CMake needs to running a model across multiple GPUs.

The setup I'm using. An AMD Ryzen 7 7800X3D CPU with two discrete Radeons — an RX 9070 XT (16 GB) and an AI PRO R9700 (32 GB), both RDNA 4 (gfx1201) — plus the CPU's integrated graphics (gfx1036). Two architectures on one machine, which step 6 addresses.

Step 1: Install ROCm

Install ROCm with the amdgpu-install script, selecting ROCm plus graphics:

sudo amdgpu-install --usecase=rocm,graphics

Reboot afterwards, then confirm your GPUs are visible:

rocminfo

Step 2: Install the ROCm Development Packages

Building from source needs the development packages, which ship the CMake config files that llama.cpp's configure step requires (hip-config.cmake, hipblas-config.cmake, rocblas-config.cmake). They are separate from the runtime install. On ROCm 7.14's packaging install them with:

sudo apt install \
    amdrocm-runtime-dev7.14 \
    amdrocm-blas-dev7.14 \
    amdrocm-core-dev7.14-gfx1201 \
    amdrocm-hipblas-common-dev7.14

A note on the packages:

• amdrocm-runtime-dev7.14 — the HIP runtime development files, including hip-config.cmake and the hip-lang package.
• amdrocm-blas-dev7.14 — the hipBLAS and rocBLAS dev files that provide the BLAS CMake configs.
• amdrocm-core-dev7.14-gfx1201 — architecture-specific core dev files; pick the target that matches your GPU.
• amdrocm-hipblas-common-dev7.14 — shared header-only hipBLAS files.

To double-check the CMake files are present before configuring, run:

find /opt/rocm -name "hip-config.cmake" -o -name "hipblas-config.cmake" -o -name "rocblas-config.cmake"

Step 3: Configure with CMake

Now configure the build. Because HIP code must be compiled by ROCm's bundled clang — not the system gcc — set both compilers explicitly. Pass GPU_TARGETS=gfx1201 to build kernels for your discrete RDNA 4 GPUs.

cmake -S . -B build \
  -DGGML_HIP=ON \
  -DGPU_TARGETS=gfx1201 \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_C_COMPILER=/opt/rocm/llvm/bin/clang \
  -DCMAKE_CXX_COMPILER=/opt/rocm/llvm/bin/clang++

If you want one binary that also runs on the integrated GPU, add it as a second target:

  -DGPU_TARGETS="gfx1201;gfx1036"

That costs a longer build and a bigger binary. I built only for gfx1201 and hid the iGPU at runtime instead (step 7).

Step 4: Build

With configure done, build — everything in parallel:

cmake --build build --config Release -j$(nproc)

This compiles the HIP kernels under ggml/src/ggml-hip for each target. On an 8-core 7800X3D it takes a while; it finishes on its own. The llama-cli, llama-server, and llama-bench binaries end up in build/bin.

Step 5: Set the Runtime Library Path

On this ROCm release the runtime libraries live under /opt/rocm/core-7.14/lib — a path the system loader doesn't know about by default. Point it there so the binaries can find their dependencies:

export LD_LIBRARY_PATH=/opt/rocm/core-7.14/lib:$LD_LIBRARY_PATH

Confirm everything resolves by checking your binary against the dynamic libraries:

ldd build/bin/llama-cli

Every libhipblas, librocblas, and libamdhip64 should resolve to a real path — none should read "not found".

Step 6: Verify the GPUs

llama.cpp reports what it sees:

./build/bin/llama-cli --list-devices
Available devices:
  ROCm0: AMD Radeon RX 9070 XT   (16304 MiB)
  ROCm1: AMD Radeon AI PRO R9700 (32624 MiB)
  ROCm2: AMD Radeon Graphics     (14953 MiB)

Step 7: Exclude the Integrated GPU

The third entry is the CPU's integrated graphics. The discrete cards are gfx1201; the iGPU is gfx1036 — a different architecture with no matching kernels in this build. Keeping it out of the process entirely is simplest; it also avoids the classic multiple-architecture conflict on machines with both an iGPU and a discrete AMD GPU. Hide it from HIP:

export HIP_VISIBLE_DEVICES=0,1

Now device list shows only the two discrete cards:

./build/bin/llama-cli --list-devices
ROCm0: AMD Radeon RX 9070 XT
ROCm1: AMD Radeon AI PRO R9700

Step 8: Set Up ~/.bashrc

Exporting those two variables each session gets tiring. Add them to ~/.bashrc so every new terminal is ready to go:

# llama.cpp + ROCm 7.14 environment
export LD_LIBRARY_PATH=/opt/rocm/core-7.14/lib:$LD_LIBRARY_PATH
export HIP_VISIBLE_DEVICES=0,1

# handy aliases
alias llama-cli='$HOME/source/repos/llama.cpp/build/bin/llama-cli'
alias llama-server='$HOME/source/repos/llama.cpp/build/bin/llama-server'
alias llama-bench='$HOME/source/repos/llama.cpp/build/bin/llama-bench'

Open a fresh terminal and everything is there.

Step 9: Run a Model

Run with a local GGUF model, using -ngl 99 to push every layer to the GPU:

llama-cli -m path/to/model.gguf -ngl 99 -p "Hello" --no-display-prompt

To use both discrete cards together, layer split is the simplest reliable option:

llama-cli -m path/to/model.gguf -ngl 99 -sm layer

For tensor split with good throughput, rebuild with -DGGML_HIP_RCCL=ON so the two GPUs can communicate directly instead of through system memory.

Wrapping Up

From a fresh install to a running multi-GPU llama.cpp, it's these eight commands' worth of steps:

• Install ROCm, then the development packages.
• Configure with ROCm's clang and GPU_TARGETS=gfx1201.
• Build, and export LD_LIBRARY_PATH to the ROCm lib dir.
• Hide the integrated GPU with HIP_VISIBLE_DEVICES.
• Pin it all in ~/.bashrc so a fresh terminal is ready immediately.

Once it's set up it just runs. I'll keep posting about squeezing tokens out of these Radeon cards.

In my previous post, Run Qwen 3.8 27B on Windows with ROCm, I set up Qwen 3.8 27B on an AMD GPU — building llama.cpp from source for maximum throughput. The setup works. The question this post answers is a different one: which quantization, and which server flags, should you actually run?

I benchmarked two 27B quantizations — unsloth's Q4_K_M and AtomicChat's AD-IQ4_XS-IQ3_S — each in two configurations: with MTP2 (multi-token-prediction drafting, n=2) and without MTP. Four configurations total, all on the same 16 GB card, all driven through the same real coding task.

Test setup

• GPU: AMD Radeon RX 9070 XT, 16 GB VRAM — dedicated to AI (a separate GPU handles display)
• CPU: AMD Ryzen 7 7800X3D
• RAM: DDR5
• Runtime: llama.cpp built from source with the ROCm backend (per the setup post), all layers offloaded to the GPU (-ngl 99)

The workload. Every configuration ran the same real task: a coding agent (the pi harness) pointed at a simple Vite.js repository — my profile website — with the prompt "explain this repo to me (all files)". That's a realistic, context-heavy workload: it reads a lot of files, holds them in context, and produces a long structured explanation.

One observation before the results: in all four configurations, the llama-server process accounted for around 13 GB of DDR5 system RAM — that's its mmap. llama.cpp memory-maps the GGUF file, so the OS counts the mapped model file against system RAM even while the server sits idle. It's the model file showing up in the RAM meter, not a leak — but it's worth knowing before you plan headroom on a machine where the GPU is also doing other work.

Results at a glance

If you only read one table in this post, make it this one — everything else below is detail.

All throughput figures are averages over the run, not exact single-sample numbers. The highlighted row is the fastest configuration.

How the tests were configured

All four runs share the same base: flash attention on (-fa on), automatic memory fitting disabled (--fit off), a single slot (--parallel 1), q4_0 KV cache for both K and V, and generation sampling of --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0. The KV cache size differed between quantizations (32K vs. 64K), and the MTP runs add --spec-type draft-mtp --spec-draft-n-max 2 with a q4_0 draft cache. Full commands per configuration are below.

unsloth Qwen3.8-27B-Q4_K_M.gguf

The most common 4-bit quantization in the ecosystem — a solid reference point for quality.

With MTP2, flash attention, single slot, no memory fitting

llama-server.exe -m D:\source\llms\unsloth\Qwen3.8-27B-Q4_K_M.gguf -ngl 99 --port 1234 -c 32096 --cache-type-k q4_0 --cache-type-v q4_0 -fa on --fit off --parallel 1 --spec-type draft-mtp --spec-draft-n-max 2 --spec-draft-type-k q4_0 --spec-draft-type-v q4_0 --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0

• Avg throughput: ~24 tokens/sec
• Quality: very good reasoning on the coding task (assessed via the pi harness)
• KV cache: 32K at q4_0
• CPU usage during eval: up to ~44%
• Context: not enough — the 32K window ran out on the full-repo task, so I had to run /compact to continue
• System RAM during eval: ~1.5 GB

Flash attention, single slot, no memory fitting (no MTP)

llama-server.exe -m D:\source\llms\unsloth\Qwen3.8-27B-Q4_K_M.gguf -ngl 99 --port 1234 -c 32096 --cache-type-k q4_0 --cache-type-v q4_0 -fa on --fit off --parallel 1 --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0

• Avg throughput: ~17 tokens/sec
• Quality: very good reasoning on the coding task
• KV cache: 32K at q4_0
• CPU usage during eval: up to ~25%
• Context: not enough — same 32K ceiling, required /compact mid-task
• System RAM during eval: ~1.5 GB

MTP2 vs no MTP — unsloth Q4_K_M (avg 24 vs 17 TPS, 32K q4_0 KV cache)

AtomicChat Qwen3.8-27B-AD-IQ4_XS-IQ3_S.gguf

A mixed-precision dynamic quantization (IQ4_XS with an IQ3_S layer set). The interesting part: it's small enough in practice to give the 64K KV cache real room to breathe.

With MTP2, flash attention, single slot, no memory fitting

llama-server.exe -m D:\source\llms\atomic\Qwen3.8-27B-AD-IQ4_XS-IQ3_S.gguf -ngl 99 --port 1234 -c 64096 --cache-type-k q4_0 --cache-type-v q4_0 -fa on --fit off --parallel 1 --spec-type draft-mtp --spec-draft-n-max 2 --spec-draft-type-k q4_0 --spec-draft-type-v q4_0 --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0

• Avg throughput: ~42 tokens/sec — the fastest configuration tested
• Quality: good reasoning on the coding task — a step below the other three runs
• KV cache: 64K at q4_0
• CPU usage during eval: up to ~27%
• Context: enough — the full-repo task completed without /compact
• System RAM during eval: ~1 GB

Flash attention, single slot, no memory fitting (no MTP)

llama-server.exe -m D:\source\llms\atomic\Qwen3.8-27B-AD-IQ4_XS-IQ3_S.gguf -ngl 99 --port 1234 -c 64096 --cache-type-k q4_0 --cache-type-v q4_0 -fa on --fit off --parallel 1 --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0

• Avg throughput: ~27.5 tokens/sec
• Quality: very good reasoning — output nearly identical to unsloth Q4_K_M on this task
• KV cache: 64K at q4_0
• CPU usage during eval: up to ~24%
• Context: enough — full-repo task completed without /compact
• System RAM during eval: ~1 GB

MTP2 vs no MTP — AtomicChat AD-IQ4_XS-IQ3_S (avg 42 vs 27.5 TPS, 64K q4_0 KV cache)

The quantization analysis behind the numbers

The AD-IQ4_XS-IQ3_S file comes from AtomicChat, who also published a quantization analysis with the measurements and instructions behind it. The takeaway for anyone choosing a 4-bit Qwen 3.8 27B quant: there is effectively no quality loss between the three — the margins are small across the board:

• unsloth Qwen3.8-27B-IQ4_XS.gguf — at the top of the group
• lmstudio-community Qwen3.8-27B-Q4_K_M.gguf — still a bit better than AD-IQ4_XS-IQ3_S, by a small margin, but a bit worse than IQ4_XS
• AtomicChat AD-IQ4_XS-IQ3_S — no meaningful loss against the other two on these measurements

That lines up with my own coding-task impressions above: the no-MTP runs on both quantizations produced nearly identical reasoning quality.

AtomicChat's quantization analysis — measurements comparing the 4-bit Qwen 3.8 27B quantizations (source: AtomicChat)

What stands out

• MTP2 is a real speedup, not a rounding error. It lifted throughput from 17 → 24 TPS (+41%) on the unsloth quant and from 27.5 → 42 TPS (+53%) on the AtomicChat quant. On this hardware, the drafting cost is clearly worth it.
• But MTP shifts load to the CPU. CPU utilization during eval was much higher in the unsloth MTP run (up to ~44%) than in its no-MTP baseline (~25%). The AtomicChat MTP run stayed mild (~27%). Worth knowing if your CPU is also busy.
• The 32K context was the practical ceiling of the unsloth Q4_K_M run on this card — not a property of the model. Q4_K_M is the heaviest quant tested here, and its footprint leaves little VRAM left for the KV cache on 16 GB, so both unsloth configurations ran out on the full-repo task and required /compact to finish. The 64K AtomicChat configurations completed it cleanly. Don't generalize the 32K ceiling to the whole Qwen 3.8 27B line: with a smaller unsloth quant, the same card could hold a larger KV cache, and the answer may change.
• Quality held up better than expected. The no-MTP AtomicChat run produced reasoning nearly identical to unsloth Q4_K_M on this workload, at 27.5 TPS with lower RAM usage. The one soft spot: AtomicChat with MTP2 felt a step weaker on reasoning quality than its own no-MTP run.
• System RAM is mostly mmap. The server's memory-mapped model file alone shows ~13 GB of system DDR5 even when idle, plus ~1 GB (AtomicChat) to ~1.5 GB (unsloth) of working set during eval. Don't panic at the RAM meter — most of that 13 GB is the mapped GGUF, not extra copies of the model.

Takeaways

• Default to AtomicChat AD-IQ4_XS-IQ3_S with MTP2 on a 16 GB card if you want maximum speed: 42 TPS, 64K context that actually fits real coding workloads, and the lowest system RAM of the four runs — accepting a slight quality trade-off under MTP.
• Pick unsloth Q4_K_M (no MTP, or MTP2 if you can spare the CPU) if quality is the priority and you accept a 32K context — that ceiling comes from Q4_K_M's footprint on 16 GB, not the model, so a smaller unsloth quant can fit more context.
• Turn on MTP2 by default and measure your own task: +40–50% tokens per second is a meaningful difference in a coding workflow.
• Leave headroom. Budget for the ~13 GB system-RAM figure from the server's mmap before you stack other workloads next to it.

What's next

On the roadmap: unsloth Qwen3.8-27B-IQ4_XS.gguf (15.7 GB) — a 4-bit dynamic quant that squeezes even closer to the 16 GB ceiling. I flagged it after reading this r/LocalLLM thread on running Qwen 3.8 27B dense fully on a single card. The analysis above puts IQ4_XS at the top of that group anyway — so quality-wise it's the one to beat if it fits. If it does fit, the KV cache gets whatever is left over — so the context math from this post is exactly what matters there.

If you're setting this up from scratch, start with the Qwen 3.8 27B setup guide and come back here for the configuration choices.

There's something satisfying about running a language model on your own hardware. No API keys, no subscriptions, no data leaving your machine. Just you and the model.

Not long ago, running a frontier-class model locally meant compromises: small models, weak answers, or handing your data to a cloud service. That's over. Qwen 3.8 27B is the current state of the art among open-weight models you can run on consumer hardware — a dense ~27-billion-parameter model with vision and reasoning, a 256K-token context window, and an Apache 2.0 license. It doesn't need a data center, just one GPU in your own PC.

This is a follow-up to my earlier post, Running Local Models with llama.cpp on Windows & ROCm. That one got you running fast with AMD's pre-built binaries — but those binaries have a hidden cost: they're an older llama.cpp, compiled once for a whole family of GPUs, and they leave performance on the table. On my own benchmarks, building the latest llama.cpp from source, tuned to my exact GPU architecture, delivered noticeably higher tokens per second (TPS). This guide takes the from-source path.

Why build from source? (vs. AMD's pre-built binaries)

AMD ships ready-made llama.cpp binaries that "just work" — that's the route my previous post took. The catch:

• They're an older llama.cpp build — every release adds faster kernels and better memory handling, and you miss all of it.
• They're compiled once for a range of GPUs (e.g. gfx110X-gfx115X-gfx120X) instead of your exact architecture, so the device code can't be fully tuned to your card.
• Building the latest source with GPU_TARGETS set to your GPU's exact gcnArchName uses newer kernels compiled specifically for your GPU.
• My before/after benchmarks: the from-source build beat the pre-built binaries on tokens per second — most visible on generation-heavy workloads.

The cost is a few extra minutes and one build. Worth it for the most out of your GPU.

Why Qwen 3.8 27B?

• 100% local — your conversations never leave your machine
• State of the art — the dense open-weight model to beat at ~30B parameters
• Free & open — Apache 2.0 license, official GGUF releases from day one
• Huge context — a 256K-token window for long documents and deep conversations
• One consumer GPU — roughly 16–17 GB of VRAM at 4-bit quantization
• Drop-in successor — same size and architecture as Qwen 3.6 27B, so existing GGUF tooling and configuration just work

What you'll have when you're done

• The latest llama.cpp, built from source for your exact GPU — higher tokens per second than AMD's pre-built binaries, and your GPU doing the math instead of your CPU
• A local server speaking the OpenAI API — point any tool at http://localhost:8080
• The tools, source code, and model downloaded, ready to rebuild or swap anytime

The journey at a glance

• Step 1 — Install the tools (driver, ROCm, Visual Studio, Git, CMake, Ninja). You.
• Step 2 — Download the llama.cpp source code. You.
• Step 3 — Download the Qwen 3.8 27B model. You.
• Step 4 — Build, verify, and fix any issues. Your AI assistant — this part has a few sharp edges (GPU architecture, multi-GPU quirks), so it deserves its own detailed checklist: BUILD_HIP_WINDOWS.md.
• Step 5 — Start the server and talk to the model. Both of you.

This guide covers steps 1–3 (the downloads). Step 4 is a technical job with a few sharp edges, so it has its own detailed document that your AI assistant will work through — more on that in Part 4.

Part 1: Install the tools

None of these are optional — each one is needed for the build. Do them in order.

1.1 AMD graphics driver

Download from amd.com/support (pick your GPU). Install, then restart your PC.

1.2 ROCm (AMD's GPU compute kit)

Go to amd.com/products/software/rocm and download the Windows installer (it's big — several GB). Run the installer. You only need ROCm Runtime, HIP, and rocBLAS — you can skip PyTorch, TensorFlow, profiling tools, MIOpen, and everything else. It lands in C:\Program Files\AMD\ROCm\<version> — remember the version number.

Note: ROCm installs a driver component. If your antivirus blocks it, add an exception for C:\Program Files\AMD\ROCm and restart.

1.3 Visual Studio (Community is free)

Download from visualstudio.microsoft.com. In the installer's workload list, tick "Desktop development with C++". You don't need anything else.

1.4 Git

Download from git-scm.com and install with the default options.

1.5 CMake

Download the Windows x64 Installer from cmake.org. During install, tick "Add CMake to the system PATH".

1.6 Ninja

Download ninja-win.zip from github.com/ninja-build/ninja/releases (get the newest release), or run winget install Ninja-build.Ninja. If you downloaded the zip: unzip it, put ninja.exe in a folder that's on your PATH (e.g. C:\Ninja), and add that folder to your PATH in System Properties → Environment Variables.

Part 2: Get the source code

llama.cpp is the engine that runs GGUF models like ours. Open PowerShell (or Git Bash) and run:

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp

The build itself happens later, with the AI's help — for now, downloading is enough.

Part 3: Get the model: Qwen 3.8 27B

1. Download the 4-bit GGUF from either of these (both are official community quants):
   • lmstudio-community/Qwen3.8-27B-GGUF — grab the file Qwen3.8-27B-Q4_K_M.gguf (LM Studio community, built with llama.cpp)
   • or unsloth/Qwen3.8-27B-GGUF — Unsloth's dynamic 4-bit GGUFs
2. Check your VRAM first (Task Manager → Performance → GPU). A 4-bit 27B needs 16–17 GB — a 16 GB+ GPU is required. Less VRAM? Download Q3_K_M instead, or pick a smaller model; the steps stay the same.
3. Download the .gguf file to a folder you'll remember, e.g. D:\models\. Keep the full path handy — you'll need it later.

Part 4: Hand over to the AI

Your PC is now ready. Point your AI assistant at the technical checklist — BUILD_HIP_WINDOWS.md — it will:

• Find your ROCm install and check your GPU's exact architecture
• Configure and build llama.cpp with the ROCm backend
• Verify the GPU is actually being used
• Fix anything that goes wrong

One thing to tell the AI before it starts: does your PC also have integrated graphics? (Windows shows a second GPU like "AMD Radeon(TM) Graphics" next to your real one.) If yes, say so — it changes how the setup is done and avoids a very common startup crash.

Wrapping Up

Running a frontier-class model locally isn't just practical — with the from-source build, it's the fastest option available on your hardware. No cloud, no subscriptions, no data leaving your machine; just the latest kernels, tuned to your exact GPU.

If you followed the previous post with AMD's pre-built binaries, you already know how far this goes. Now imagine the same setup with noticeably more tokens per second. That's what building from source buys you.

The model we used is just the start. The GGUF ecosystem has hundreds of models on Hugging Face — pick one that fits your VRAM and start experimenting. Welcome to local AI.

There's something satisfying about running a language model on your own hardware. No API keys, no subscriptions, no data leaving your machine. Just you and the model.

If you have an AMD GPU and a Windows machine, you're in luck. With llama.cpp and ROCm, you can run models locally with performance that rivals — and sometimes beats — tools like LM Studio. The catch? Building llama.cpp from source on Windows with ROCm support is notoriously painful. But luckily, AMD provides pre-built binaries that skip all that hassle.

This guide walks you through two things:

• Part 1: Installing the C++ build dependencies you'll need via Visual Studio
• Part 2: Downloading and running llama.cpp with a real model on your AMD GPU

Part 1: Setting Up C++ Dependencies with Visual Studio

Before we touch llama.cpp, we need to make sure your system has the C++ build tools installed. Even though we're using pre-built binaries (not compiling from source), some dependencies and runtime libraries still expect these to be present.

Step 1: Open Visual Studio Installer

If you don't have it yet, download Visual Studio Community from visualstudio.microsoft.com. If it's already installed, open the Visual Studio Installer from your Start Menu.

You'll land on a screen like this — click the Modify button (or Install if it's a fresh install):

Visual Studio Installer — click Modify to add workloads

Step 2: Select the Right Workloads

In the Workloads tab, check the following:

• .NET desktop development — provides core .NET runtime tools and C# support
• Desktop development with C++ — this is the important one. It installs the MSVC compiler, Windows SDK, and C++ standard libraries that llama.cpp binaries depend on at runtime

On the right panel, make sure the optional components are checked — especially anything related to C++ build tools and the latest Windows SDK.

Select .NET desktop development and Desktop development with C++

Step 3: Install / Modify

Click Install (or Modify) in the bottom right corner. The installer will download and set up everything. It takes a few minutes depending on your connection.

Once it's done, you're ready for the fun part.

Part 2: Running Local Models with llama.cpp

llama.cpp is a lightweight C++ implementation of Meta's LLaMA architecture. It runs models in the GGUF format directly on your hardware. Combined with ROCm (AMD's GPU compute platform), it can offload model layers to your AMD GPU for serious speed.

Why not use something like LM Studio? LM Studio is great for getting started, but it abstracts away control. With llama.cpp, you get:

• Full control over context length, GPU layers, and sampling parameters
• An OpenAI-compatible API server you can plug into any tool
• Benchmarking tools to actually measure your hardware's performance
• No Electron overhead — just a single executable

The downside? Building from source on Windows with ROCm is a nightmare. But AMD publishes pre-built binaries that just work.

Step 1: Download the Pre-built Binaries

AMD hosts validated builds at repo.radeon.com/rocm/llama.cpp/windows. Grab the latest package:

curl.exe -o llama-bin-windows.zip "https://repo.radeon.com/rocm/llama.cpp/windows/rocm-rel-7.2.1/llama-b8407-windows-rocm-7.2.1-gfx110X-gfx115X-gfx120X-x64.zip"

Step 2: Extract the Archive

Unzip it into a clean directory:

Expand-Archive -Path "llama-bin-windows.zip" -DestinationPath ".\llama_cpp_binaries"

Then navigate into the inner folder:

cd ./llama_cpp_binaries/<specific_folder_name>

Step 3: Download a Model

The binaries are the engine — you still need a model file (in GGUF format) to actually run something. For testing, we'll grab GPT-OSS-20B:

curl.exe -L -o test_model.gguf "https://huggingface.co/ggml-org/gpt-oss-20b-GGUF/resolve/main/gpt-oss-20b-mxfp4.gguf"

This is a ~20B parameter model. Depending on your GPU VRAM, you might want to pick a smaller model. The process is the same regardless.

Step 4: Start the Server

llama-server is a lightweight, OpenAI-compatible web server included with llama.cpp. It hosts your model locally and gives you a chat interface in your browser.

# Start the server
# -ngl 99: Offload all layers to your AMD GPU (Crucial for performance)
# -c: Context Length
# -fa: Enable Flash Attention to reduce memory usage and increase speed
.\llama-server.exe -m test_model.gguf -c 2048 -ngl 99 -fa on --port 8080

Open your browser and go to http://localhost:8080. You'll see a clean chat interface where you can talk to the model.

The -ngl 99 flag is key — it tells llama.cpp to offload all model layers to your AMD GPU. Without it, you're running on CPU, which is painfully slow for anything beyond tiny models.

Step 5: (Optional) Run a Benchmark

Want to know how fast your GPU is actually running? Use the built-in benchmark tool:

# Run the benchmark with the downloaded model.
# -m: specifies the model file
.\llama-bench.exe -m .\test_model.gguf -fa 1

This will output two key metrics:

• PP (Prompt Processing) — how fast the model processes your input
• TG (Token Generation) — how fast it generates output tokens

These numbers are useful for comparing different models or tuning your -ngl and -c parameters.

Wrapping Up

Running local models isn't just a novelty — it's becoming practical. With AMD GPUs and pre-built llama.cpp binaries, you can have a fully local, OpenAI-compatible API running on your machine in under 10 minutes.

The model we used (GPT-OSS-20B) is just a starting point. The GGUF ecosystem has hundreds of models available on Hugging Face, from tiny 1B models to 70B+ beasts. Pick one that fits your VRAM and start experimenting.

I'll be writing more about practical AI engineering — building agents, running inference, and shipping AI-powered products. Follow along if that's your thing.

The goal was never to build a finished product. It was to build a starting point — something others could contribute to, improve, and eventually turn into a real system that helps hearing-impaired people communicate more easily.

How It Works

The system supports two modes of operation:

Word-Level Recognition

Each sign language gesture is treated as an independent input. You perform a word, press 'q' to stop recording, and the model predicts what you signed. It's accurate, simple, and fast — perfect for real-time use.

• Input space is discrete — each video is one word
• Returns top-n predictions sorted by confidence
• Works with any camera (webcam, phone, etc.)

Sentence-Level Recognition

The system can also accept continuous input — sentences rather than individual words. It's slower and less accurate than word-level, but it handles real-world input where you don't pause between words.

• Input space is continuous
• Accepts any video format (mp4, flv, webm, etc.)
• More flexible, but trades off speed and accuracy

Results

Word-level system:

Sentence-level system:

Features

• REST API — serve the system as an HTTP endpoint, making it easy to integrate into any application without mixing Python dependencies
• WampServer / Xampp — local web interface for testing, with a simple UI demonstrating use cases
• Multiprocessing — the full pipeline is multi-processed for performance (though some OS-level quirks on Windows can cause bottlenecks)
• CPU & GPU — runs on both; GPU support requires CUDA and cuDNN
• OS — tested on Ubuntu 18 and Windows 10

Quick Start

Clone the repo and install requirements:

git clone https://github.com/AtaaEddin/beautiful-sign-language-tr
cd beautiful-sign-language-tr
pip install -r requirements.txt

Run a quick webcam test:

python main.py -run webcam -pred_type word -download True

While it's running, perform one of the 10 supported Turkish sign language words, then press 'q' to end recording and get a prediction.

How to Contribute

This project was designed to grow with the community. If you want to help:

• Find or create a sign language video dataset (even recording with your phone works)
• Label the videos and train using the existing models
• Open an issue describing your dataset and training results
• We'll merge datasets and do overall training together

All submitted datasets will be made publicly available for anyone who wants to train using this project's models or their own.

What's Next

The system currently recognizes 10 Turkish sign language words, but the dream is to scale that up significantly — hundreds of words, full sentences, and real-time performance that makes it usable in daily life.

The code is open source on GitHub. If this interests you, contribute, open issues, or just star the repo to show support.

This was one of my earlier projects — a chatbot that could hold a conversation in Turkish, complete with a wake word to activate it and a physical-looking interface to interact with. The idea was simple: train a sequence-to-sequence model on Turkish Twitter conversations and make it feel like talking to someone.

The query "sen güzel bir kızsın" (you're a beautiful girl) would get a response like "sen daha güzelsin!!" (you're even more beautiful!!). Not perfect, but fun.

How It Works

The system has two main parts:

1. Trigger Word Detection

Before you can talk to the bot, you need to wake it up — just like saying "Alexa" to an Amazon Echo. Our trigger phrase was "uyan ay" (wake up, moon). When the bot hears it, it activates and starts listening.

2. ChatBot (Seq2Seq Model)

The chatbot is a task-specific system focused on general conversation. For any input text — a question, a joke, a statement — it generates a response derived from patterns it learned from Turkish Twitter data.

The flow is:

• Voice → Text: Your speech is converted to text using Google's voice recognition API
• Text → Text: The seq2seq model generates a response
• Text → Voice: The response is spoken back through the human interface

query: seni çok seviyorum. (I love you so much.)

ChatBot: ben de seni çok seviyorum yanımızda olduğun için sonsuz teşekkürler. (I love you too, thank you so much for being with us.)

Tech Stack

• Python 3
• TensorFlow ≤ 1.11.0 — for the seq2seq model
• TensorLayer ≥ 1.6.3 — high-level deep learning library
• Google Voice Recognition API — for speech-to-text
• Internet connection required for voice APIs

Installation

Ubuntu:

sudo apt install libasound-dev portaudio19-dev libportaudio2 libportaudiocpp0 ffmpeg libav-tools
sudo apt-get install python3-tk
pip3 install -r requirements.txt

Windows:

pip install -r requirements.txt

Running It

# Ubuntu (Python 3)
python3 my_robot.py

# Ubuntu (Python 2)
python my_robot.py

# Windows
python my_robot.py

Looking Back

This project taught me a lot about NLP, sequence models, and the challenges of working with Turkish — a language with complex morphology that doesn't play nicely with standard tokenization. The seq2seq approach was state-of-the-art at the time, before transformers took over.

If I were to rebuild this today, I'd use a fine-tuned transformer model and skip the voice recognition API entirely — running Whisper locally would make the whole thing work offline. But that's the beauty of old projects: they show you how far the field has come.