llama.cpp runs GGUF models directly on your AMD GPU when built against ROCm. On Linux you often want a source build so you can match your exact GPUs and rebuild on a whim. This guide walks through the whole thing in order, from installing the ROCm pieces CMake needs to running a model across multiple GPUs.

The setup I'm using. An AMD Ryzen 7 7800X3D CPU with two discrete Radeons — an RX 9070 XT (16 GB) and an AI PRO R9700 (32 GB), both RDNA 4 (gfx1201) — plus the CPU's integrated graphics (gfx1036). Two architectures on one machine, which step 6 addresses.

Step 1: Install ROCm

Install ROCm with the amdgpu-install script, selecting ROCm plus graphics:

sudo amdgpu-install --usecase=rocm,graphics

Reboot afterwards, then confirm your GPUs are visible:

rocminfo

Step 2: Install the ROCm Development Packages

Building from source needs the development packages, which ship the CMake config files that llama.cpp's configure step requires (hip-config.cmake, hipblas-config.cmake, rocblas-config.cmake). They are separate from the runtime install. On ROCm 7.14's packaging install them with:

sudo apt install \
    amdrocm-runtime-dev7.14 \
    amdrocm-blas-dev7.14 \
    amdrocm-core-dev7.14-gfx1201 \
    amdrocm-hipblas-common-dev7.14

A note on the packages:

• amdrocm-runtime-dev7.14 — the HIP runtime development files, including hip-config.cmake and the hip-lang package.
• amdrocm-blas-dev7.14 — the hipBLAS and rocBLAS dev files that provide the BLAS CMake configs.
• amdrocm-core-dev7.14-gfx1201 — architecture-specific core dev files; pick the target that matches your GPU.
• amdrocm-hipblas-common-dev7.14 — shared header-only hipBLAS files.

To double-check the CMake files are present before configuring, run:

find /opt/rocm -name "hip-config.cmake" -o -name "hipblas-config.cmake" -o -name "rocblas-config.cmake"

Step 3: Configure with CMake

Now configure the build. Because HIP code must be compiled by ROCm's bundled clang — not the system gcc — set both compilers explicitly. Pass GPU_TARGETS=gfx1201 to build kernels for your discrete RDNA 4 GPUs.

cmake -S . -B build \
  -DGGML_HIP=ON \
  -DGPU_TARGETS=gfx1201 \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_C_COMPILER=/opt/rocm/llvm/bin/clang \
  -DCMAKE_CXX_COMPILER=/opt/rocm/llvm/bin/clang++

If you want one binary that also runs on the integrated GPU, add it as a second target:

  -DGPU_TARGETS="gfx1201;gfx1036"

That costs a longer build and a bigger binary. I built only for gfx1201 and hid the iGPU at runtime instead (step 7).

Step 4: Build

With configure done, build — everything in parallel:

cmake --build build --config Release -j$(nproc)

This compiles the HIP kernels under ggml/src/ggml-hip for each target. On an 8-core 7800X3D it takes a while; it finishes on its own. The llama-cli, llama-server, and llama-bench binaries end up in build/bin.

Step 5: Set the Runtime Library Path

On this ROCm release the runtime libraries live under /opt/rocm/core-7.14/lib — a path the system loader doesn't know about by default. Point it there so the binaries can find their dependencies:

export LD_LIBRARY_PATH=/opt/rocm/core-7.14/lib:$LD_LIBRARY_PATH

Confirm everything resolves by checking your binary against the dynamic libraries:

ldd build/bin/llama-cli

Every libhipblas, librocblas, and libamdhip64 should resolve to a real path — none should read "not found".

Step 6: Verify the GPUs

llama.cpp reports what it sees:

./build/bin/llama-cli --list-devices
Available devices:
  ROCm0: AMD Radeon RX 9070 XT   (16304 MiB)
  ROCm1: AMD Radeon AI PRO R9700 (32624 MiB)
  ROCm2: AMD Radeon Graphics     (14953 MiB)

Step 7: Exclude the Integrated GPU

The third entry is the CPU's integrated graphics. The discrete cards are gfx1201; the iGPU is gfx1036 — a different architecture with no matching kernels in this build. Keeping it out of the process entirely is simplest; it also avoids the classic multiple-architecture conflict on machines with both an iGPU and a discrete AMD GPU. Hide it from HIP:

export HIP_VISIBLE_DEVICES=0,1

Now device list shows only the two discrete cards:

./build/bin/llama-cli --list-devices
ROCm0: AMD Radeon RX 9070 XT
ROCm1: AMD Radeon AI PRO R9700

Step 8: Set Up ~/.bashrc

Exporting those two variables each session gets tiring. Add them to ~/.bashrc so every new terminal is ready to go:

# llama.cpp + ROCm 7.14 environment
export LD_LIBRARY_PATH=/opt/rocm/core-7.14/lib:$LD_LIBRARY_PATH
export HIP_VISIBLE_DEVICES=0,1

# handy aliases
alias llama-cli='$HOME/source/repos/llama.cpp/build/bin/llama-cli'
alias llama-server='$HOME/source/repos/llama.cpp/build/bin/llama-server'
alias llama-bench='$HOME/source/repos/llama.cpp/build/bin/llama-bench'

Open a fresh terminal and everything is there.

Step 9: Run a Model

Run with a local GGUF model, using -ngl 99 to push every layer to the GPU:

llama-cli -m path/to/model.gguf -ngl 99 -p "Hello" --no-display-prompt

To use both discrete cards together, layer split is the simplest reliable option:

llama-cli -m path/to/model.gguf -ngl 99 -sm layer

For tensor split with good throughput, rebuild with -DGGML_HIP_RCCL=ON so the two GPUs can communicate directly instead of through system memory.

Wrapping Up

From a fresh install to a running multi-GPU llama.cpp, it's these eight commands' worth of steps:

• Install ROCm, then the development packages.
• Configure with ROCm's clang and GPU_TARGETS=gfx1201.
• Build, and export LD_LIBRARY_PATH to the ROCm lib dir.
• Hide the integrated GPU with HIP_VISIBLE_DEVICES.
• Pin it all in ~/.bashrc so a fresh terminal is ready immediately.

Once it's set up it just runs. I'll keep posting about squeezing tokens out of these Radeon cards.