llama.cpp inference on ROCm

llama.cpp inference on ROCm#

llama.cpp is an open-source inference engine for running LLMs locally. It runs quantized models efficiently on everyday hardware and has become a pillar of local LLM inference. This page explains how to set up and run llama.cpp on AMD Radeon™ GPUs and Ryzen™ APUs.

Device family

AMD Radeon

AMD Ryzen

Instinct GPU

Radeon GPU

Ryzen APU

Operating system

Ubuntu

Windows

ROCm version

7.14.0

ROCm installation

pip

Tarball

Prerequisites#

Install ROCm#

llama.cpp requires a ROCm installation on the host system.

Device family: AMD Radeon
Select your GPU – importantly, gfx architecture – from the dropdown list.
Use case: Compute
Operating system: Ubuntu
Installation method: pip

Device family: AMD Radeon
Select your GPU – importantly, gfx architecture – from the dropdown list.
Use case: Compute
Operating system: Ubuntu
Installation method: Tarball

Device family: AMD Radeon
Select your GPU – importantly, gfx architecture – from the dropdown list.
Use case: Compute
Operating system: Windows
Installation method: pip

Device family: AMD Radeon
Select your GPU – importantly, gfx architecture – from the dropdown list.
Use case: Compute
Operating system: Windows
Installation method: Tarball

Device family: AMD Ryzen
Select your GPU – importantly, gfx architecture – from the dropdown list.
Use case: Compute
Operating system: Ubuntu
Installation method: pip

Device family: AMD Ryzen
Select your GPU – importantly, gfx architecture – from the dropdown list.
Use case: Compute
Operating system: Ubuntu
Installation method: Tarball

Device family: AMD Ryzen
Select your GPU – importantly, gfx architecture – from the dropdown list.
Use case: Compute
Operating system: Windows
Installation method: pip

Device family: AMD Ryzen
Select your GPU – importantly, gfx architecture – from the dropdown list.
Use case: Compute
Operating system: Windows
Installation method: Tarball

Install llama.cpp#

AMD does not publish its own binaries for llama.cpp. Download a prebuilt ROCm-enabled release from the llama.cpp releases page. Release assets are named in the following format:

Note

The archive contains only the llama.cpp executables and their ggml backend libraries. It does not include the ROCm runtime, which must be installed separately as described in the previous section.

Configure your environment#

llama.cpp loads the ROCm math libraries at run time and must be able to find them.

A pip installation places the ROCm runtime inside the virtual environment in two directories:

$VIRTUAL_ENV/lib/python3.14/site-packages/_rocm_sdk_core/lib
$VIRTUAL_ENV/lib/python3.14/site-packages/_rocm_sdk_libraries/lib

Add them to LD_LIBRARY_PATH. Source your virtual environment where you installed ROCm or replace $VIRTUAL_ENV with the actual path (you might also need to change the Python version directory). Use the following command:

export LD_LIBRARY_PATH="$VIRTUAL_ENV/lib/python3.14/site-packages/_rocm_sdk_core/lib:$VIRTUAL_ENV/lib/python3.14/site-packages/_rocm_sdk_libraries/lib${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}"

This setting applies to the current shell session. To make it permanent, add the lines to your shell startup script (such as ~/.bashrc).

A pip installation places the ROCm runtime inside the virtual environment in two directories:

<venv>\Lib\site-packages\_rocm_sdk_core\bin
<venv>\Lib\site-packages\_rocm_sdk_libraries\bin

Add both to your PATH in the command prompt you use to launch llama.cpp, replacing <venv> with the actual path of your virtual environment, for example:

set "PATH=C:\llamacpp\.venv\Lib\site-packages\_rocm_sdk_core\bin;C:\llamacpp\.venv\Lib\site-packages\_rocm_sdk_libraries\bin;%PATH%"

This setting applies to the current command prompt only. To make it permanent, add both directories to your user or system PATH in your Windows settings.

If the ROCm libraries are not on your PATH, llama.cpp does not report an error. It silently falls back to the CPU backend and runs several times slower. Use Verify the installation to confirm the GPU is detected.

Copy the HIP runtime libraries#

Copy the following three files from your ROCm installation into the directory that contains llama-cli.exe: <venv>\Lib\site-packages\_rocm_sdk_core\bin

amdhip64_7.dll
rocm_kpack.dll
amd_comgr.dll

Copy all three files. Copying amdhip64_7.dll on its own prevents llama.cpp from using the GPU: the HIP runtime supplied by ROCm depends on the matching rocm_kpack.dll and amd_comgr.dll, and rocm_kpack.dll is not present in System32.

copy "<venv>\Lib\site-packages\_rocm_sdk_core\bin\amdhip64_7.dll" C:\llamacpp\
copy "<venv>\Lib\site-packages\_rocm_sdk_core\bin\rocm_kpack.dll" C:\llamacpp\
copy "<venv>\Lib\site-packages\_rocm_sdk_core\bin\amd_comgr.dll"  C:\llamacpp\

Copy the following three files from your ROCm installation into the directory that contains llama-cli.exe: C:\TheRock\build\bin

amdhip64_7.dll
rocm_kpack.dll
amd_comgr.dll

Important

Copy all three files. Copying amdhip64_7.dll on its own prevents llama.cpp from using the GPU: the HIP runtime supplied by ROCm depends on the matching rocm_kpack.dll and amd_comgr.dll, and rocm_kpack.dll is not present in System32.

copy "C:\TheRock\build\bin\amdhip64_7.dll" C:\llamacpp\
copy "C:\TheRock\build\bin\rocm_kpack.dll" C:\llamacpp\
copy "C:\TheRock\build\bin\amd_comgr.dll"  C:\llamacpp\

Note

The AMD display driver installs its own copy of amdhip64_7.dll in C:\Windows\System32. Windows searches System32 before PATH, so the driver’s version is loaded in preference to the one supplied by ROCm, even when ROCm is on PATH. Because Windows searches the directory containing the executable first, copying these libraries next to llama-cli.exe ensures the ROCm version is used.

Verify the installation#

List the devices visible to llama.cpp. Make sure to cd into where the llama.cpp binaries are located and that you’re in the same shell in which you set your PATH.

The output shows the detected GPU and its available memory. For example:

Available devices:
  ROCm0: AMD Radeon RX 7900 XT (20464 MiB, 20315 MiB free)

If no devices are listed at all, the ROCm runtime cannot be found. Review Configure your environment.

Run a llama.cpp example#

Once your llama.cpp environment is set up, experiment with the following steps to run a model and benchmark your installation. The prebuilt llama.cpp release includes the executables to exercise the functionality of your installation.

The two most popular use cases are:

llama-cli: The main executable to run the model interactively or get a response to a prompt.
llama-bench: Run a benchmark of your model with different configurations.

llama-cli#

Use the CLI tool to start the client, replacing <model>.gguf with the path to a model on your system:

A prompt appears when the client is ready, and you can start interacting with the model:

> Explain what a GPU kernel is in one sentence.
A GPU kernel is a specialized program within a GPU (Graphics Processing Unit) designed to execute
instructions for rendering graphical content, such as images or video, on a GPU's hardware architecture.

[ Prompt: 593.4 t/s | Generation: 286.4 t/s ]

To exit, enter /exit or press Ctrl+C.

To send a single prompt and exit instead of starting an interactive session, add the -p and -st flags:

Note

llama-bench#

Use the CLI tool to start the application, replacing <model>.gguf with the path to a model on your system:

The result of the command above should be similar to the following when running on an AMD Radeon RX 7900 XT system:

ggml_cuda_init: found 1 ROCm devices (Total VRAM: 20464 MiB):
  Device 0: AMD Radeon RX 7900 XT, gfx1100 (0x1100), VMM: no, Wave Size: 32, VRAM: 20464 MiB
load_backend: loaded ROCm backend from C:\llamacpp\ggml-hip.dll
load_backend: loaded RPC backend from C:\llamacpp\ggml-rpc.dll
load_backend: loaded CPU backend from C:\llamacpp\ggml-cpu-haswell.dll
| model                          |       size |     params | backend    | ngl |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --------------: | -------------------: |
| qwen2 1B Q4_K - Medium         | 462.96 MiB |   630.17 M | ROCm       | 999 |            pp16 |     2406.95 ± 704.75 |
| qwen2 1B Q4_K - Medium         | 462.96 MiB |   630.17 M | ROCm       | 999 |            pp32 |     3842.56 ± 500.15 |
| qwen2 1B Q4_K - Medium         | 462.96 MiB |   630.17 M | ROCm       | 999 |            pp64 |    7104.29 ± 1218.73 |
| qwen2 1B Q4_K - Medium         | 462.96 MiB |   630.17 M | ROCm       | 999 |           pp128 |     8628.19 ± 928.69 |
| qwen2 1B Q4_K - Medium         | 462.96 MiB |   630.17 M | ROCm       | 999 |           pp256 |   16444.29 ± 1662.23 |
| qwen2 1B Q4_K - Medium         | 462.96 MiB |   630.17 M | ROCm       | 999 |           pp512 |   24628.13 ± 2498.10 |
| qwen2 1B Q4_K - Medium         | 462.96 MiB |   630.17 M | ROCm       | 999 |          pp1024 |     26143.12 ± 73.58 |
| qwen2 1B Q4_K - Medium         | 462.96 MiB |   630.17 M | ROCm       | 999 |            tg64 |        373.29 ± 9.81 |
| qwen2 1B Q4_K - Medium         | 462.96 MiB |   630.17 M | ROCm       | 999 |           tg128 |        375.70 ± 3.49 |

build: 78ec4c378 (10539)

The backend column shows ROCm when the GPU is in use. If it shows CPU, the ROCm libraries were not found. Review the environment configuration steps to troubleshoot.

Flash Attention is enabled with -fa 1 and is supported on Radeon GPUs through rocWMMA:

llama.cpp inference on ROCm

Contents

llama.cpp inference on ROCm#

Prerequisites#

Install ROCm#

Install llama.cpp#

Verify the installation#

Run a llama.cpp example#

llama-cli#

llama-bench#