GPU¶

• GPU: compute capability 7.0 or higher (e.g., V100, T4, RTX20xx, A100, L4, H100, etc.)

• GPU: Validated on gfx942 (It should be supported on the AMD GPUs that are supported by vLLM.)

• GPU: Validated on Intel® Arc™ B-Series.

• GPU: Moore Threads GPU with MUSA SDK installed (validated on MTT S5000)

Installation of vLLM¶

These pre-built wheel instructions install the published vLLM-Omni 0.28.0 release. For the 0.30 development line, use the source-install instructions below.

vLLM-Omni is built based on vLLM. Please install it with command below.

uv pip install vllm==0.28.0 --torch-backend=auto

Installation of vLLM-Omni¶

uv pip install vllm-omni==0.28.0

To run Gradio demos, also install the optional extras:

uv pip install 'vllm-omni[demo]==0.28.0'

Installation of vLLM¶

These pre-built wheel instructions install the published vLLM-Omni 0.28.0 release. For the 0.30 development line, use the source-install instructions below.

vLLM-Omni is built based on vLLM. Please install it with command below.

uv pip install vllm==0.28.0+rocm723 --extra-index-url https://wheels.vllm.ai/rocm/0.28.0/rocm723

Installation of vLLM-Omni¶

# we need to add --no-build-isolation as the torch
# is not obtained from pypi, we have to install using the
# torch installed in our environment
uv pip install vllm-omni==0.28.0

# Optional if want to run Qwen3 TTS
uv pip uninstall onnxruntime # should be removed before we can install onnxruntime-rocm
uv pip install onnxruntime-rocm

Installation of vLLM¶

If you do not need to modify source code of vLLM, you can directly install the stable 0.30.0 release version of the library

uv pip install vllm==0.30.0 --torch-backend=auto

The 0.30.0 release of vLLM ships CUDA 13.0-compatible binaries by default. If you need a different CUDA variant or want to reuse an existing PyTorch installation, build vLLM from source instead.

Installation of vLLM-Omni¶

Since vllm-omni is rapidly evolving, it's recommended to install it from source

git clone https://github.com/vllm-project/vllm-omni.git
cd vllm-omni
uv pip install -e .

To run Gradio demos, install with optional extras:

uv pip install -e '.[demo]'

git clone https://github.com/vllm-project/vllm.git
cd vllm
git checkout v0.30.0

#For CUDA 13.0 (the default for v0.30.0; the wheel filename has no `+cu130` suffix)
export VLLM_PRECOMPILED_WHEEL_LOCATION=https://github.com/vllm-project/vllm/releases/download/v0.30.0/vllm-0.30.0-cp38-abi3-manylinux_2_28_x86_64.whl

uv pip install --editable .

python use_existing_torch.py
uv pip install -r requirements/build/cuda.txt
uv pip install --no-build-isolation --editable .

Installation of vLLM¶

If you do not need to modify source code of vLLM, you can directly install the stable 0.30.0 release version of the library

uv pip install vllm==0.30.0+rocm723 --extra-index-url https://wheels.vllm.ai/rocm/0.30.0/rocm723

The pre-built 0.30.0 vLLM wheel targets ROCm 7.2.3. If you need a different ROCm stack or want to reuse an existing PyTorch installation, build vLLM from source instead.

Installation of vLLM-Omni¶

Since vllm-omni is rapidly evolving, it's recommended to install it from source

git clone https://github.com/vllm-project/vllm-omni.git
cd vllm-omni
VLLM_OMNI_TARGET_DEVICE=rocm uv pip install -e .
# OR
uv pip install -e . --no-build-isolation

git clone https://github.com/vllm-project/vllm.git
cd vllm
git checkout v0.30.0
python3 -m pip install -r requirements/rocm.txt
python3 setup.py develop

Prerequisites¶

• MUSA SDK: Download from MUSA SDK Download
• torchada: CUDA→MUSA compatibility layer for PyTorch (pip install torchada)
• mthreads-ml-py: MTML Python bindings (pip install mthreads-ml-py)
• MATE: MUSA AI Tensor Engine (GitHub)

Installation of vLLM-MUSA¶

git clone https://github.com/MooreThreads/vllm-musa.git
cd vllm-musa
git checkout v0.18.0-dev
pip install . --no-build-isolation -v

Installation of vLLM-Omni¶

git clone https://github.com/vllm-project/vllm-omni.git
cd vllm-omni
VLLM_OMNI_TARGET_DEVICE=musa pip install -e . --no-build-isolation

For Gradio demos:

pip install -e '.[demo]' --no-build-isolation

Environment Variables¶

export MUSA_VISIBLE_DEVICES=0,1
export VLLM_WORKER_MULTIPROC_METHOD=spawn
export VLLM_MUSA_CUSTOM_OP_USE_NATIVE=false

vLLM-Omni offers an official docker image for deployment. These images are built on top of vLLM docker images and available on Docker Hub as vllm/vllm-omni. The version of vLLM-Omni indicates which release of vLLM it is based on.

Nightly CUDA images built from the latest main are also available:

• vllm/vllm-omni:nightly is the rolling multi-architecture tag.
• vllm/vllm-omni:nightly-<commit> provides commit-pinned multi-architecture tags. The newest 14 are retained.

Use a commit-pinned tag when you need a reproducible nightly environment.

Here's an example deployment command that has been verified on 2 x H100's:

docker run --runtime nvidia --gpus 2 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=$HF_TOKEN" \
    -p 8091:8091 \
    --ipc=host \
    vllm/vllm-omni:v0.28.0 \
    vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091

vLLM-Omni offers an official docker image for deployment. These images are built on top of vLLM docker images and available on Docker Hub as vllm/vllm-omni-rocm. The version of vLLM-Omni indicates which release of vLLM it is based on.

Launch vLLM-Omni Server¶

Here's an example deployment command that has been verified on 2 x MI300's:

docker run --rm \
  --group-add=video \
  --ipc=host \
  --cap-add=SYS_PTRACE \
  --security-opt seccomp=unconfined \
  --device /dev/kfd \
  --device /dev/dri \
  -v <path/to/model>:/app/model \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  --env "HF_TOKEN=$HF_TOKEN" \
  -p 8091:8091 \
  vllm/vllm-omni-rocm:v0.28.0 \
  --model Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091

Launch an interactive terminal with prebuilt docker image¶

If you want to run in dev environment you can launch the docker image as follows:

docker run --rm -it \
  --network=host \
  --group-add=video \
  --ipc=host \
  --cap-add=SYS_PTRACE \
  --security-opt seccomp=unconfined \
  --device /dev/kfd \
  --device /dev/dri \
  -v <path/to/model>:/app/model \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  --env "HF_TOKEN=$HF_TOKEN" \
  --entrypoint bash \
  vllm/vllm-omni-rocm:v0.28.0

Build docker image¶

DOCKER_BUILDKIT=1 docker build -f docker/Dockerfile.cuda -t vllm-omni-cuda .

If you want to specify the base vLLM version:

DOCKER_BUILDKIT=1 docker build \
  -f docker/Dockerfile.cuda \
  --build-arg BASE_IMAGE=vllm/vllm-openai:v0.30.0 \
  -t vllm-omni-cuda .

Launch the docker image¶

Launch with OpenAI API Server¶

docker run --runtime nvidia --gpus 2 \
  -v ${HF_HOME:-$HOME/.cache/huggingface}:/root/.cache/huggingface \
  --env "HF_TOKEN=$HF_TOKEN" \
  -p 8091:8091 \
  --ipc=host \
  vllm-omni-cuda \
  vllm serve --omni --model Qwen/Qwen3-Omni-30B-A3B-Instruct --port 8091

By default, this mounts $HOME/.cache/huggingface as the model cache directory. To use a custom location, set the HF_HOME environment variable before running the command (e.g., export HF_HOME=/data/models).

Launch with interactive session for development¶

docker run --runtime nvidia --gpus all -it --rm \
  -v ${HF_HOME:-$HOME/.cache/huggingface}:/root/.cache/huggingface \
  --env "HF_TOKEN=$HF_TOKEN" \
  -p 8091:8091 \
  --ipc=host \
  --entrypoint bash \
  vllm-omni-cuda

Build docker image¶

The source build defaults to the published upstream base image vllm/vllm-openai-rocm:v0.30.0, aligned with the vLLM release used by CI. Older or custom bases must provide the vLLM APIs checked by the Dockerfile's image-build canary. This upstream base is distinct from the prebuilt vllm/vllm-omni-rocm images discussed below; their published-tag availability is tracked in #7405.

DOCKER_BUILDKIT=1 docker build -f docker/Dockerfile.rocm -t vllm-omni-rocm .

To select the upstream ROCm base explicitly, pass BASE_IMAGE:

DOCKER_BUILDKIT=1 docker build \
  -f docker/Dockerfile.rocm \
  --build-arg BASE_IMAGE=vllm/vllm-openai-rocm:v0.30.0 \
  -t vllm-omni-rocm .

Launch the docker image¶

Launch with OpenAI API Server¶

docker run --rm \
--group-add=video \
--ipc=host \
--cap-add=SYS_PTRACE \
--security-opt seccomp=unconfined \
--device /dev/kfd \
--device /dev/dri \
-v ~/.cache/huggingface:/root/.cache/huggingface \
--env "HF_TOKEN=$HF_TOKEN" \
-p 8091:8091 \
--ipc=host \
vllm-omni-rocm \
--model Qwen/Qwen3-Omni-30B-A3B-Instruct --port 8091

Launch with interactive session for development¶

docker run --rm -it \
--network=host \
--group-add=video \
--ipc=host \
--cap-add=SYS_PTRACE \
--security-opt seccomp=unconfined \
--device /dev/kfd \
--device /dev/dri \
-v <path/to/model>:/app/model \
-v ~/.cache/huggingface:/root/.cache/huggingface \
--entrypoint bash \
vllm-omni-rocm

Build docker image¶

DOCKER_BUILDKIT=1 docker build -f docker/Dockerfile.xpu -t vllm-omni-xpu --shm-size=4g .

This layers vLLM-Omni on top of the published vllm/vllm-openai-xpu:<VLLM_VERSION> base image (v0.30.0 by default), which Docker pulls automatically. To target a different vLLM release, pass --build-arg VLLM_VERSION=<tag>. If that tag has not been published yet, build the base from upstream's own Dockerfile first and point the build at it:

git clone --depth 1 --branch "$VLLM_VERSION" https://github.com/vllm-project/vllm /tmp/vllm
DOCKER_BUILDKIT=1 docker build -t vllm-openai-xpu:local --target vllm-openai \
  -f /tmp/vllm/docker/Dockerfile.xpu /tmp/vllm
DOCKER_BUILDKIT=1 docker build -f docker/Dockerfile.xpu -t vllm-omni-xpu --shm-size=4g \
  --build-arg VLLM_BASE=vllm-openai-xpu:local .

Launch the docker image¶

Launch with OpenAI API Server¶

docker run -it -d --shm-size 10g \
  --name {container_name} \
  --net=host \
  --ipc=host \
  --privileged \
  -v /dev/dri/by-path:/dev/dri/by-path \
  --device /dev/dri:/dev/dri \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  --env "HF_TOKEN=$HF_TOKEN" \
  vllm-omni-xpu \
  --model Qwen/Qwen2.5-Omni-3B --port 8091