Pipeline and deploy configurations¶

In vLLM-Omni, a model's PipelineConfig defines its fixed stage topology, while a deploy configuration controls how those stages run.

Pipeline configuration¶

PipelineConfig and its StagePipelineConfig entries are Python definitions owned by the model implementation and registered in OMNI_PIPELINES. They describe fixed model topology and are not accepted in deploy YAMLs.

Common PipelineConfig fields include:

Common StagePipelineConfig fields include:

To add or change topology, define and register a new pipeline variant. Use deploy YAML only for runtime placement, resource sizing, connectors, and other deployment overrides.

Deploy configuration schema¶

The new deploy schema lives under vllm_omni/deploy/ and is paired with a frozen PipelineConfig registered by the model's pipeline.py. Each deploy YAML has these top-level fields:

For fields whose deploy default is null, the deploy layer contributes no override. The effective value may still come from a platform section, an explicit CLI or stage override, or the downstream vLLM engine default.

Note: for the diffusion path, an omitted distributed_executor_backend selects uni on a single GPU (in-process worker, no MessageQueue / /dev/shm output segments) and mp when num_gpus > 1. Set mp explicitly to keep a worker subprocess on one GPU. ray / external_launcher are not fully supported yet.

Stage fields¶

Each entry under stages: accepts any StageDeployConfig field directly (no nested engine_args:). Only fields whose value legitimately varies across stages live here; pipeline-wide settings (trust_remote_code, distributed_executor_backend, dtype, quantization, prefix/chunked prefill, DP/PP sizes) are declared at the top level and applied to every stage. Unknown keys fall through to engine_extras: and are forwarded to the engine. Frequently used fields are listed below; the source-of-truth schema is StageDeployConfig in vllm_omni/config/stage_config.py.

Note: a stage's devices count must equal its local world size (tensor_parallel_size × data_parallel_size_local × pipeline_parallel_size, falling back to global data_parallel_size when the local size is unset), or num_replicas × that product for a replica pool. A mismatch fails early and names the offending stage. A top-level --tensor-parallel-size is broadcast to every stage, so it can make a single-GPU stage violate this contract ( issue #5003); fix that case with --stage-overrides (set tensor_parallel_size and devices together per stage) or set TP only on the multi-GPU stage.

Connector schema¶

Each entry under top-level connectors: follows this shape:

connectors:
  <connector_name>:
    name: <ConnectorClassName>     # required — class registered in vllm_omni.distributed
    extra:                         # optional — forwarded to the connector's __init__
      <key>: <value>
      # Additional connector-specific options

A stage references a connector by name in its input_connectors / output_connectors:

connectors:
  shm:
    name: SharedMemoryConnector

stages:
  - stage_id: 0
    output_connectors: {to_stage_1: shm}
  - stage_id: 1
    input_connectors:  {from_stage_0: shm}

CLI flags¶

Stage-Based CLI Paradigm¶

The stage-based CLI paradigm facilitates the execution of discrete pipeline stages within isolated processes:

• Stage 0 typically encapsulates the orchestrator and the primary API server. Invocation requires --stage-id 0, --omni-master-address, --omni-master-port, and standard port declarations (e.g., --port).
• Worker Stages operate without a distinct API server (i.e., using --headless), are assigned sequential --stage-id identifiers, and must reference the corresponding --omni-master-address and --omni-master-port parameters to successfully register with Stage 0.

For migrated architectures, the system automatically resolves and loads the bundled deployment YAML. Consequently, the primary execution path does not necessitate the explicit definition of --deploy-config: the example below uses CUDA_VISIBLE_DEVICES=0 for Stage 0 and CUDA_VISIBLE_DEVICES=1 for Stage 1.

CUDA_VISIBLE_DEVICES=0 vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \
    --port 8091 \
    --stage-id 0 \
    --omni-master-address 127.0.0.1 \
    --omni-master-port 26000

CUDA_VISIBLE_DEVICES=1 vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \
    --stage-id 1 \
    --headless \
    --omni-master-address 127.0.0.1 \
    --omni-master-port 26000

When instantiating a custom deployment YAML, append the --deploy-config /path/to/override.yaml directive to all node invocations.

In the context of standard initialization architectures, utilizing the --stage-overrides parameter operates as the optimal methodology for delineating stage-specific tuning from the CLI interface:

vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091 \
    --stage-overrides '{"1": {"gpu_memory_utilization": 0.5}}'

Conversely, in the context of the stage-based CLI paradigm, given that each execution process exclusively instantiates a single pipeline stage, configuration override attributes can be defined uniformly via explicit CLI flags on the corresponding instantiation command, rendering composite --stage-overrides JSON strings unnecessary:

CUDA_VISIBLE_DEVICES=1 vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \
    --stage-id 1 \
    --headless \
    --gpu-memory-utilization 0.5 \
    --omni-master-address 127.0.0.1 \
    --omni-master-port 26000

Precedence¶

From highest to lowest:

1. Per-stage overrides (--stage-overrides JSON)
2. Explicit global CLI flags (--gpu-memory-utilization 0.85, etc.)
3. Platform section (platforms.npu.stages, etc.) on top of the base stages:
4. Overlay YAML (via base_config:) on top of the base YAML
5. Parser defaults

Worked override example¶

Starting from the bundled vllm_omni/deploy/qwen3_omni_moe.yaml:

# vllm_omni/deploy/qwen3_omni_moe.yaml (excerpt)
async_chunk: true
stages:
  - stage_id: 0
    gpu_memory_utilization: 0.9
    max_num_seqs: 32
  - stage_id: 1
    gpu_memory_utilization: 0.7
    max_num_seqs: 16

A user-authored overlay that inherits the base and overrides only stage 1:

# my_overrides.yaml
base_config: /path/to/vllm_omni/deploy/qwen3_omni_moe.yaml
stages:
  - stage_id: 1
    gpu_memory_utilization: 0.5     # smaller GPU

Launched with both an explicit global flag and a per-stage override:

vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091 \
    --deploy-config my_overrides.yaml \
    --max-model-len 16384 \
    --stage-overrides '{"0": {"max_num_seqs": 8}}'

Within the stage-based CLI paradigm, equivalent configuration parameters can inherently be passed directly as command-line arguments to the designated single-stage process instantiation:

CUDA_VISIBLE_DEVICES=0 vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \
    --stage-id 0 \
    --max-num-seqs 8 \
    --omni-master-address 127.0.0.1 \
    --omni-master-port 26000

Effective config per stage after the merge:

Therefore, as a core part of vLLM-Omni, a model's pipeline and deployment configurations have several main functions:

• Claim partition of stages and their corresponding class implementation in model_executor/models.
• The disaggregated configuration for each stage and the communication topology among them.
• Engine arguments for each engine within the stage.
• Input and output dependencies for each stage.
• Default input parameters.

To override specific parameters, explicitly inject the customized configuration schema in both online and offline instantiation flows. Use the --deploy-config flag when loading a deploy configuration.

Examples:

For offline inference (assuming the necessary dependencies have been imported):

model_name = "Qwen/Qwen2.5-Omni-7B"
omni = Omni(model=model_name, deploy_config="/path/to/deploy_config.yaml")

For online serving:

vllm serve Qwen/Qwen2.5-Omni-7B --omni --port 8091 --deploy-config /path/to/deploy_config.yaml

Qwen3-TTS with Model Runner V2¶

Qwen3-TTS runs the native CUDA Model Runner V2 pipeline by default on vLLM 0.29.0. MRV2 is an experimental feature for this model: the bundled default profile selects it on CUDA only, and its scheduler and delivery paths are still being qualified. Set model_runner: v1 in a copy of the deploy config to opt out; the platforms: sections of qwen3_tts.yaml keep V1 on NPU, XPU, ROCm and MUSA. Select one of these deployment profiles:

# MRV2 is the default; pass a copy with `model_runner: v1` to force V1.
vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-Base --omni \
  --deploy-config /path/to/qwen3_tts_v1.yaml

The V2 profiles bound Talker prefill to 512 tokens per step and select the Talker AR runner and the Code2Wav generation runner together. Native inter-stage delivery carries codec payloads directly; request-owned snapshots preserve buffers through asynchronous completion and CUDA graph reuse. Terminal completion waits for upstream stage metrics before releasing request state. Platform sections retain V1 on NPU, XPU, ROCm and MUSA; this change does not qualify MRV2 on those backends or enable other model families. Selecting V2 for a stage without supports_native_mrv2_data_plane emits a warning when pipeline and deployment settings are merged. That stage retains the legacy transport path; the warning does not establish support for that combination.

MRV2 model hooks use capability declarations rather than architecture or stage names. Omni lifecycle flags select the model state, and _returns_tuple declares the capture output contract. The optional batched predictor hook is mtp, with an explicit mtp_output_key (a string or two-part payload key), optional mtp_validity_key, and mtp_graph_safe/mtp_disable_graph capture controls. Its inputs are token IDs, embeddings, previous hidden states and per-row conditioning; it returns updated embeddings and prediction codes. Models may supply mtp_sampling_params and get_mtp_seed(sampling_params) for model-local explicit seeds, and declare mtp_accepts_per_row_generators, mtp_accepts_req_infos, or mtp_sample_uniforms (with mtp_sample_steps and mtp_sample_vocab_size) as needed. Qwen3-TTS retains its existing talker_mtp entry point for V1.

Included performance work¶

• Request snapshots have a fast path for immutable scalar leaves, including waveform lists. Mutable containers, aliases and cycles keep deep-copy semantics. This is not the historical shallow dict(prompt) experiment.
• The API reference cache holds owned float32 arrays with a default capacity of 1024 entries and a 512 MiB waveform-payload budget. Cache hits return independent lists. Qwen3-TTS model artifact caches also default to 1024 entries. These caches have different owners and eviction policies; equal entry limits do not make them a single coherent cache.
• Talker state stays on GPU where its lifecycle allows it. BOS/EOS projections are cached in their projection dtype and invalidated on weight loading.
• Code2Wav packs CPU codec inputs before device transfer. Optional B2/B4 graph buckets batch compatible requests without sharing their per-request state.
• Native output materialization runs in a bounded worker and drains before closing the data plane. The existing control-only shortcut avoids launching model work for a step that has only lifecycle events.

MTP prefix re-prefill remains disabled in the high-concurrency V2 profile. Adding more graph shapes increases compilation cost and has not established a stable end-to-end gain for this extracted version.

Decoder batches and playback¶

Changing decode_cudagraph_batch_sizes selects the captured batch buckets. Keep decode_batch_max_size consistent with the intended maximum too; the stateful graph path currently groups according to the captured buckets, while the stateless path also uses the explicit maximum.

B4 remains experimental. First-packet latency alone does not establish uninterrupted playback. Validate inter-chunk arrival times, buffering, WER and speaker similarity before adopting either batching preset for a production workload. Floating-point decoder outputs can differ across batch sizes; this PR does not claim bitwise or quality equivalence.

Optional MPS deployment¶

NVIDIA MPS is an optional operator setting for colocated CUDA processes, not a YAML option or a library default. This PR does not establish a throughput or first-packet latency benefit from MPS. Measure the exact deployment with and without MPS before enabling it.

Use only assigned GPUs and an independent MPS pipe directory. A private MPS server does not provide exclusive GPU ownership or MIG isolation. For a single-GPU deployment, explicitly place both stages on that GPU; the supplied high-concurrency profile places its two stages on different GPUs by default.