Skip to content

vLLM-Omni Benchmark CLI Guide

The vllm bench command launches the vLLM-Omni benchmark to evaluate the performance of multimodal models.

Notes

vLLM-Omni registers the openai-chat-omni, openai-audio-speech, openai-image-edits-omni, daily-omni, and openai-realtime-duplex serving benchmark backends. It also adds the omniinteract dataset.

Basic Parameter Description

You can use vllm bench serve --omni --help=all to get descriptions of all parameters. The commonly used parameters are described below:

  • --omni Enable Omni (multimodal) mode, supporting multimodal inputs and outputs such as images, videos, and audio.

  • --backend Specify the backend adapter. vLLM-Omni adds openai-chat-omni, openai-audio-speech, openai-image-edits-omni, daily-omni, and openai-realtime-duplex to the upstream vLLM backend choices.

  • --model The model identifier to load, filled according to the models supported by vLLM-Omni.

  • --endpoint The API endpoint exposed externally, to which clients send their requests.

  • --dataset-name The name of the dataset used; random-mm generates random multimodal inputs (images, videos, audio), omniinteract replays official OmniInteract videos, and videomme loads Video-MME MCQ videos (lmms-eval/Video-MME, MiniCPM OmniEvalKit packing).

  • --num-prompts The total number of requests to send, an integer.

  • --max-concurrency "Maximum number of concurrent requests. This can be used " "to help simulate an environment where a higher level component " "is enforcing a maximum number of concurrent requests. While the " "--request-rate argument controls the rate at which requests are " "initiated, this argument will control how many are actually allowed " "to execute at a time. This means that when used in combination, the " "actual request rate may be lower than specified with --request-rate, " "if the server is not processing requests fast enough to keep up."

  • --request-rate "Number of requests per second. If this is inf, " "then all the requests are sent at time 0. " "Otherwise, we use Poisson process or gamma distribution " "to synthesize the request arrival times."

  • --ignore-eos "Set ignore_eos flag when sending the benchmark request."

  • --metric-percentiles Comma-separated list of percentiles for selected metrics. " "To report 25-th, 50-th, and 75-th percentiles, use \"25,50,75\". " "Default value is \"99\"." "Use \"--percentile-metrics\" to select metrics.

  • --percentile-metrics "Comma-separated list of selected metrics to report percentiles." "This argument specifies the metrics to report percentiles." 'Allowed metric names are "ttft", "tpot", "itl", "ttfc", "tpoc", "icl", ' '"tpop", "e2el", "audio_ttfp", "audio_rtf", "audio_duration". '

  • --print-stage Print per-stage benchmark metrics for --omni serving when stage metrics are returned by the server. Disabled by default.

  • --save-result Specify to save benchmark results to a json file

  • --save-detailed "When saving the results, whether to include per request " "information such as response, error, ttfs, tpots, etc."

  • --result-dir "Specify directory to save benchmark json results." "If not specified, results are saved in the current directory."

  • --result-filename "Specify the filename to save benchmark json results." "If not specified, results will be saved in " "{label}-{args.request_rate}qps-{base_model_id}-{current_dt}.json"

  • --random-prefix-len Number of fixed prefix tokens before the random context in a request. The total input length is the sum of random-prefix-len and a random context length sampled from [input_len (1 - range_ratio), input_len (1 + range_ratio)].Only the random and random-mm modes support this parameter.

  • --random-input-len Number of input tokens per request.Only the random and random-mm modes support this parameter.

  • --random-output-len Number of output tokens per request.Only the random and random-mm modes support this parameter.

  • --random-range-ratio Range ratio for sampling input/output length, used only for random sampling. Must be in the range [0, 1) to define a symmetric sampling range [length (1 - range_ratio), length (1 + range_ratio)]. Only the random and random-mm modes support this parameter.

  • --random-mm-base-items-per-request Base number of multimodal items per request for random-mm. Actual per-request count is sampled around this base using --random-mm-num-mm-items-range-ratio. Only the random-mm mode supports this parameter.

  • --random-mm-limit-mm-per-prompt Per-modality hard caps for items attached per request, e.g. '{"image": 3, "video": 1, "audio": 1}'. The sampled per-request item count is clamped to the sum of these limits. When a modality reaches its cap, its buckets are excluded and probabilities are renormalized. Only the random-mm mode supports this parameter.

  • --random-mm-num-mm-items-range-ratio Range ratio r in [0, 1] for sampling items per request. We sample uniformly from the closed integer range [floor(n(1-r)), ceil(n(1+r))] where n is the base items per request. r=0 keeps it fixed; r=1 allows 0 items. The maximum is clamped to the sum of per-modality limits from --random-mm-limit-mm-per-prompt. An error is raised if the computed min exceeds the max. Only the random-mm mode supports this parameter.

  • --random-mm-bucket-config The bucket config is a dictionary mapping a multimodal item sampling configuration to a probability. Currently allows for 3 modalities: audio, images and videos. A bucket key is a tuple of (height, width, num_frames) The value is the probability of sampling that specific item. Example: --random-mm-bucket-config "{(256, 256, 1): 0.5, (720, 1280, 16): 0.4, (0, 1, 5): 0.10}" First item: images with resolution 256x256 w.p. 0.5 Second item: videos with resolution 720x1280 and 16 frames Third item: audios with 1s duration and 5 channels w.p. 0.1 OBS.: If the probabilities do not sum to 1, they are normalized. Only the random-mm mode supports this parameter

Usage Examples

Online Benchmark

Show more

First start serving your model:

vllm serve Qwen/Qwen2.5-Omni-7B --omni

Then run the benchmarking for sharegpt:

# download dataset
# wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json
vllm bench serve \
  --omni \
  --port 43845 \
  --model /home/models/Qwen/Qwen3-Omni-30B-A3B-Instruct \
  --endpoint /v1/chat/completions \
  --backend openai-chat-omni \
  --num-prompts 2 \
  --dataset-name sharegpt \
  --dataset-path ShareGPT_V3_unfiltered_cleaned_split.json \
  --percentile-metrics ttft,tpot,itl,e2el

If successful, you will see the following output:

============ Serving Benchmark Result ============
Successful requests:                     2
Failed requests:                         0
Benchmark duration (s):                  81.63
Request throughput (req/s):              0.02
Peak concurrent requests:                2.00
----------------End-to-end Latency----------------
Mean E2EL (ms):                          56966.13
Median E2EL (ms):                        56966.13
P99 E2EL (ms):                           81016.80
================== Text Result ===================
Total input tokens:                      36
Total generated tokens:                  5926
Output token throughput (tok/s):         72.60
Peak output token throughput (tok/s):    103.00
Peak concurrent requests:                2.00
Total Token throughput (tok/s):          73.04
---------------Time to First Token----------------
Mean TTFT (ms):                          124.76
Median TTFT (ms):                        124.76
P99 TTFT (ms):                           156.10
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          481.30
Median TPOT (ms):                        481.30
P99 TPOT (ms):                           947.55
---------------Inter-token Latency----------------
Mean ITL (ms):                           25.11
Median ITL (ms):                         0.33
P99 ITL (ms):                            25.17
================== Audio Result ==================
Total audio duration generated(s):       3.95
Total audio frames generated:            94890
Audio throughput(audio duration/s):      0.05
==================================================

Or run the benchmarking for random:

vllm bench serve \
  --omni \
  --port 43845 \
  --endpoint /v1/chat/completions \
  --backend openai-chat-omni \
  --model /home/models/Qwen/Qwen3-Omni-30B-A3B-Instruct \
  --dataset-name random \
  --num-prompts 2 \
  --random-prefix-len 5 \
  --random-input-len 10 \
  --random-output-len 100 \
  --percentile-metrics ttft,tpot,itl,e2el,audio_ttfp,audio_rtf \
  --ignore-eos

If successful, you will see the following output:

============ Serving Benchmark Result ============
Successful requests:                     2
Failed requests:                         0
Benchmark duration (s):                  3.89
Request throughput (req/s):              0.51
Peak concurrent requests:                2.00
----------------End-to-end Latency----------------
Mean E2EL (ms):                          3824.76
Median E2EL (ms):                        3824.76
P99 E2EL (ms):                           3888.54
================== Text Result ===================
Total input tokens:                      30
Total generated tokens:                  10101
Output token throughput (tok/s):         2595.57
Peak output token throughput (tok/s):    111.00
Peak concurrent requests:                2.00
Total Token throughput (tok/s):          2603.28
---------------Time to First Token----------------
Mean TTFT (ms):                          117.15
Median TTFT (ms):                        117.15
P99 TTFT (ms):                           142.69
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          0.73
Median TPOT (ms):                        0.73
P99 TPOT (ms):                           0.74
---------------Inter-token Latency----------------
Mean ITL (ms):                           16.47
Median ITL (ms):                         16.19
P99 ITL (ms):                            52.55
================== Audio Result ==================
Total audio duration generated(s):       15.79
Total audio frames generated:            379050
Audio throughput(audio duration/s):      4.06
---------------Time to First Packet---------------
Mean AUDIO_TTFP (ms):                    3701.37
Median AUDIO_TTFP (ms):                  3701.37
P99 AUDIO_TTFP (ms):                     3762.25
-----------------Real Time Factor-----------------
Mean AUDIO_RTF:                          0.47
Median AUDIO_RTF:                        0.47
P99 AUDIO_RTF:                           0.48
==================================================

Notes: We use audio generation time / audio duration to calculate RTF.

OmniInteract Realtime Benchmark

OmniInteract runs each video as one native-duplex WebSocket sample in the standard serving-benchmark lifecycle:

vllm bench serve --omni \
  --backend openai-realtime-duplex \
  --dataset-name omniinteract \
  --dataset-path /path/to/OmniInteract \
  --model openbmb/MiniCPM-o-4_5 \
  --base-url http://127.0.0.1:8000 \
  --endpoint /v1/realtime \
  --omniinteract-ref-audio /path/to/reference.wav \
  --omniinteract-output-dir ./omniinteract-artifacts \
  --num-prompts 3 \
  --num-warmups 0

--dataset-path accepts an extracted directory, data.tar[.gz], or a Hugging Face dataset ID; omitting it uses lucky-lance/OmniInteract. --num-prompts is the total across subsets and defaults to 3 for OmniInteract; explicit 0 selects all and oversize values use all available cases. Reference audio is required, and OmniInteract uses the /v1/realtime endpoint.

To replay an existing sample set (for example sampled_cases.jsonl from a prior run), pass --omniinteract-video-list instead of discovering cases. List order is preserved; --num-prompts takes the prefix of the list (0 or a value larger than the list length runs every row). Do not combine video-list mode with --seed, --dataset-path, --omniinteract-scenario-tags, or --omniinteract-scenario-focus.

Use --omniinteract-scenario-tags to steer small runs toward paper-table scenarios (realtime, proactive, nested, interrupted, 1qna). By default the sampler covers each requested tag with at least one matching case (when available under --omniinteract-subsets), then fills the remaining --num-prompts budget from other cases. Add --omniinteract-scenario-focus to run only cases that match those tags. Example smoke coverage:

  --num-prompts 5 \
  --omniinteract-scenario-tags realtime proactive nested interrupted 1qna

Audio is replayed as 16 kHz PCM16 in 200 ms chunks and video at 1 FPS with real-time pacing. All selected media is decoded before timing and remains in client memory for the run, so --max-concurrency does not limit media preparation memory; use explicit --num-prompts 0 only when the client has enough RAM for the full dataset. Media commands are bounded by --omniinteract-media-timeout-s, and concurrency defaults to 1. Standard request-rate, warmup, result-saving, and summary options apply. Use --omniinteract-require-response only for functional E2E cases; LISTEN is a valid benchmark result.

Each completed case writes output.wav, wav_transcript.json, events.json, result.json, and a final .done marker under --omniinteract-output-dir. The root also contains batch_summary.json, official_eval_manifest.jsonl (eligible cases only), and sampled_cases.jsonl (every sampled case as absolute video_path / official output_name / subset rows for Lucky-Lance OmniInteract batch_inference_minicpmo.py --video_list); failed cases write .failed.json. Runs sharing one output root are serialized. Completion validates transport, response lifecycle, and artifacts, not answer accuracy. Transcript timestamps are serialized playback-queue times. Playback ACKs report cumulative progress incrementally along that serialized clock, like a live listener; the first ack for a response goes out as soon as its audio arrives, checkpointing the response's history position so a later committed user input updates it in place. A residual playback_ack_too_late rejection is recorded as an artifact warning rather than failing the case. Clipped or cancelled outputs are ineligible and omitted from the official manifest; audio_clipped_bytes records output beyond the rounded video horizon.

Accuracy evaluation is opt-in and requires an already-running text judge with an OpenAI-compatible Chat Completions API. The benchmark does not launch or stop the judge server. Add the following options to the command above:

vllm serve Qwen/Qwen2.5-7B-Instruct \
  --served-model-name Qwen2.5-7B-Instruct \
  --port 8001

Then add to the benchmark command:

  --omniinteract-evaluate \
  --omniinteract-judge-base-url http://127.0.0.1:8001 \
  --omniinteract-judge-model Qwen2.5-7B-Instruct

--omniinteract-judge-model must match that judge process's --served-model-name (or its --model string if --served-model-name is omitted). It is not a checkpoint path to load, and it is not the Omni DUT --served-model-name. Point --omniinteract-judge-base-url at the judge, not at the Omni realtime endpoint.

The early / core / interrupted-partial judge prompts follow the official English templates in Lucky-Lance/OmniInteract eval/evaluation/llm_judge.py (commit de304cef35fd9a50a5caadb5090c34cfbf0dd868). They correspond to the OmniInteract paper appendix (arXiv:2605.26485) Listing A.1 (early-stage), Listing A.2 (interrupted partial quality), and Listing A.3 (core-stage). A local OpenAI-compatible judge is not the paper's GPT-4o judge, so reported IA-QTF1 numbers are protocol-compatible rather than official paper-table scores.

After artifact publication, the evaluator builds [start, t_a, end) slots, judges early and core response text, writes per-case details plus evaluation/unified_eval_summary.json, and prints IA-QTF1 columns (1Q1A realtime / proactive / nested / Global, 1QnA, All Global), interruption diagnostics (NOR / PAQ / CSM), and nested metrics (NCCS, inner / outer IA-QTF1, missed outer). Slices and sections with no slots in the evaluated sample are omitted from the terminal report. The header lists how many evaluated cases carry each scenario tag (realtime, proactive, nested, interrupted, 1qna). Realtime and proactive exclude nested inner/outer slots; 1Q1A Global recomputes F1 from those three TP/FP/FN aggregates; All Global includes every scored slot plus unmatched-chunk false positives. Plain wav_transcript.json timestamps provide chunk-level approximate timing. If transcript chunks include aligned_words, the evaluator splits boundary-crossing chunks and derives trigger timing from word alignment.

Per-response TTFT and TTFP start when the server begins executing the native model-turn request that owns the response. RTF continues to use client receipt of response.created through the last audio packet, divided by emitted audio duration. Global TTFT, TTFP, and RTF cover the complete input-stream window. TPOT/ITL use engine stage-0 timing; ITL is emitted only when every token interval is present within a continuous generation segment. Model-unit pacing and gaps between generation segments are excluded from TPOT/ITL. Raw request metrics retain response_created_to_first_text_ms and response_created_to_first_audio_ms as client-envelope diagnostics.

The checked-in local performance configuration measures four deterministic cases from 1q1a, 1q1a_math, and 1qna (12 videos total), with no benchmark warmups and a maximum concurrency of two. Each subset also sends one readiness request before its measured cases. The runner starts MiniCPM-o 4.5 on the first visible GPU and a text judge (Qwen/Qwen2.5-7B-Instruct) on the second (CUDA_VISIBLE_DEVICES=1 relative to the process). After artifacts land it scores All Global IA-QTF1 through that local judge. This is protocol-compatible with the paper metric, not a GPT-4o paper-table score. Run it from the repository root with two visible GPUs:

export HF_HOME=/path/to/persistent/huggingface-cache
export BENCHMARK_DIR=tests/dfx/perf/results
bash tools/nightly/run_nightly_jobs.sh \
  --test-type local \
  --model-type omni \
  --label-substr minicpmo_4_5_omniinteract

The first run downloads the pinned OmniInteract archive and the judge weights into HF_HOME; later runs reuse that cache.

It requires every case to commit its input, complete any emitted response lifecycles, and publish the expected WAV, transcript, event, and result artifacts without errors. A valid LISTEN-only case may have no response audio or transcript chunks. Official-manifest eligibility is reported separately because clipped or cancelled output is a benchmark-quality signal, not a transport failure. Accuracy must finish with status=ok on every subset. After all three subsets finish, All Global IA-QTF1 is recomputed from pooled Global_TP / Global_FP / Global_FN and must be at or above omniinteract_aggregate_min_ia_qtf1 (checked in as 0.2).

Video-MME Benchmark

Video-MME (--dataset-name videomme) scores multiple-choice video QA. Default packing is OmniEvalKit MiniCPM minicpm-frames (up to 96 sampled frames as image_url). Pass a local mirror with --dataset-path / --videomme-parquet + --videomme-video-dir, or a Hugging Face dataset id (lmms-eval/Video-MME by default; any org/name is accepted when --dataset-name videomme is explicit). Relative --videomme-video-dir values are resolved to absolute file:// URLs. For those URLs, start the server with --allowed-local-media-path covering the video root; otherwise use --videomme-inline-local-video.

For MiniCPM-o 4.5, the default deployment allows 64 images per request. To run the 96-frame recipe, start the server with a stage 0 override (replace /path/to/Video-MME with the local video root shared by the benchmark client and server):

vllm serve openbmb/MiniCPM-o-4_5 --omni \
  --allowed-local-media-path /path/to/Video-MME \
  --stage-overrides '{"0":{"limit_mm_per_prompt":{"image":96,"audio":64,"video":1}}}'

The override applies to this server invocation only. Alternatively, use --videomme-max-frames 64 in the benchmark to stay within the default image limit; this evaluates a different frame-sampling setting from the 96-frame recipe.

Run the benchmark against that server:

vllm bench serve --omni \
  --backend openai-chat-omni \
  --dataset-name videomme \
  --dataset-path /path/to/Video-MME \
  --videomme-pack-mode minicpm-frames \
  --videomme-max-frames 96 \
  --model openbmb/MiniCPM-o-4_5 \
  --endpoint /v1/chat/completions \
  --num-prompts 8 \
  --max-concurrency 1 \
  --save-result

Accuracy keys (videomme_accuracy, per-duration / domain / task breakdowns) are written into the saved JSON. videomme_accuracy excludes HTTP failures; videomme_accuracy_incl_http_fail counts them as wrong. videomme_submitted / videomme_unique_question_ids show when oversampling made the request count differ from unique-question coverage. videomme_skipped_rows counts rows skipped during sampling because of missing or unreadable media or invalid question fields. The opt-in accuracy runner rejects skipped rows, HTTP failures, duplicate question IDs, and missing gold answers. This checks the sampled subset; it does not establish that a supplied local mirror contains the entire official dataset. Use --videomme-save-eval-items (or VIDEOMME_SAVE_EVAL_ITEMS=1) for per-request videomme_eval_items rows.

Multi-Modal Benchmark

Show more

Benchmark the performance of multi-modal requests in vLLM-Omni.

Generate synthetic image、video、audio inputs alongside random text prompts to stress-test vision models without external datasets.

Notes:

  • Works only with online benchmark via:
    • the OpenAI chat backend (--backend openai-chat-omni) and endpoint /v1/chat/completions.
    • the OpenAI edit image backend (--backend openai-image-edits-omni) and endpoint /v1/images/edits.

Start the server (example):

vllm serve Qwen/Qwen2.5-Omni-7B --omni

It is recommended to use the flag --ignore-eos to simulate real responses. You can set the size of the output via the arg random-output-len.

Then run the benchmarking script:

vllm bench serve \
    --omni \
  --dataset-name random-mm \
  --port 40849 \
  --model /home/models/Qwen/Qwen3-Omni-30B-A3B-Instruct \
  --endpoint /v1/chat/completions \
  --backend openai-chat-omni \
  --request-rate 1 \
  --num-prompts 1 \
  --random-input-len 10 \
  --random-range-ratio 0.0 \
  --random-mm-base-items-per-request 2 \
  --random-mm-num-mm-items-range-ratio 0 \
  --random-mm-limit-mm-per-prompt '{"image":1,"video":1, "audio": 1}' \
  --random-mm-bucket-config '{"(32, 32, 1)": 0.5, "(0, 1, 1)": 0.1, "(32, 32, 2)":0.4}' \
  --ignore-eos \
  --percentile-metrics ttft,tpot,itl \
  --random-output-len 2 \
  --extra-body '{"modalities": ["text"]}'

If successful, you will see the following output:

============ Serving Benchmark Result ============
Successful requests:                     1
Failed requests:                         0
Request rate configured (RPS):           1.00
Benchmark duration (s):                  1.21
Request throughput (req/s):              0.83
Peak concurrent requests:                1.00
================== Text Result ===================
Total input tokens:                      10
Total generated tokens:                  3
Output token throughput (tok/s):         2.49
Peak output token throughput (tok/s):    3.00
Peak concurrent requests:                1.00
Total Token throughput (tok/s):          10.77
---------------Time to First Token----------------
Mean TTFT (ms):                          179.74
Median TTFT (ms):                        179.74
P99 TTFT (ms):                           179.74
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          12.76
Median TPOT (ms):                        12.76
P99 TPOT (ms):                           12.76
---------------Inter-token Latency----------------
Mean ITL (ms):                           12.76
Median ITL (ms):                         12.76
P99 ITL (ms):                            25.24
================== Audio Result ==================
Total audio duration generated(s):       0.00
Total audio frames generated:            0
Audio throughput(audio duration/s):      0.00
==================================================

Behavioral notes:

  • If the requested base item count cannot be satisfied under the provided per-prompt limits, the tool raises an error rather than silently clamping.

How sampling works:

  • Determine per-request item count k by sampling uniformly from the integer range defined by --random-mm-base-items-per-request and --random-mm-num-mm-items-range-ratio, then clamp k to at most the sum of per-modality limits.
  • For each of the k items, sample a bucket (H, W, T) according to the normalized probabilities in --random-mm-bucket-config, while tracking how many items of each modality have been added.
  • If a modality (e.g., image) reaches its limit from --random-mm-limit-mm-per-prompt, all buckets of that modality are excluded and the remaining bucket probabilities are renormalized before continuing. This should be seen as an edge case, and if this behavior can be avoided by setting --random-mm-limit-mm-per-prompt to a large number. Note that this might result in errors due to engine config --limit-mm-per-prompt.
  • The resulting request contains synthetic image data in multi_modal_data (OpenAI Chat format). When random-mm is used with the OpenAI Chat backend, prompts remain text and MM content is attached via multi_modal_data.

Image / video generation

For /v1/images/generations, /v1/images/edits, and /v1/videos, set --endpoint to that path and omit --backend. The endpoint path is registered as the bench request adapter, so a separate named backend is not required.

Example (/v1/images/generations):

vllm bench serve --omni \
  --endpoint /v1/images/generations \
  --dataset-name random \
  --model ~/models/Qwen/Qwen-Image \
  --tokenizer ~/models/Qwen/Qwen-Image/tokenizer \
  --max-concurrency 1 \
  --num-warmups 2 \
  --num-prompts 10 \
  --random-input-len 64 \
  --random-output-len 1 \
  --ignore-eos \
  --percentile-metrics e2el \
  --extra-body '{
    "num_inference_steps": 20,
    "seed": 42,
    "true_cfg_scale": 4.0
  }'

If successful, following output is like:

============ Serving Benchmark Result ============
Successful requests:                     3  
Failed requests:                         0  
Maximum request concurrency:             1  
Benchmark duration (s):                  8.42  
Request throughput (req/s):              0.36  
Peak concurrent requests:                2.00  
-------------------Peak Memory--------------------
Mean PEAK_MEMORY_MB (MB):                58832.00  
Median PEAK_MEMORY_MB (MB):              58832.00  
P99 PEAK_MEMORY_MB (MB):                 58832.00  
----------------End-to-end Latency----------------
Mean E2EL (ms):                          2798.16  
Median E2EL (ms):                        2799.83  
P99 E2EL (ms):                           2800.03  
================== Image Result ==================
Total images generated:                  3  
Image throughput (img/s):                0.36  
Average pixels per image:                1048576.00
Mean denoise step latency (ms):          136.59  
---------------- Image Generation ----------------
Mean IMAGE_GENERATION (ms):              2731.84  
Median IMAGE_GENERATION (ms):            2732.02  
P99 IMAGE_GENERATION (ms):               2734.65  
==================================================

Use the same pattern with --endpoint /v1/images/edits or --endpoint /v1/videos (and model-specific --extra-body as needed). Pure image/video runs omit the Text Result section when there is no generated text.

/v1/images/edits defaults to non-streaming JSON (stream=false) so single-stage edit models are not rejected by the server. For multi-stage pipelines that need SSE (AR TTFT / image chunks), pass "stream": true in --extra-body.

/v1/videos is an async job API: the client creates a job, then polls until completed/failed. Measured e2el therefore includes client poll sleep and any overshoot after the job actually finishes. Tune polling via --extra-body:

  • poll_interval_s (default 2.0): sleep between status polls
  • poll_timeout_s (default 21600, i.e. 6 hours): give up waiting for the job

VIDEO_RTF prefers server-reported generation time when available:

  • video_rtf = video_generation_time_ms / 1000 / video_duration, where video_generation_time_ms comes from response stage_durations (or inference_time_s as fallback)
  • If generation time is missing, falls back to e2el / video_duration (this fallback does include poll overhead)

Multi-Stage Benchmark

Show more

Start the server (change the port and model):

vllm serve --omni --port 29999 --model ~/Qwen3-Omni-30B-A3B-Instruct

Then run the benchmarking script (remember to add the flag --print-stage):

vllm bench serve --omni \
  --dataset-name random \
  --port 29999 \
  --max-concurrency 1 \
  --num-warmups 2 \
  --model ~/Qwen3-Omni-30B-A3B-Instruct \
  --endpoint /v1/chat/completions \
  --backend openai-chat-omni \
  --num-prompts 3 \
  --random-input-len 2500 \
  --ignore-eos \
  --percentile-metrics ttft,tpot,itl,e2el,audio_ttfp,audio_rtf,ttfc,tpoc,icl \
  --random-output-len 900 \
  --extra-body '{"modalities": ["text","audio"]}' \
  --print-stage

Besides the "Serving Benchmark Result", there will be "Stage Benchmark Result" below:

============= Stage Benchmark Result =============
=============== Stage 0 (thinker) ================
-------------------Stage Timing-------------------
Mean stage_gen_time (ms):                18243.16
Median stage_gen_time (ms):              18220.83
P99 stage_gen_time (ms):                 18296.75
================== Text Result ===================
Stage generated tokens:                  2700
-----------Serving Time to First Token------------
Mean Serving TTFT (ms):                  816.31
Median Serving TTFT (ms):                818.94
P99 Serving TTFT (ms):                   829.76
-----Time per Output Token (excl. 1st token)------
Mean TPOP (ms):                          19.41
Median TPOP (ms):                        19.40
P99 TPOP (ms):                           19.47
---------------Inter-token Latency----------------
Mean ITL (ms):                           19.41
Median ITL (ms):                         19.12
P99 ITL (ms):                            23.58
================ Stage 1 (talker) ================
-------------------Stage Timing-------------------
Mean stage_gen_time (ms):                18849.64
Median stage_gen_time (ms):              13641.15
P99 stage_gen_time (ms):                 33655.58
============= Internal Stream Result =============
-----------Serving Time to First Chunk------------
Mean Serving TTFC (ms):                  1029.14
Median Serving TTFC (ms):                1027.98
P99 Serving TTFC (ms):                   1032.59
-----Time per Output Chunk (excl. 1st chunk)------
Mean TPOP (ms):                          17.53
Median TPOP (ms):                        18.70
P99 TPOP (ms):                           18.88
---------------Inter-chunk Latency----------------
Mean ICL (ms):                           16.27
Median ICL (ms):                         18.19
P99 ICL (ms):                            25.86
=============== Stage 2 (code2wav) ===============
-------------------Stage Timing-------------------
Mean stage_gen_time (ms):                18860.74
Median stage_gen_time (ms):              13655.84
P99 stage_gen_time (ms):                 33659.79
================== Audio Result ==================
Stage audio duration generated(s):       260.11
Stage audio frames generated:            6242730
-----------Serving Time to First Packet-----------
Mean Serving AUDIO_TTFP (ms):            1366.16
Median Serving AUDIO_TTFP (ms):          1367.20
P99 Serving AUDIO_TTFP (ms):             1368.26
-----------------Real Time Factor-----------------
Mean AUDIO_RTF:                          0.24
Median AUDIO_RTF:                        0.26
P99 AUDIO_RTF:                           0.27

Explanation:

  • stage_gen_time: Time from submitting a request to a specific stage to that stage finishing generation.

  • Serving TTFC (Time to First Chunk): Time from the HTTP request being accepted by the serving frontend to the stage producing its first non-empty output chunk. Keep the name TTFT in text stage, TTFP in audio stage for easier mapping.

  • TPOP (Time per Output Chunk): Average time from the first output chunk to stage completion, divided by the number of remaining output chunks. The TPOP abbreviation follows Qwen3.5-Omni Technical Report.

  • ICL (Inter-Chunk Latency): Time between two consecutive output chunks produced by the same stage. Keep the name ITL in text stage for easier mapping.