Speech API¶

vLLM-Omni provides an OpenAI-compatible API for text-to-speech (TTS) generation. Supported TTS models include:

• Qwen3-TTS (Qwen/Qwen3-TTS-12Hz-*) -- Qwen3-based TTS with CustomVoice, VoiceDesign, and Base (voice cloning) task types. Native output: 24 kHz; API output can be resampled to 8 kHz.
• Fish Speech S2 Pro (fishaudio/s2-pro) -- Dual-AR TTS with DAC codec. Supports text-to-speech and voice cloning via reference audio. Output: 44.1 kHz.
• Voxtral TTS (mistralai/Voxtral-4B-TTS-2603) -- AR + FlowMatching TTS with preset voices. Output: 24 kHz.
• CosyVoice3 (FunAudioLLM/Fun-CosyVoice3-0.5B-2512) -- 2-stage talker + flow-matching code2wav. Voice cloning via ref_audio + ref_text (no presets). Output: 24 kHz.
• Gepard-1.0 (nineninesix/gepard-1.0) -- single-stage native-AR TTS with a 22.05 kHz NanoCodec. Zero-shot default voice only.

See the Supported Models section below for the full list, including OmniVoice, VoxCPM2, MOSS-TTS-Nano, and Breeze-TTS-2.

Each server instance runs a single model (specified at startup via vllm serve <model> --omni).

Quick Start¶

Start the Server¶

# Qwen3-TTS: CustomVoice model (predefined speakers)
vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice \
    --deploy-config vllm_omni/deploy/qwen3_tts.yaml \
    --omni \
    --port 8091 \
    --trust-remote-code \
    --enforce-eager

# Fish Speech S2 Pro
vllm serve fishaudio/s2-pro --omni --port 8091

# Voxtral TTS
vllm serve mistralai/Voxtral-4B-TTS-2603 --omni --port 8091

# CosyVoice3 (voice cloning only — supply ref_audio + ref_text per request)
vllm serve FunAudioLLM/Fun-CosyVoice3-0.5B-2512 \
    --omni --port 8091 --trust-remote-code

# Gepard-1.0 (zero-shot default voice; packaged gepard.yaml)
vllm-omni serve nineninesix/gepard-1.0 --omni --port 8091 --trust-remote-code \
    --stage-init-timeout 900 \
    --deploy-config vllm_omni/deploy/gepard.yaml

Generate Speech¶

Qwen3-TTS CustomVoice, using curl:

curl -X POST http://localhost:8091/v1/audio/speech \
    -H "Content-Type: application/json" \
    -d '{
        "input": "Hello, how are you?",
        "voice": "vivian",
        "language": "English"
    }' --output output.wav

Gepard-1.0, using curl:

Gepard accepts input (required), voice ("default"), response_format, stream / stream_format, max_new_tokens, and seed. Unsupported fields include speed, language, instructions, task_type, ref_audio, ref_text, extra_params, and word_timestamps.

curl -X POST http://localhost:8091/v1/audio/speech \
    -H "Content-Type: application/json" \
    -d '{
        "input": "Hello, this is Gepard speaking.",
        "voice": "default",
        "seed": 7
    }' --output output.wav

Qwen3-TTS CustomVoice, using Python:

import httpx

response = httpx.post(
    "http://localhost:8091/v1/audio/speech",
    json={
        "input": "Hello, how are you?",
        "voice": "vivian",
        "language": "English",
    },
    timeout=300.0,
)

with open("output.wav", "wb") as f:
    f.write(response.content)

Using OpenAI SDK:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8091/v1", api_key="none")

response = client.audio.speech.create(
    model="Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice",
    voice="vivian",
    input="Hello, how are you?",
)

response.stream_to_file("output.wav")

API Reference¶

Endpoint¶

POST /v1/audio/speech
Content-Type: application/json

Request Parameters¶

OpenAI Standard Parameters¶

vLLM-Omni Extension Parameters¶

Supported languages: Only applicable to Qwen3-TTS. Derived from the model configuration (talker_config.codec_language_id in the checkpoint's config.json), plus Auto, which is always accepted. Official Qwen3-TTS checkpoints support: Auto, Chinese, English, Japanese, Korean, German, French, Russian, Portuguese, Spanish, Italian.

Voice Clone Parameters (Base task)¶

Response Format¶

The response shape depends on the streaming parameters:

Non-streaming (default). With stream=false and no stream_format, returns the complete clip as binary audio data with an appropriate Content-Type header (e.g. audio/wav). Because the raw-bytes body has no JSON carrier, successful non-streaming responses from non-diffusion speech servers report usage through response headers:

Diffusion-mode speech servers route through a separate response path and do not emit these headers.

Raw audio stream (stream_format="audio"). Streams raw audio bytes (PCM or WAV) as they are decoded.

Both streaming modes (stream_format="audio" and "sse") require response_format="pcm" or "wav", and speed must be 1.0 (or omitted).

SSE stream (stream=true or stream_format="sse"). Streams OpenAI speech.audio.* Server-Sent Events. Each event has an event: line and a JSON data: line:

• speech.audio.delta — a base64 audio chunk:

  { "type": "speech.audio.delta", "audio": "<base64>", "response_format": "pcm" }
  

• speech.audio.done — terminal event, carrying token usage:

  {
      "type": "speech.audio.done",
      "usage": {
          "input_tokens": 119,
          "output_tokens": 77,
          "total_tokens": 196,
          "input_token_details": { "text_tokens": 18, "audio_tokens": 101 }
      }
  }
  

• speech.audio.error — emitted instead of speech.audio.done if generation fails:

  { "type": "speech.audio.error", "error": { "message": "...", "type": "server_error", "param": null, "code": 500 } }
  

The usage object on speech.audio.done is the same shape returned per item by the batch endpoint:

• input_tokens = text_tokens + audio_tokens
  • text_tokens: tokens of the synthesized text (input plus instructions)
  • audio_tokens: reference-audio codec frames, non-zero only for in-context voice cloning (Base task); 0 for CustomVoice/VoiceDesign or x-vector-only
• output_tokens: generated codec tokens
• total_tokens = input_tokens + output_tokens

Voices Endpoint¶

GET /v1/audio/voices

Lists available voices for the loaded model.

{
    "voices": ["aiden", "dylan", "eric", "ono_anna", "ryan", "serena", "sohee", "uncle_fu", "vivian", "custom_voice_1"],
    "uploaded_voices": [
        {
            "name": "custom_voice_1",
            "consent": "user_consent_id",
            "created_at": 1738660000,
            "file_size": 1024000,
            "mime_type": "audio/wav",
            "ref_text": "The exact transcript of the audio sample.",
            "speaker_description": "warm narrator"
        }
    ]
}

uploaded_voices is always present (empty list when no custom voices have been uploaded). Fields ref_text and speaker_description are omitted per-entry when not provided at upload time.

POST /v1/audio/voices
Content-Type: multipart/form-data

Upload a new voice sample for voice cloning in Base task TTS requests.

Form Parameters:

Response Example:

{
  "success": true,
  "voice": {
    "name": "custom_voice_1",
    "consent": "user_consent_id",
    "created_at": 1738660000,
    "mime_type": "audio/wav",
    "file_size": 1024000,
    "ref_text": "The exact transcript of the audio sample.",
    "speaker_description": "warm narrator"
  }
}

Fields ref_text and speaker_description are omitted when not provided at upload time.

Naming rules:

• Names that collide with one of the model's built-in or precomputed voices are rejected (400). Voice files already on disk under such a name are ignored at startup with a warning, so an upload can never shadow a built-in voice.
• Re-uploading an existing uploaded name overwrites it in place (the previous audio file is deleted). Set VLLM_OMNI_SPEAKER_REGISTRATION_POLICY=immutable on the server to reject duplicates instead, requiring an explicit DELETE /v1/audio/voices/{name} before re-registering — useful when the endpoint is reachable by multiple writers and silent overwrites are a risk.
• The voice registry has no per-user ownership: any client that can reach the endpoint can overwrite (default policy) or delete any uploaded voice. For multi-tenant deployments, add authentication at a proxy and namespace voice names per user (e.g. {user}.{name}).

Usage Example:

curl -X POST http://localhost:8091/v1/audio/voices \
  -F "audio_sample=@/path/to/voice_sample.wav" \
  -F "consent=user_consent_id" \
  -F "name=custom_voice_1" \
  -F "ref_text=The exact transcript of the audio sample." \
  -F "speaker_description=warm narrator"

Streaming Text Input (WebSocket)¶

The /v1/audio/speech/stream WebSocket endpoint accepts text incrementally. By default (split_granularity=none) it buffers until input.done and synthesizes that flush as one TTS request, which keeps long-form timbre stable. Set split_granularity to sentence or clause to emit a request at each detected boundary (lower time-to-first-audio for STT/LLM pipelines).

Note: stream_audio only changes how audio bytes are framed (one WAV/PCM payload vs chunked PCM). Text segmentation is controlled separately by split_granularity.

WebSocket Protocol¶

Client -> Server:

Server -> Client:

Flushing vs. Closing¶

input.done is a flush, not a disconnect. The server synthesizes the buffered text, emits session.done, and then waits on the same connection for the next utterance, so a client that speaks repeatedly (for example one driven by an upstream LLM) pays the WebSocket handshake once instead of once per utterance.

• The session config is sticky. Send input.text again straight after session.done to reuse it, or send another session.config first to change voice, format, or reference audio. A session.config sent in the middle of an utterance is rejected so no pending input is silently dropped and so a split utterance cannot end up half in one voice and half in another.
• An utterance is the flush unit, not a linguistic one: it is one input.done cycle. utterance_index counts those flushes across the connection, so it tells you which input.done a frame belongs to. sentence_index counts the TTS requests inside one flush and so pairs with total_sentences: with the default split_granularity=none that is always sentence_index: 0 of total_sentences: 1 (or 0 for an empty buffer), while sentence or clause counts the linguistic units actually synthesized.
• End the connection with session.close, or by closing the socket. An idle connection is still closed after the server's idle timeout, which now also applies to the gap between utterances.

Session Config Parameters¶

All REST API parameters are supported, plus:

ASCII punctuation only ends a unit when whitespace or input.done follows it, and decimals (3.14), thousands separators (1,000), abbreviations (Dr., e.g.) and initials (J. R.) are not treated as boundaries. A punctuation run and any closing quote or bracket stay with the unit they close, so Wait... and He said "Hello." are one request each.

DELETE /v1/audio/voices/{name}

Delete an uploaded voice sample.

Path Parameters:

Response Example:

{
  "success": true,
  "message": "Voice 'custom_voice_1' deleted successfully"
}

Error Response (404 Not Found):

{
  "success": false,
  "error": "Voice 'unknown_voice' not found"
}

Usage Example:

curl -X DELETE http://localhost:8091/v1/audio/voices/custom_voice_1

Examples¶

CustomVoice with Style Instruction¶

curl -X POST http://localhost:8091/v1/audio/speech \
    -H "Content-Type: application/json" \
    -d '{
        "input": "I am so excited!",
        "voice": "vivian",
        "instructions": "Speak with great enthusiasm"
    }' --output excited.wav

VoiceDesign (Natural Language Voice Description)¶

# Start server with VoiceDesign model first
vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign \
    --deploy-config vllm_omni/deploy/qwen3_tts.yaml \
    --omni \
    --port 8091 \
    --trust-remote-code \
    --enforce-eager

curl -X POST http://localhost:8091/v1/audio/speech \
    -H "Content-Type: application/json" \
    -d '{
        "input": "Hello world",
        "task_type": "VoiceDesign",
        "instructions": "A warm, friendly female voice with a gentle tone"
    }' --output designed.wav

Base (Voice Cloning)¶

# Start server with Base model first
vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-Base \
    --deploy-config vllm_omni/deploy/qwen3_tts.yaml \
    --omni \
    --port 8091 \
    --trust-remote-code \
    --enforce-eager

curl -X POST http://localhost:8091/v1/audio/speech \
    -H "Content-Type: application/json" \
    -d '{
        "input": "Hello, this is a cloned voice",
        "task_type": "Base",
        "ref_audio": "https://example.com/reference.wav",
        "ref_text": "Original transcript of the reference audio",
        "non_streaming_mode": true
    }' --output cloned.wav

Upload Voice¶

Upload voice (speaker embedding only):

curl -X POST http://localhost:8091/v1/audio/voices \
  -F "audio_sample=@/path/to/voice_sample.wav" \
  -F "consent=user_consent_id" \
  -F "name=custom_voice_1"

Upload voice with transcript (in-context cloning, higher quality):

curl -X POST http://localhost:8091/v1/audio/voices \
  -F "audio_sample=@/path/to/voice_sample.wav" \
  -F "consent=user_consent_id" \
  -F "name=custom_voice_2" \
  -F "ref_text=The exact transcript of the audio sample."

Use Uploaded Voice¶

curl -X POST http://localhost:8091/v1/audio/speech \
    -H "Content-Type: application/json" \
    -d '{
        "input": "Hello, this is a cloned voice",
        "voice": "custom_voice_1"
    }' --output cloned.wav

For Qwen3-TTS, an uploaded voice is a Base voice-cloning input. The server infers task_type="Base" when voice names an uploaded entry, so the request must be sent to a Base checkpoint. Built-in presets such as vivian and ryan remain CustomVoice speakers and require a CustomVoice checkpoint.

Voice Storage & Caching¶

Uploaded voices are persisted to disk as a single .safetensors file per voice (audio samples + metadata — name, consent, ref_text, sample_rate, created_at — in the file header). On server restart the directory is scanned and all previously uploaded voices are restored automatically, so uploads survive process restarts.

Uploading an existing name overwrites the previous entry (a warning is logged).

Feature extraction artifacts (ref_code, speaker_embedding, DAC codes, etc.) are cached in-process with a shared LRU so repeated requests with the same voice=... skip the extraction pipeline. The cache is a true singleton across all TTS model types; deleting a voice invalidates every model-type slot at once.

Decoded reference waveforms use a separate LRU cache of owned, contiguous float32 arrays at the source sampling rate. Its byte budget counts numeric buffers (four bytes per mono sample), excluding cache metadata. Numeric storage avoids retaining a Python float object for every sample during garbage collection. MOSS reference encoding consumes these arrays directly; other list-based interfaces, including MOSS Nano, receive temporary lists that are not retained in the resolve cache. The MOSS reference encoder also releases completed batch inputs before waiting for more work.

Precomputed Custom Voices¶

Qwen3-TTS Base and VoxCPM2 can load offline-precomputed voices at startup. Qwen3-TTS precomputed voices follow the same Base task and checkpoint-matching rules as uploaded voices.

Generate a directory containing custom_voice_manifest.json plus one .safetensors file per voice, then set the pipeline-wide deploy config field:

custom_voice_dir: /path/to/custom_voices

Qwen3-TTS profiles are created with:

python examples/online_serving/text_to_speech/qwen3_tts/precompute_custom_voice.py \
  --model Qwen/Qwen3-TTS-12Hz-1.7B-Base \
  --voice-name alice \
  --ref-audio /path/to/reference.wav \
  --ref-text "Original transcript of the reference audio" \
  --mode icl \
  --output-dir /path/to/custom_voices

VoxCPM2 profiles are created with:

python examples/online_serving/text_to_speech/voxcpm2/precompute_custom_voice.py \
  --model openbmb/VoxCPM2 \
  --voice-name alice \
  --ref-audio /path/to/reference.wav \
  --mode ref_continuation \
  --prompt-text "Original transcript of the reference audio" \
  --output-dir /path/to/custom_voices

Only profiles whose safetensors payload can be loaded and validated are exposed by GET /v1/audio/voices. Valid precomputed voices can be used in POST /v1/audio/speech by passing voice="alice" without ref_audio.

Configuration (environment variables):

The in-memory LRU has a fixed 512 MiB byte budget.

Batch Speech Generation¶

The batch endpoint synthesizes multiple texts in a single request, returning all results as JSON with base64-encoded audio.

Endpoint¶

POST /v1/audio/speech/batch
Content-Type: application/json

Request Parameters¶

Each item in the items array requires only input (the text). All other fields are optional and override the batch-level defaults when set:

Response Format¶

{
    "id": "speech-batch-abc123",
    "results": [
        {
            "index": 0,
            "status": "success",
            "audio_data": "<base64-encoded audio>",
            "media_type": "audio/wav",
            "usage": {
                "input_tokens": 119,
                "output_tokens": 77,
                "total_tokens": 196,
                "input_token_details": { "text_tokens": 18, "audio_tokens": 101 }
            }
        },
        {
            "index": 1,
            "status": "error",
            "error": "Input text cannot be empty"
        }
    ],
    "total": 2,
    "succeeded": 1,
    "failed": 1
}

Each successful item carries a usage object (errored items omit it):

• input_tokens = text_tokens + audio_tokens
  • text_tokens: tokens of the synthesized text (input plus instructions)
  • audio_tokens: reference-audio codec frames, non-zero only for in-context voice cloning (Base task); 0 for CustomVoice/VoiceDesign or x-vector-only
• output_tokens: generated codec tokens
• total_tokens = input_tokens + output_tokens

This is the same usage object emitted on the terminal speech.audio.done event of the single endpoint's SSE stream (stream_format="sse").

Examples¶

Basic batch with shared defaults:

curl -X POST http://localhost:8091/v1/audio/speech/batch \
    -H "Content-Type: application/json" \
    -d '{
        "items": [
            {"input": "Hello, how are you?"},
            {"input": "Goodbye, see you later!"}
        ],
        "voice": "vivian",
        "language": "English"
    }'

Per-item overrides (different voices and formats):

curl -X POST http://localhost:8091/v1/audio/speech/batch \
    -H "Content-Type: application/json" \
    -d '{
        "items": [
            {"input": "Hello!", "voice": "vivian", "response_format": "mp3"},
            {"input": "你好！", "voice": "ryan", "language": "Chinese"}
        ],
        "response_format": "wav"
    }'

Voice cloning with shared reference audio (Base task):

curl -X POST http://localhost:8091/v1/audio/speech/batch \
    -H "Content-Type: application/json" \
    -d '{
        "items": [
            {"input": "First sentence in the cloned voice."},
            {"input": "Second sentence in the cloned voice."}
        ],
        "task_type": "Base",
        "ref_audio": "https://example.com/reference.wav",
        "ref_text": "Transcript of the reference audio"
    }'

Setting ref_audio at the batch level applies it to all items, avoiding the need to repeat it per item.

Decoding the response in Python:

import base64
import httpx

response = httpx.post(
    "http://localhost:8091/v1/audio/speech/batch",
    json={
        "items": [
            {"input": "First sentence."},
            {"input": "Second sentence."},
        ],
        "voice": "vivian",
    },
    timeout=300.0,
)

for result in response.json()["results"]:
    if result["status"] == "success":
        audio_bytes = base64.b64decode(result["audio_data"])
        with open(f"output_{result['index']}.wav", "wb") as f:
            f.write(audio_bytes)

Configuration¶

All items are fanned out to generate() concurrently. The engine's stage worker automatically batches them up to the configured max_num_seqs and queues the rest — no client-side throttling needed.

For best throughput, set both stages' max_num_seqs above 1 via --stage-overrides. On the current Qwen3-TTS CustomVoice benchmark, stage 1 performed best at max_num_seqs: 10:

vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice \
    --omni --port 8091 --trust-remote-code --enforce-eager \
    --stage-overrides '{"0":{"max_num_seqs":10,"gpu_memory_utilization":0.2},
                        "1":{"max_num_seqs":10,"gpu_memory_utilization":0.2}}'

The bundled qwen3_tts.yaml uses a multi-request default and lets stage 1 batch chunks across in-flight requests. For latency-sensitive deployments, avoid forcing stage 1 back to max_num_seqs: 1; benchmark before reducing it below 10.

The bundled config also sets initial_codec_chunk_frames: 1. This emits only the first audio chunk early for lower TTFA, then returns to the normal codec_chunk_frames window so Code2Wav does not repeatedly decode tiny overlapping chunks.

Supported Models¶

Qwen3-TTS¶

Fish Speech S2 Pro¶

Fish Speech uses ref_audio and ref_text for voice cloning (no task_type needed). The voice field should be set to "default". See the Fish Speech section of the online TTS hub for details.

Voxtral TTS¶

CosyVoice3¶

Gepard-1.0¶

Gepard request fields:

See the Gepard section of the online TTS hub for launch commands. Native-AR recompute preemption is a known limitation under concurrency.

OmniVoice¶

OmniVoice uses packed variable-length attention for batched generator execution. The attention operator accepts FP16 and BF16 inputs, so its query, key, and value tensors are evaluated in BF16 even when the stage is configured with dtype: float32; the attention output is converted back to the model's hidden-state dtype before the output projection. Consequently, a float32 stage configuration does not imply FP32 attention arithmetic.

VoxCPM2¶

Startup LoRA adapter¶

To serve a fine-tuned VoxCPM2 voice, set voxcpm2_runtime_config.startup_lora_path in the stage's hf_overrides. Copy vllm_omni/deploy/voxcpm2.yaml to a local deploy config and add this key alongside its existing runtime settings:

# Under stages[0].engine_extras.hf_overrides.voxcpm2_runtime_config:
startup_lora_path: /absolute/path/to/adapter

Then launch with the modified config:

vllm serve openbmb/VoxCPM2 --omni --deploy-config /absolute/path/to/voxcpm2-lora.yaml

The directory must be accessible to the worker and contain the native VoxCPM2 training export: lora_config.json (with a lora_config object containing r, alpha, enabled groups, and target module names) and lora_weights.safetensors. PEFT checkpoints and pickle checkpoints are not accepted. Missing, unexpected, non-finite, or incorrectly shaped adapter tensors fail model loading rather than silently loading a partial adapter.

The adapter is merged into the base LM, residual LM, LocDiT, and optional projection layers selected by its configuration, after base weight loading and before compilation or CUDA graph capture. All requests use that adapter; voice still selects a reference voice, not a LoRA adapter. Changing adapters requires restarting the server. Runtime loading/unloading, per-request multi-LoRA selection, and load_format=dummy are not supported by this path. Weight fusion rounds to the base model's dtype, so numerical and speech-quality parity should be checked against the upstream adapter on your deployment.

MOSS-TTS-Nano¶

Breeze-TTS-2¶

Error Responses¶

400 Bad Request¶

Invalid parameters:

{
    "error": {
        "message": "Input text cannot be empty",
        "type": "BadRequestError",
        "param": null,
        "code": 400
    }
}

Qwen3-TTS task/checkpoint mismatch:

{
    "error": {
        "message": "Qwen3-TTS CustomVoice checkpoint does not support task_type='Base'. Use task_type='CustomVoice' or load the matching Base checkpoint.",
        "type": "BadRequestError",
        "param": null,
        "code": 400
    }
}

404 Not Found¶

Model not found:

{
    "error": {
        "message": "The model `xxx` does not exist.",
        "type": "NotFoundError",
        "param": "model",
        "code": 404
    }
}

Troubleshooting¶

"TTS model did not produce audio output"¶

Ensure you're using the correct model variant for your task type:

• CustomVoice task → CustomVoice model
• VoiceDesign task → VoiceDesign model
• Base task → Base model

Uploaded and precomputed Qwen3-TTS voices select the Base task, even when the client supplies a different task_type; serve them with a Base checkpoint. Built-in Qwen3-TTS presets remain CustomVoice voices.

Server Not Running¶

# Check if server is responding
curl http://localhost:8091/v1/audio/voices

Out of Memory¶

If you encounter OOM errors:

1. Use smaller model variant: Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice
2. Reduce --gpu-memory-utilization

Unsupported Speaker¶

Use /v1/audio/voices to list available voices for the loaded model.

Orchestration Loop (experimental)¶

Multi-stage omni deployments route stage outputs through a single orchestrator loop. The legacy loop polls every stage replica on a 1 ms cadence. The event-driven mode replaces the poll with one reader task per live stage replica awaiting its client directly, and switches the serving-side final-output drain to a condition-variable wakeup at the same time. Qwen3-TTS uses this mode by default; other pipelines keep the legacy poll unless enabled.

Configuration (environment variables):

Set it on the process that runs the orchestrator (stage 0 of an omni deployment) before starting the server when overriding the pipeline default. For example, explicitly disable event-driven orchestration for Qwen3-TTS:

VLLM_OMNI_EVENT_DRIVEN_ORCH=0 \
vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-Base \
    --omni \
    --port 8091

The server logs the selected loop mode and its reader/poller counts once at startup, so you can confirm which loop is live.

Routing, output ordering, and terminal-state behavior are identical on both loops; only the poll cadence changes. Leaving the variable unset selects the pipeline default: event-driven for Qwen3-TTS, and legacy polling for other pipelines, including Qwen3-Omni.

Known limitations:

• The measured serving A/B (idle CPU 2.43% to 0.07%; TTFP p99 -32% at concurrency 8) predates the rebuild on the per-replica fault-isolation work in #4285. That work changed dead-replica handling and reader/poller lifecycle rather than the steady-state output path, and the parity suite covers it, but the serving A/B has not been re-run on the current head.
• The diffusion-poller branch is covered by unit tests only. Deployments whose stages all run as standard engine cores never exercise it, including GLM-TTS, which deploys its DiT without stage_type: diffusion.
• Concurrency 1 and 32 measured at parity with the legacy loop. At 32 the latency is admission-bound, which this mode does not address.

Development¶

Enable debug logging:

vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice \
    --deploy-config vllm_omni/deploy/qwen3_tts.yaml \
    --omni \
    --port 8091 \
    --trust-remote-code \
    --enforce-eager \
    --uvicorn-log-level debug