Text-To-Speech (Online Serving)¶

Source https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/text_to_speech.

vLLM-Omni exposes TTS models through the OpenAI-compatible POST /v1/audio/speech endpoint, launched with vllm serve <model> --omni. Each TTS model has its own subdirectory containing client snippets, gradio demos, and helper scripts; this README is the single doc entry point for all of them.

For offline inference, see examples/offline_inference/text_to_speech. For the full list of supported architectures across all modalities, see Supported Models.

Supported Models¶

CosyVoice3 is intentionally absent: no online example exists for it yet. See its offline section instead.

Common Quick Start¶

Launch the server (defaults shown — adjust --port, --gpu-memory-utilization, etc. as needed):

vllm serve <hf-repo-or-local-path> --omni --port 8091

Send a TTS request via curl. These generic snippets assume a model with a preset/default voice; voice-cloning-only models such as IndexTTS-2 require ref_audio or an uploaded audio voice (see model-specific sections below).

curl -X POST http://localhost:8091/v1/audio/speech \
    -H "Content-Type: application/json" \
    -d '{
        "input": "Hello, how are you?",
        "voice": "default",
        "response_format": "wav"
    }' --output output.wav

Or via Python httpx:

import httpx

response = httpx.post(
    "http://localhost:8091/v1/audio/speech",
    json={
        "input": "Hello, how are you?",
        "voice": "default",
        "response_format": "wav",
    },
    timeout=300.0,
)
open("output.wav", "wb").write(response.content)

Or via the OpenAI SDK:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8091/v1", api_key="none")
response = client.audio.speech.create(
    model="<hf-repo>",
    voice="default",
    input="Hello, how are you?",
)
response.stream_to_file("output.wav")

Streaming PCM output (where supported) — set stream=true, stream_format="audio", and response_format="pcm":

curl -X POST http://localhost:8091/v1/audio/speech \
    -H "Content-Type: application/json" \
    -d '{
        "input": "Hello, how are you?",
        "voice": "default",
        "stream": true,
        "stream_format": "audio",
        "response_format": "pcm"
    }' --no-buffer | play -t raw -r 24000 -e signed -b 16 -c 1 -

Adjust the player's sample rate to match the model (44.1 kHz for Fish Speech, 48 kHz for VoxCPM2, 22.05 kHz for Gepard and IndexTTS-2, and 24 kHz for many others).

For full request-shape documentation (all parameters, response formats, error codes), see the Speech API reference.

Gepard-1.0¶

Single-stage native AR TTS at 22.05 kHz mono. Zero-shot only: omit voice or pass "default". Voice cloning from reference audio is not available yet.

Prerequisites¶

Same NeMo NanoCodec install as the offline Gepard section. On a host whose CUDA toolkit cannot build kernels — no nvcc/ninja, or a consumer Blackwell (sm_120) card — also export VLLM_USE_FLASHINFER_SAMPLER=0 before launch (same note as offline). That env is not a deploy-YAML field.

Launch¶

vllm-omni serve nineninesix/gepard-1.0 --omni --port 8091 --trust-remote-code \
    --stage-init-timeout 900 \
    --deploy-config vllm_omni/deploy/gepard.yaml
# or:
./gepard/run_server.sh

--stage-init-timeout 900 matches the online e2e fixture and the offline example. Serve defaults to 300s, which is often too short for a cold download of the talker plus NanoCodec.

The packaged vllm_omni/deploy/gepard.yaml must be passed with --deploy-config. The checkpoint self-identifies as qwen3_5_text, so omitting the YAML launches a diffusion fallback instead of the Gepard pipeline. The YAML sets async_chunk: false, max_num_seqs: 4, and currently pins seed: 42, so serving is deterministic by default until that YAML seed is removed. Pass an explicit per-request seed in tests and clients rather than depending on either default.

Sending requests¶

python examples/online_serving/text_to_speech/gepard/speech_client.py \
    --text "Hello, this is Gepard speaking."

python examples/online_serving/text_to_speech/gepard/speech_client.py \
    --text "Hello, this is Gepard speaking." --seed 7 --stream --output output.pcm

Notes¶

• Output: 22.05 kHz mono. max_new_tokens is a frame budget (1 token = 1 frame = 1024 samples ≈ 46.4 ms at 21.5 fps; adapter bounds 1..4096).
• Supported request fields: input (required), voice (default only), response_format (wav default; wav/pcm/flac/mp3 non-streaming; opus 400 because 22.05 kHz is not an Opus sample rate; streaming pcm/wav only), stream / stream_format, max_new_tokens, seed.
• Unsupported: speed, extra_params (including temperature/top_p/top_k), ref_audio, ref_text, speaker_embedding, task_type, instructions, language, and word_timestamps.
• Concurrent requests at max_num_seqs: 4 are supported. Native-AR recompute preemption is a known limitation of this architecture (a request that is preempted mid-generation can resume incorrectly); keep concurrency at or below max_num_seqs and treat preemption as out of scope until the platform fix lands.
• Optional comparison against the upstream Gepard reference server needs Blackwell/Hopper + CUDA 13 + Postgres and is not part of CI.

GLM-TTS¶

2-stage TTS (AR + DiT flow-matching) at 24 kHz. Every request requires ref_audio + ref_text.

Launch¶

vllm serve zai-org/GLM-TTS --omni --trust-remote-code --port 8091
# or:
bash examples/online_serving/text_to_speech/glm_tts/run_server.sh /path/to/GLM-TTS

Sending requests¶

# Voice cloning (required)
python examples/online_serving/text_to_speech/glm_tts/openai_speech_client.py \
    --text "你好，这是语音克隆测试。" \
    --ref-audio file:///path/to/ref.wav \
    --ref-text "这是参考音频的文本内容。"

# Custom format
python examples/online_serving/text_to_speech/glm_tts/openai_speech_client.py \
    --text "Hello, this is a voice cloning test." \
    --ref-audio file:///path/to/ref.wav \
    --ref-text "Transcript of the reference audio." \
    --response-format mp3 -o output.mp3

Gradio demo¶

bash examples/online_serving/text_to_speech/glm_tts/run_gradio_demo.sh

Notes¶

• Output: 24 kHz mono WAV via HiFT vocoder.
• ref_audio + ref_text are required together on every request. Reference audio should be 3-10 seconds.
• Voice cloning feature extraction (WhisperVQ, CampPlus, mel) runs on the model side — no external dependency on the serving layer.

IndexTTS-2 and IndexTTS-2.5¶

2-stage TTS at 22.05 kHz. Requests use ref_audio for voice cloning, or an uploaded audio voice from /v1/audio/voices. IndexTTS-2.5 uses a multilingual tokenizer, CAMPPlus speaker projection, EnhancedCodec, and code-only Stage 0→1 transfer by default. Both versions support emotion conditioning via emo_audio, emo_text, or emo_vector passed in extra_params.

Prerequisites¶

IndexTTS-2 uses its existing frontend and does not require the IndexTTS-2.5 text dependencies. Before launching IndexTTS-2.5, install its optional frontend dependencies:

pip install 'vllm-omni[indextts2]'

Launch¶

vllm serve IndexTeam/IndexTTS-2 --omni --trust-remote-code --port 8092
# or, to pass the bundled deploy config explicitly:
bash examples/online_serving/text_to_speech/indextts2/run_server.sh

# IndexTTS-2.5 default: use_gpt_latent=false
MODEL_VERSION=2.5 \
MODEL=/path/to/indextts-2.5 \
bash examples/online_serving/text_to_speech/indextts2/run_server.sh

Sending requests¶

# Voice cloning (ref_audio required)
python examples/online_serving/text_to_speech/indextts2/speech_client.py \
    --text "你好，世界！" \
    --ref-audio /path/to/reference.wav

# With emotion audio
python examples/online_serving/text_to_speech/indextts2/speech_client.py \
    --text "今天心情很好！" \
    --ref-audio /path/to/ref.wav \
    --emo-audio /path/to/happy.wav

# IndexTTS-2.5 Japanese request
python examples/online_serving/text_to_speech/indextts2/speech_client.py \
    --model-version 2.5 \
    --model /path/to/indextts-2.5 \
    --lang ja \
    --text "こんにちは、音声合成のテストです。" \
    --ref-audio /path/to/ref.wav

# IndexTTS-2.5 native speed control
python examples/online_serving/text_to_speech/indextts2/speech_client.py \
    --model-version 2.5 \
    --model /path/to/indextts-2.5 \
    --speed 2.0 \
    --text "这是两倍速度的语音合成测试。" \
    --ref-audio /path/to/ref.wav

Notes¶

• Output: 22.05 kHz mono WAV.
• Provide ref_audio on the documented raw request path, or pass voice only when it names an uploaded audio voice; neither version provides a built-in text-only preset voice.
• IndexTTS-2.5 request controls lang and text_normalization are carried in extra_params; the client does this for --model-version 2.5.
• IndexTTS-2.5 accepts the public speed request field in [0.5, 2.0]: 2.0 generates shorter, faster speech and 0.5 generates longer, slower speech. This is native Stage 1 duration control, so serving does not apply the generic playback-speed adjustment a second time. IndexTTS-2 does not use this control.
• IndexTTS-2.5 accepts language codes such as zh, en, zhen (mixed Chinese/English), ja, and yue. Mandarin is a vLLM-Omni convenience alias for zh. Japanese (ja) uses fugashi tokenization and produces audio, but it does not automatically expand numbers, dates, or percentages; callers should first write those inputs as readable Japanese text.
• A request seed controls Stage 0 AR sampling and per-request CFM noise. Different concurrent batch compositions do not guarantee a bit-identical waveform.
• IndexTTS-2.5 Stage 0 uses plain vLLM sampling and does not reproduce the official default num_beams=3 beam search. For parity comparisons, run upstream with num_beams=1; output quality can differ from the official beam-search result.
• Emotion params (emo_audio, emo_text, emo_vector, emo_alpha, use_emo_text, use_random) are passed via the extra_params field. Official precedence is use_emo_text > emo_vector > emo_audio > same emotion as the speaker reference.
• IndexTTS-2.5 uses vllm_omni/deploy/indextts2_5.yaml with the official code-only Stage 0 to Stage 1 contract.
• For IndexTTS-2.5, set MODEL to the local native bundle; asset discovery accepts its nested checkpoints/ layout.

Audio8 TTS Preview¶

0.6B DualAR TTS at 44.1 kHz, 11 languages, zero-shot voice cloning.

Prerequisites¶

None beyond the base install: the neural audio codec is implemented in tree and its weights (codec.pth) ship with the checkpoint.

Launch¶

vllm serve Audio8/Audio8-TTS-Preview-0.6b --omni --port 8092
# or:
./audio8_tts/run_server.sh

The deploy config auto-loads from vllm_omni/deploy/audio8_tts.yaml (HF model_type is arktts). Do not pass --trust-remote-code: vllm-omni registers its own arktts config, and transformers would otherwise prefer the checkpoint's remote code and bypass it.

Text-only synthesis¶

curl -X POST http://localhost:8092/v1/audio/speech \
    -H "Content-Type: application/json" \
    -d '{
        "input": "Welcome to Audio8 TTS.",
        "response_format": "wav"
    }' --output output.wav

Voice cloning¶

curl -X POST http://localhost:8092/v1/audio/speech \
    -H "Content-Type: application/json" \
    -d '{
        "input": "Welcome to Audio8 TTS.",
        "ref_audio": "https://example.com/reference.wav",
        "ref_text": "The exact transcript of the reference recording."
    }' --output cloned.wav

ref_audio also accepts a data:audio/wav;base64,... URL. ref_text is mandatory whenever ref_audio is present and must match the recording. Uploading a voice via POST /v1/audio/voices lets the server reuse the encoded reference codes across requests (voice: "<name>").

Streaming¶

python audio8_tts/speech_client.py --text "Welcome to Audio8 TTS." --stream --output out.pcm
ffplay -f s16le -ar 44100 -ac 1 out.pcm

Gradio demo¶

./audio8_tts/run_gradio_demo.sh          # server + demo
python audio8_tts/gradio_demo.py --api-base http://localhost:8092   # demo only

Benchmark¶

export VLLM_OMNI_BENCH_AUDIO_SAMPLE_RATE=44100   # required: harness defaults to 24 kHz
vllm bench serve --omni --host 127.0.0.1 --port 8092 \
    --model Audio8/Audio8-TTS-Preview-0.6b \
    --backend openai-audio-speech --endpoint /v1/audio/speech \
    --dataset-name seed-tts-text --dataset-path benchmarks/build_dataset/seed_tts_smoke \
    --seed-tts-locale en --num-prompts 20 --num-warmups 2 \
    --max-concurrency 1 --request-rate inf \
    --percentile-metrics ttft,e2el,audio_rtf,audio_ttfp,audio_duration,audio_underrun

Measured on 1x H20 shared with an unrelated job holding ~80% SM, so treat these as a lower bound. 20 prompts from seed_tts_smoke/en, mean audio 4.5 s, deploy defaults:

~5.3x lower per-request RTF than the upstream reference implementation at concurrency 1, ~12.8x aggregate throughput at concurrency 8.

Stage-0 max_num_seqs: first-packet vs throughput¶

This subsection is operator tuning guidance, not a claim this PR optimizes anything: the shipped deploy default is unchanged (max_num_seqs: 4). The numbers below are a single run on the same contended H20 as above (one unrelated job holding ~80% SM, no repeats), so read them as directional, not as measured speedups. Same setup, varying stage 0's max_num_seqs (stage 1 stays at 1):

Reading of this:

• Below the limit (concurrency <= max_num_seqs) the setting is irrelevant, as expected — c=1 and c=4 are identical for both.
• At c=8, raising the limit to 8 admits every request instead of queueing half of them: in this single run, roughly +27% throughput and ~4.5x lower first-packet latency.
• Throughput saturates around 3.7-3.9 req/s (~17 s of audio per second). Past c=8 you only buy queueing delay.
• The cost of admitting more streams is a 0.22-0.29 s worst-case buffer deficit in 1 of 20 requests (Streaming continuity OK rate 95%), because stage 1 runs max_num_seqs: 1 and the codec window round-robins across streams. Client-side prebuffering (the bundled Gradio demo holds 2 chunks) absorbs that; raising stage 1's max_num_seqs is the untested knob if you need both.

The shipped default stays at 4 because every measured point there had zero underrun. Raise it to 8 if first-packet latency under load matters more than a rare sub-300 ms gap.

Notes¶

• Output: 44.1 kHz mono; ~21.5 codec frames per second.
• No built-in speaker presets. Omit voice for a random timbre, or clone one.
• max_new_tokens caps generated codec frames (1 frame ~= 46 ms).
• Context is 2048 packed text+audio positions (max_model_len: 2048).
• Do not send temperature: 0: greedy decoding is degenerate for this checkpoint and truncates the utterance. The deploy defaults (0.7 / top_k 50 / top_p 0.9) mirror generation_config.json.

Fish Speech S2 Pro¶

4B dual-AR TTS at 44.1 kHz. Server uses the DAC codec, which is vendored in vLLM-Omni (no extra packages required).

Kvcache attention fast path¶

Fish Speech S2 Pro uses a Triton decode-only kvcache attention fast path by default on CUDA builds. Set VLLM_OMNI_FISH_KVCACHE_ATTN=0 to disable it, or VLLM_OMNI_FISH_KVCACHE_ATTN=required to fail fast if the fast path cannot be installed.

# Verify fast path availability.
python - <<'PY'
from vllm_omni.attention import fish_kvcache_attn

print(fish_kvcache_attn.is_available())
print(fish_kvcache_attn.load_error())
PY

# Optional: disable the runtime fast path.
export VLLM_OMNI_FISH_KVCACHE_ATTN=0

Launch¶

vllm serve fishaudio/s2-pro --omni --port 8091
# or:
./fish_speech/run_server.sh

The deploy config auto-loads from vllm_omni/deploy/fish_qwen3_omni.yaml (the HF model_type on the fishaudio checkpoint is fish_qwen3_omni).

Voice cloning¶

curl -X POST http://localhost:8091/v1/audio/speech \
    -H "Content-Type: application/json" \
    -d '{
        "input": "Hello, this is a cloned voice.",
        "voice": "default",
        "ref_audio": "https://example.com/reference.wav",
        "ref_text": "Transcript of the reference audio."
    }' --output cloned.wav

CLI client¶

cd examples/online_serving/text_to_speech/fish_speech
python speech_client.py --text "Hello, how are you?"
python speech_client.py --text "Hello world" --stream --output output.pcm

Gradio demo¶

./fish_speech/run_gradio_demo.sh             # launches server + Gradio
python fish_speech/gradio_demo.py --api-base http://localhost:8091  # if server already running

Notes¶

• Output: 44.1 kHz mono.
• Streaming PCM player command must use -r 44100.

Ming-omni-tts¶

Dense 0.5B two-stage TTS served through /v1/audio/speech. Ming uses the standard speech endpoint plus structured controls in instructions, voice, language, ref_audio, ref_text, and speaker_embedding.

Launch¶

bash examples/online_serving/text_to_speech/ming_tts/run_server.sh

Equivalent manual command:

vllm-omni serve inclusionAI/Ming-omni-tts-0.5B \
    --deploy-config vllm_omni/deploy/ming_tts.yaml \
    --host 0.0.0.0 --port 8091 \
    --enforce-eager --omni

Sending requests¶

python examples/online_serving/text_to_speech/ming_tts/openai_speech_client.py \
    --text "你好，这是 Ming 在线语音合成测试。"

Structured dialect control:

python examples/online_serving/text_to_speech/ming_tts/openai_speech_client.py \
    --text "我觉得社会企业同个人都有责任" \
    --instruction-json '{"方言":"广粤话"}' \
    --ref-audio /path/to/yue_prompt.wav

Zero-shot cloning:

python examples/online_serving/text_to_speech/ming_tts/openai_speech_client.py \
    --text "我们的愿景是构建未来服务业的数字化基础设施，为世界带来更多微小而美好的改变。" \
    --ref-audio /path/to/10002287-00000094.wav \
    --ref-text "在此奉劝大家别乱打美白针。"

Notes¶

• run_curl.sh keeps a small sanity subset; use the Ming README for the broader request cookbook.
• Online serving is speech-shaped today; music-only bgm and text-to-audio tta remain offline examples.
• Full request details live in ming_tts/README.md.

Ming-flash-omni-TTS¶

Standalone talker-only deployment of Ming-flash-omni-2.0. Voice is controlled through caption text passed via instructions.

Launch¶

# from repo root
bash examples/online_serving/text_to_speech/ming_flash_omni_tts/run_server.sh

Equivalent manual command:

vllm serve Jonathan1909/Ming-flash-omni-2.0 \
    --deploy-config vllm_omni/deploy/ming_flash_omni_tts.yaml \
    --host 0.0.0.0 --port 8091 \
    --trust-remote-code --omni

Sending requests¶

python examples/online_serving/text_to_speech/ming_flash_omni_tts/speech_client.py \
    --text "我们当迎着阳光辛勤耕作，去摘取，去制作，去品尝，去馈赠。" \
    --output ming_online.wav

ASMR-style caption via instructions:

python examples/online_serving/text_to_speech/ming_flash_omni_tts/speech_client.py \
    --text "我会一直在这里陪着你，直到你慢慢、慢慢地沉入那个最温柔的梦里……好吗？" \
    --instructions "这是一种ASMR耳语，属于一种旨在引发特殊感官体验的创意风格。这个女性使用轻柔的普通话进行耳语，声音气音成分重。" \
    --output ming_online_asmr.wav

Notes¶

• Server uses use_zero_spk_emb=True and the cookbook decode defaults (max_decode_steps=200, cfg=2.0, sigma=0.25, temperature=0.0). For other caption fields (语速, 基频, IP, BGM, etc.) or overriding decode args, use the offline example where additional_information is set explicitly.
• This is the online counterpart of examples/offline_inference/text_to_speech/ming_flash_omni_tts/.
• For multimodal Ming-flash-omni online serving, see examples/online_serving/ming_flash_omni/.

MOSS-TTS Local Transformer v1.5¶

For a single H200, the optional moss_tts_local_h200.yaml deployment places both the talker and codec on logical GPU 0:

CUDA_VISIBLE_DEVICES=0 VLLM_OMNI_EVENT_DRIVEN_ORCH=1 \
    vllm serve OpenMOSS-Team/MOSS-TTS-Local-Transformer-v1.5 \
    --omni --trust-remote-code \
    --deploy-config vllm_omni/deploy/moss_tts_local_h200.yaml \
    --disable-log-stats

Run from the repository root and select an available physical GPU with CUDA_VISIBLE_DEVICES. The command enables event-driven orchestration and disables detailed per-request statistics logging to reduce CPU overhead at high concurrency. Remove --disable-log-stats when those statistics are needed; keep these settings identical when comparing performance. This preset configures 256 request slots per stage, a 32 GiB talker KV cache, talker CUDA Graph buckets through 512 scheduled tokens, and codec batch buckets through 256. It requires more than 80 GiB of GPU memory; request capacity also depends on input and generated lengths. The default moss_tts_local.yaml remains available for smaller deployments.

The preset enables the optional codec backend with hf_overrides.codec_attention_backend: triton on stage 1. It preserves the streaming ring-cache mask and uses BF16 attention with 64-dimensional heads; other attention shapes use PyTorch SDPA. Set the backend to sdpa to use the default attention implementation. Codec terminal tails share execution only when they already map to the same padded CUDA Graph; returned audio retains each request's actual length.

Voice cloning requests use ref_audio and ref_text. For streaming output, set stream: true, stream_format: "audio", and response_format: "pcm"; the native PCM format is 48 kHz, stereo, signed 16-bit little-endian. Compute audio throughput as PCM bytes / (48000 * 2 * 2) / elapsed seconds, and compare the same requests, concurrency, and reference-cache state.

MOSS-TTS-Nano¶

Single-stage 0.1B AR LM + MOSS-Audio-Tokenizer-Nano codec at 48 kHz mono. Every request must include ref_audio; there are no built-in speaker presets.

The OpenAI-schema voice and ref_text fields are accepted but ignored — voice_clone does not consume a transcript, and upstream's continuation mode (the only path that accepts prompt_text) emits near-silent output, so it is not exposed here. Sample reference clips ship in the upstream repo under assets/audio/.

Launch¶

vllm serve OpenMOSS-Team/MOSS-TTS-Nano --omni --port 8091
# or:
./moss_tts_nano/run_server.sh

The deploy config at vllm_omni/deploy/moss_tts_nano.yaml auto-loads; no --deploy-config, --trust-remote-code, or --enforce-eager flags are needed.

Sending requests¶

# One-off fetch of a sample reference clip; cache under XDG_CACHE_HOME.
REF_DIR="${XDG_CACHE_HOME:-$HOME/.cache}/moss-tts-nano"
mkdir -p "$REF_DIR"
REF_WAV="$REF_DIR/zh_1.wav"
[ -s "$REF_WAV" ] || curl -L -o "$REF_WAV" https://raw.githubusercontent.com/OpenMOSS/MOSS-TTS-Nano/main/assets/audio/zh_1.wav
REF_AUDIO=$(base64 -w 0 "$REF_WAV")

curl -X POST http://localhost:8091/v1/audio/speech \
    -H "Content-Type: application/json" \
    -d "{
        \"input\": \"你好，这是语音合成测试。\",
        \"ref_audio\": \"data:audio/wav;base64,${REF_AUDIO}\",
        \"response_format\": \"wav\"
    }" --output output.wav

Streaming PCM¶

curl -X POST http://localhost:8091/v1/audio/speech \
    -H "Content-Type: application/json" \
    -d "{
        \"input\": \"Hello, streaming output from MOSS-TTS-Nano.\",
        \"ref_audio\": \"data:audio/wav;base64,${REF_AUDIO}\",
        \"stream\": true,
        \"stream_format\": \"audio\",
        \"response_format\": \"pcm\"
    }" --no-buffer | play -t raw -r 48000 -e signed -b 16 -c 1 -

Gradio demo¶

# Option 1: launch server + Gradio together
./moss_tts_nano/run_gradio_demo.sh

# Option 2: server already running
python moss_tts_nano/gradio_demo.py --api-base http://localhost:8091

Then open http://localhost:7860 in your browser.

Notes¶

• Output is 48 kHz mono PCM (the upstream tokenizer is internally stereo at 48 kHz; the wrapper averages to mono before reaching the engine).
• Standard /v1/audio/speech request shape: input, ref_audio (base64 data URL), response_format, stream, max_new_tokens. The voice and ref_text fields from the OpenAI schema are accepted but ignored.

OmniVoice¶

Zero-shot multilingual TTS (600+ languages). Online serving currently exposes auto voice only; voice cloning and voice design are available offline.

Prerequisites¶

huggingface-cli download k2-fsa/OmniVoice

Voice cloning (offline) needs transformers>=5.3.0; auto voice works with transformers>=4.57.0.

Launch¶

vllm serve k2-fsa/OmniVoice --omni --port 8091 --trust-remote-code
# or:
./omnivoice/run_server.sh

CLI client¶

cd examples/online_serving/text_to_speech/omnivoice
# Text-only (auto voice)
python speech_client.py --text "Hello, how are you?"

# Language hint
python speech_client.py --text "Bonjour, comment allez-vous?" --language French
# Voice cloning (reference audio + optional ref_text)
python speech_client.py \
--text "Bonjour, comment allez-vous?" \
--ref-audio /path/to/ref_audio.wav \
--ref-text "Bonjour, comment allez-vous?"

# Style instruction (voice design-style control)
python speech_client.py \
--text "Bonjour, comment allez-vous?" \
--language French \
--instructions "loud voice"

# Deterministic output with seed parameter
python speech_client.py --text "Hello, how are you?" --seed 42

The client supports --api-base, --model, --text, --response-format, --language, --voice, --ref-audio, --ref-text, --instructions, --seed, and --output.

Qwen3-TTS¶

Three model variants exposed via separate checkpoints:

Each variant ships smaller 0.6B companions where available.

Launch¶

vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice --omni --port 8091
# or:
./qwen3_tts/run_server.sh                # default: CustomVoice
./qwen3_tts/run_server.sh VoiceDesign
./qwen3_tts/run_server.sh Base

For a local deployment with multiple API frontend processes sharing one set of TTS stage engines, add --api-server-count:

vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice \
    --omni --api-server-count 2 --port 8091

This mode supports local EngineCore stages only; headless or remote stage deployments are not supported. Runtime voice upload and deletion are disabled with multiple API frontends because their registries are process-local. Built-in voices, inline ref_audio, and voices restored from custom_voice_dir at startup are supported.

Executor backend¶

Single-GPU serves now default to the uniproc executor (lower IPC overhead, the Base cloning use case from #2603 / #2604). vllm_omni/deploy/qwen3_tts.yaml is the only Qwen3-TTS deploy config; pass --deploy-config <path> to override.

To opt out of chunked streaming, pass --no-async-chunk — the pipeline auto-dispatches to the end-to-end codec processor.

Sending requests¶

# CustomVoice with a predefined speaker
python qwen3_tts/openai_speech_client.py \
    --model Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice \
    --text "今天天气真好" \
    --speaker ryan \
    --instructions "用开心的语气说"

# VoiceDesign with a style description
python qwen3_tts/openai_speech_client.py \
    --model Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign \
    --task-type VoiceDesign \
    --text "哥哥，你回来啦" \
    --instructions "体现撒娇稚嫩的萝莉女声，音调偏高"

# Base voice cloning
python qwen3_tts/openai_speech_client.py \
    --model Qwen/Qwen3-TTS-12Hz-1.7B-Base \
    --task-type Base \
    --text "Hello, this is a cloned voice" \
    --ref-audio /path/to/reference.wav \
    --ref-text "Original transcript of the reference audio"

Voices endpoint¶

List available voices, or upload a custom one for Base cloning:

# List
curl http://localhost:8091/v1/audio/voices

# Upload
curl -X POST http://localhost:8091/v1/audio/voices \
    -F "audio_sample=@/path/to/voice_sample.wav" \
    -F "consent=user_consent_id" \
    -F "name=custom_voice_1" \
    -F "ref_text=The exact transcript of the audio sample." \
    -F "speaker_description=warm narrator"

For Qwen3-TTS, uploaded voices are Base voice-cloning inputs and require a Base checkpoint. When a request names an uploaded voice, the server infers task_type="Base". Built-in presets such as vivian and ryan remain CustomVoice speakers and require a CustomVoice checkpoint.

The runtime upload and delete routes require a single API frontend; with --api-server-count > 1, use inline ref_audio or restore precomputed voices from custom_voice_dir at startup.

Precomputed custom voices¶

For reused Base voice-cloning speakers, precompute the reference artifacts once and load them at server startup. Precomputed voices use the same Base task and checkpoint-matching rules as uploaded voices:

python qwen3_tts/precompute_custom_voice.py \
    --model Qwen/Qwen3-TTS-12Hz-1.7B-Base \
    --voice-name alice \
    --ref-audio /path/to/reference.wav \
    --ref-text "Original transcript of the reference audio" \
    --mode icl \
    --output-dir /path/to/custom_voices

--mode icl stores both speaker_embedding and ref_code; --mode xvec stores only the speaker embedding. Add the output directory to a deploy config:

custom_voice_dir: /path/to/custom_voices

Then start the server with that config and call the Speech API with only the voice name:

vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-Base --omni --deploy-config /path/to/qwen3_tts_custom_voice.yaml

curl -X POST http://localhost:8091/v1/audio/speech \
    -H "Content-Type: application/json" \
    -d '{"input":"Hello from a precomputed voice.","voice":"alice","task_type":"Base"}' \
    --output alice.wav

Streaming PCM¶

curl -X POST http://localhost:8091/v1/audio/speech \
    -H "Content-Type: application/json" \
    -d '{
        "input": "Hello, how are you?",
        "voice": "vivian",
        "language": "English",
        "stream": true,
        "stream_format": "audio",
        "response_format": "pcm"
    }' --no-buffer | play -t raw -r 24000 -e signed -b 16 -c 1 -

Raw PCM streaming requires stream_format="audio", response_format="pcm", and async_chunk: true in the deploy config (default in qwen3_tts.yaml). speed is not supported when streaming.

Streaming WebSocket¶

The /v1/audio/speech/stream endpoint accepts text incrementally and, by default, synthesizes the buffered text as one continuous request on input.done:

python qwen3_tts/streaming_speech_client.py --text "Hello world. How are you? I am fine."
python qwen3_tts/streaming_speech_client.py --text "..." --simulate-stt --stt-delay 0.1

For per-sentence audio (including Indic danda ।) before input.done:

python qwen3_tts/streaming_speech_client.py \
    --text "नमस्ते। कैसे हो?" \
    --split-granularity sentence

input.done flushes without closing, so repeating --text synthesizes several utterances over one connection:

python qwen3_tts/streaming_speech_client.py \
    --text "First utterance." \
    --text "Second utterance, same connection."

To receive word-level timestamps, launch the server with a forced aligner:

vllm-omni serve Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice \
    --omni \
    --deploy-config vllm_omni/deploy/qwen3_tts.yaml \
    --trust-remote-code \
    --forced-aligner Qwen/Qwen3-ForcedAligner-0.6B

Then request PCM JSON sidecar chunks:

python qwen3_tts/streaming_speech_client.py \
    --text "Hello world. How are you?" \
    --stream-audio \
    --response-format pcm \
    --word-timestamps

The client writes one PCM file per sentence and a matching sentence_XXX_timestamps.json sidecar.

Non-streaming requests can also ask for timestamps: pass "word_timestamps": true to POST /v1/audio/speech and read the X-Word-Timestamps response header (JSON list of {word, start_ms, end_ms}, ASCII-escaped). Past 4 KB the header is replaced by X-Word-Timestamps-Omitted: oversize; bytes=<n>; limit=4096 and the audio still returns — use the WebSocket path for long transcripts.

To see the alignment instead of reading a JSON sidecar, run the word-timestamp Gradio demo (server must be launched with --forced-aligner):

python qwen3_tts/word_timestamps_demo.py --api-base http://localhost:8091

Each sentence's audio plays in an <audio> element while its text is rendered as inline word spans; the current word highlights as audio.currentTime crosses each start_ms. The Stop (barge-in) button cuts playback and reports the last-spoken word, useful for the voice-agent barge-in case.

Gradio demos¶

./qwen3_tts/run_gradio_demo.sh                              # CustomVoice (default)
./qwen3_tts/run_gradio_demo.sh --task-type VoiceDesign
./qwen3_tts/run_gradio_demo.sh --task-type Base

Speaker embedding interpolation¶

qwen3_tts/speaker_embedding_interpolation.py blends two predefined speakers' embeddings to produce intermediate voices. See the script for usage.

Batch client¶

qwen3_tts/batch_speech_client.py issues many concurrent requests for throughput measurement.

Notes¶

• Base voice cloning has uniproc-vs-mp tradeoffs depending on per-request reference audio cost; see the executor-backend section above.
• With async chunking, Qwen3-TTS Base voice cloning sends the full reference context in the first Code2Wav packet, then caches that prefix on the Code2Wav stage for follow-up chunks in the same request.
• vllm_omni/deploy/qwen3_tts.yaml is the default deploy config (loaded by HF model_type); per-stage runtime overrides are available via --stage-N-<field> <value>.

VoxCPM2¶

Single-stage native AR TTS at 48 kHz.

Launch¶

vllm serve openbmb/VoxCPM2 --omni --host 0.0.0.0 --port 8000

Deploy config auto-loads from vllm_omni/deploy/voxcpm2.yaml. Pass --deploy-config <path> to override or --stage-N-<field> <value> for per-stage runtime tweaks.

Sending requests¶

# Zero-shot synthesis
python voxcpm2/openai_speech_client.py --text "Hello, this is VoxCPM2."

# Voice cloning
python voxcpm2/openai_speech_client.py \
    --text "This should sound like the reference speaker." \
    --ref-audio /path/to/reference.wav

The ref_audio field accepts local file paths (auto-base64), HTTP URLs, or data:audio/wav;base64,... data URIs.

Precomputed custom voices¶

For repeated VoxCPM2 speakers, precompute the prompt cache and load it through custom_voice_dir:

python voxcpm2/precompute_custom_voice.py \
    --model openbmb/VoxCPM2 \
    --voice-name alice \
    --ref-audio /path/to/reference.wav \
    --mode ref_continuation \
    --prompt-text "Original transcript of the reference audio" \
    --output-dir /path/to/custom_voices

Add the output directory to the deploy config:

custom_voice_dir: /path/to/custom_voices

After startup, /v1/audio/voices lists alice, and /v1/audio/speech can use voice="alice" without sending ref_audio.

Gradio demo (gapless streaming via AudioWorklet)¶

python voxcpm2/gradio_demo.py

Uses an AudioWorklet-based player adapted from the Qwen3-TTS demo for gap-free playback. Raw PCM audio is streamed from the OpenAI Speech endpoint with stream=true and stream_format="audio".

Voxtral TTS¶

Voxtral-4B-TTS (Mistral). Uses the mistral_common SpeechRequest protocol; voice presets are model-specific.

Prerequisites¶

Latest mistral_common with SpeechRequest support:

pip install -e /path/to/mistral-common  # or upgrade from PyPI when available

Launch¶

vllm serve mistralai/Voxtral-4B-TTS-2603 --omni --port 8091

Deploy config auto-loads from vllm_omni/deploy/voxtral_tts.yaml.

Gradio demo¶

python voxtral_tts/gradio_demo.py

The demo handles voice-preset selection and reference-audio upload. voxtral_tts/text_preprocess.py provides the text-normalization helpers used by the demo (also available for other clients).

Notes¶

• Voice presets are listed on the HF model card (mistralai/Voxtral-4B-TTS-2603).
• Voice cloning is gated upstream and may require a recent mistral_common.
• A standalone CLI client is not yet shipped; the gradio demo is the canonical reference for now.

Breeze-TTS-2¶

Two-stage AR TTS (T5Gemma2 + Qwen3 talker with a depth decoder → bundled Qwen3-TTS codec) at 24 kHz mono, from BreezeBlue. Four modes are selected automatically from the request fields: plain (input), voice design (instructions), clone (ref_audio+ref_text), and voice direction (reference trio + instructions).

Launch¶

vllm serve BreezeBlue/Breeze-TTS-2 --omni --port 8091
# or:
./breeze_tts_2/run_server.sh

The deploy config at vllm_omni/deploy/breeze_tts_2.yaml auto-loads (async-chunk streaming with 1, 2, 4, then 5-frame inter-stage chunks).

Sending requests¶

# Plain synthesis (speaker tags S0..S9, default S0)
curl -X POST http://localhost:8091/v1/audio/speech \
    -H "Content-Type: application/json" \
    -d '{
        "input": "Hello, this is Breeze TTS 2 on vLLM-Omni.",
        "voice": "S0",
        "response_format": "wav",
        "sample_rate": 24000
    }' --output breeze_plain.wav

# Voice direction (clone a reference, then steer the delivery)
curl -X POST http://localhost:8091/v1/audio/speech \
    -H "Content-Type: application/json" \
    -d '{
        "input": "We need to discuss what happened last night.",
        "ref_audio": "file:///path/to/reference.wav",
        "ref_text": "The exact transcript of the reference audio.",
        "instructions": "Speak slowly with a restrained, serious tone.",
        "response_format": "wav"
    }' --output breeze_direction.wav

Streaming PCM¶

curl -N -X POST http://localhost:8091/v1/audio/speech \
    -H "Content-Type: application/json" \
    -d '{
        "input": "Streaming output from Breeze TTS 2.",
        "stream": true,
        "stream_format": "audio",
        "response_format": "pcm",
        "sample_rate": 24000
    }' --output breeze_stream.pcm

Notes¶

• Only sample_rate=24000 is accepted; speed must be 1.0.
• guidance_scale/cfg_scale must be 1.0; negative_prompt is rejected (CFG companion support is a follow-up).
• instructions without ref_audio is voice design; with ref_audio+ref_text it is voice direction.
• See recipes/BreezeBlue/Breeze-TTS-2.md for the full recipe and the offline example under examples/offline_inference/text_to_speech/breeze_tts_2/.

Example materials¶

#!/bin/bash
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project

# Launch the Audio8 TTS Preview server + Gradio demo together.
#
# Usage:
#   ./run_gradio_demo.sh
#   CUDA_VISIBLE_DEVICES=0 PORT=8092 GRADIO_PORT=7861 ./run_gradio_demo.sh

set -e

MODEL="${MODEL:-Audio8/Audio8-TTS-Preview-0.6b}"
PORT="${PORT:-8092}"
GRADIO_PORT="${GRADIO_PORT:-7861}"
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"

echo "Starting Audio8 TTS Preview server (port $PORT)..."
vllm serve "$MODEL" \
    --omni \
    --host 0.0.0.0 \
    --port "$PORT" &
SERVER_PID=$!

cleanup() {
    echo "Stopping server (PID $SERVER_PID)..."
    kill $SERVER_PID 2>/dev/null
    wait $SERVER_PID 2>/dev/null
}
trap cleanup EXIT

echo "Waiting for server to start..."
for _ in $(seq 1 120); do
    if curl -s "http://localhost:$PORT/health" > /dev/null 2>&1; then
        echo "Server ready."
        break
    fi
    sleep 2
done

echo "Starting Gradio demo (port $GRADIO_PORT)..."
python "$SCRIPT_DIR/gradio_demo.py" \
    --api-base "http://localhost:$PORT" \
    --port "$GRADIO_PORT"

#!/bin/bash
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project

# Launch a vLLM-Omni server for Audio8 TTS Preview 0.6B.
#
# Usage:
#   ./run_server.sh
#   CUDA_VISIBLE_DEVICES=0 PORT=8092 ./run_server.sh

set -e

MODEL="${MODEL:-Audio8/Audio8-TTS-Preview-0.6b}"
PORT="${PORT:-8092}"

echo "Starting Audio8 TTS Preview server with model: $MODEL"

# --trust-remote-code is deliberately NOT passed: vllm-omni registers its own
# `arktts` config, and transformers would otherwise prefer the checkpoint's
# remote code and bypass it.
vllm serve "$MODEL" \
    --omni \
    --host 0.0.0.0 \
    --port "$PORT"

# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project

"""Client for Audio8 TTS Preview via the /v1/audio/speech endpoint.

Examples:
    # Basic TTS
    python speech_client.py --text "Welcome to Audio8 TTS."

    # Zero-shot voice cloning (the transcript must match the reference audio)
    python speech_client.py --text "Welcome to Audio8 TTS." \
        --ref-audio ref.wav --ref-text "The exact transcript of the reference recording."

    # Streaming PCM output
    python speech_client.py --text "Welcome to Audio8 TTS." --stream --output output.pcm
"""

import argparse
import base64
import os

import httpx

DEFAULT_API_BASE = "http://localhost:8092"
DEFAULT_API_KEY = "EMPTY"
DEFAULT_MODEL = "Audio8/Audio8-TTS-Preview-0.6b"
#: The codec runs at 44.1 kHz, so raw PCM must be played back at that rate.
PCM_SAMPLE_RATE = 44100


def encode_audio_to_base64(audio_path: str) -> str:
    """Encode a local audio file as a base64 data URL."""
    if not os.path.exists(audio_path):
        raise FileNotFoundError(f"Audio file not found: {audio_path}")
    ext = audio_path.lower().rsplit(".", 1)[-1]
    mime_type = {"wav": "audio/wav", "mp3": "audio/mpeg", "flac": "audio/flac", "ogg": "audio/ogg"}.get(
        ext, "audio/wav"
    )
    with open(audio_path, "rb") as handle:
        audio_b64 = base64.b64encode(handle.read()).decode("utf-8")
    return f"data:{mime_type};base64,{audio_b64}"


def build_payload(args) -> dict:
    payload = {
        "model": args.model,
        "input": args.text,
        "response_format": args.response_format,
    }
    if args.ref_audio:
        if args.ref_audio.startswith(("http://", "https://")):
            payload["ref_audio"] = args.ref_audio
        else:
            payload["ref_audio"] = encode_audio_to_base64(args.ref_audio)
    if args.ref_text:
        payload["ref_text"] = args.ref_text
    if args.max_new_tokens is not None:
        payload["max_new_tokens"] = args.max_new_tokens
    if args.stream:
        payload["stream"] = True
        payload["stream_format"] = "audio"
        payload["response_format"] = "pcm"
    return payload


def run_tts(args) -> None:
    payload = build_payload(args)
    api_url = f"{args.api_base}/v1/audio/speech"
    headers = {"Content-Type": "application/json", "Authorization": f"Bearer {args.api_key}"}

    print(f"Model: {args.model}")
    print(f"Text: {args.text}")
    if args.ref_audio:
        print(f"Voice cloning: ref_audio={args.ref_audio!r} ref_text={args.ref_text!r}")

    if args.stream:
        output_path = args.output or "output.pcm"
        with (
            httpx.Client(timeout=300.0) as client,
            client.stream("POST", api_url, json=payload, headers=headers) as resp,
        ):
            if resp.status_code != 200:
                print(f"Error {resp.status_code}: {resp.read().decode()}")
                return
            total_bytes = 0
            with open(output_path, "wb") as handle:
                for chunk in resp.iter_bytes():
                    handle.write(chunk)
                    total_bytes += len(chunk)
        seconds = total_bytes / 2 / PCM_SAMPLE_RATE
        print(f"Streamed {total_bytes} bytes (~{seconds:.2f}s of s16le @ {PCM_SAMPLE_RATE} Hz) to {output_path}")
        print(f"Play with: ffplay -f s16le -ar {PCM_SAMPLE_RATE} -ac 1 {output_path}")
        return

    with httpx.Client(timeout=300.0) as client:
        response = client.post(api_url, json=payload, headers=headers)
    if response.status_code != 200:
        print(f"Error {response.status_code}: {response.text}")
        return
    if response.headers.get("content-type", "").startswith("application/json"):
        print(f"Error: {response.text}")
        return

    output_path = args.output or f"output.{args.response_format}"
    with open(output_path, "wb") as handle:
        handle.write(response.content)
    print(f"Audio saved to: {output_path} ({len(response.content)} bytes)")


def main() -> None:
    parser = argparse.ArgumentParser(description="Audio8 TTS Preview client")
    parser.add_argument("--api-base", default=DEFAULT_API_BASE, help="API base URL")
    parser.add_argument("--api-key", default=DEFAULT_API_KEY, help="API key")
    parser.add_argument("--model", "-m", default=DEFAULT_MODEL, help="Model name")
    parser.add_argument("--text", required=True, help="Text to synthesize")
    parser.add_argument("--ref-audio", default=None, help="Reference audio for voice cloning (path or URL)")
    parser.add_argument("--ref-text", default=None, help="Transcript of the reference audio")
    parser.add_argument("--max-new-tokens", type=int, default=None, help="Cap the number of generated codec frames")
    parser.add_argument("--stream", action="store_true", help="Stream PCM instead of returning a whole file")
    parser.add_argument(
        "--response-format",
        default="wav",
        choices=["wav", "mp3", "flac", "pcm", "aac", "opus"],
        help="Audio format (default: wav)",
    )
    parser.add_argument("--output", "-o", default=None, help="Output file path")
    run_tts(parser.parse_args())


if __name__ == "__main__":
    main()

#!/usr/bin/env bash
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
# Launch Breeze-TTS-2 online serving with the async-chunk deploy config.

set -euo pipefail

MODEL="${1:-BreezeBlue/Breeze-TTS-2}"
PORT="${PORT:-8091}"

exec vllm-omni serve "${MODEL}" \
    --deploy-config vllm_omni/deploy/breeze_tts_2.yaml \
    --omni --port "${PORT}"

#!/bin/bash
# Launch vLLM-Omni server for CosyVoice3 TTS
#
# Usage:
#   ./run_server.sh
#   CUDA_VISIBLE_DEVICES=0 ./run_server.sh
#
# Streaming (async-chunk) is on by default via vllm_omni/deploy/cosyvoice3.yaml.
# Set NO_ASYNC_CHUNK=1 to use the legacy synchronous path.

set -e

MODEL="${MODEL:-FunAudioLLM/Fun-CosyVoice3-0.5B-2512}"
PORT="${PORT:-8091}"

EXTRA_ARGS=()
if [[ -n "${NO_ASYNC_CHUNK:-}" ]]; then
    EXTRA_ARGS+=(--no-async-chunk)
fi

echo "Starting CosyVoice3 server with model: $MODEL"

vllm serve "$MODEL" \
    --host 0.0.0.0 \
    --port "$PORT" \
    --trust-remote-code \
    --omni \
    "${EXTRA_ARGS[@]}"

"""Client for CosyVoice3 TTS via /v1/audio/speech endpoint.

CosyVoice3 has no built-in voice presets: every request is voice cloning
driven by ``ref_audio`` + ``ref_text``. The defaults below point at the
official upstream zero-shot prompt so the script runs out of the box.

Examples:
    # Voice cloning with the default upstream prompt
    python speech_client.py --text "收到好友从远方寄来的生日礼物。"

    # Custom reference clip + transcript
    python speech_client.py --text "Hello, this is a cloned voice." \
        --ref-audio /path/to/reference.wav \
        --ref-text "Transcript of the reference audio."

    # Streaming PCM output
    python speech_client.py --text "Hello world" --stream --output output.pcm
"""

import argparse
import base64
import os

import httpx

DEFAULT_API_BASE = "http://localhost:8091"
DEFAULT_API_KEY = "EMPTY"
DEFAULT_MODEL = "FunAudioLLM/Fun-CosyVoice3-0.5B-2512"

# Official CosyVoice zero-shot prompt and its transcript.
DEFAULT_REF_AUDIO = "https://raw.githubusercontent.com/FunAudioLLM/CosyVoice/main/asset/zero_shot_prompt.wav"
DEFAULT_REF_TEXT = "希望你以后能够做的比我还好呦。"


def encode_audio_to_base64(audio_path: str) -> str:
    """Encode a local audio file to a base64 data URL."""
    if not os.path.exists(audio_path):
        raise FileNotFoundError(f"Audio file not found: {audio_path}")
    ext = audio_path.lower().rsplit(".", 1)[-1]
    mime_map = {"wav": "audio/wav", "mp3": "audio/mpeg", "flac": "audio/flac", "ogg": "audio/ogg"}
    mime_type = mime_map.get(ext, "audio/wav")
    with open(audio_path, "rb") as f:
        audio_b64 = base64.b64encode(f.read()).decode("utf-8")
    return f"data:{mime_type};base64,{audio_b64}"


def run_tts(args) -> None:
    """Generate speech via the /v1/audio/speech API."""
    payload = {
        "model": args.model,
        "input": args.text,
        "response_format": args.response_format,
    }

    if args.ref_audio.startswith(("http://", "https://")):
        payload["ref_audio"] = args.ref_audio
    else:
        payload["ref_audio"] = encode_audio_to_base64(args.ref_audio)
    payload["ref_text"] = args.ref_text

    if args.stream:
        payload["stream"] = True
        payload["stream_format"] = "audio"
        payload["response_format"] = "pcm"

    print(f"Model: {args.model}")
    print(f"Text: {args.text}")
    print(f"Voice cloning: ref_audio={args.ref_audio}, ref_text={args.ref_text}")
    print("Generating audio...")

    api_url = f"{args.api_base}/v1/audio/speech"
    headers = {
        "Content-Type": "application/json",
        "Authorization": f"Bearer {args.api_key}",
    }

    if args.stream:
        output_path = args.output or "output.pcm"
        with httpx.Client(timeout=300.0) as client:
            with client.stream("POST", api_url, json=payload, headers=headers) as resp:
                if resp.status_code != 200:
                    print(f"Error: {resp.status_code}")
                    print(resp.read().decode())
                    return
                total_bytes = 0
                with open(output_path, "wb") as f:
                    for chunk in resp.iter_bytes():
                        f.write(chunk)
                        total_bytes += len(chunk)
                print(f"Streamed {total_bytes} bytes to: {output_path}")
    else:
        with httpx.Client(timeout=300.0) as client:
            response = client.post(api_url, json=payload, headers=headers)

        if response.status_code != 200:
            print(f"Error: {response.status_code}")
            print(response.text)
            return

        try:
            text = response.content.decode("utf-8")
            if text.startswith('{"error"'):
                print(f"Error: {text}")
                return
        except UnicodeDecodeError:
            pass

        output_path = args.output or "output.wav"
        with open(output_path, "wb") as f:
            f.write(response.content)
        print(f"Audio saved to: {output_path}")


def main():
    parser = argparse.ArgumentParser(description="CosyVoice3 TTS client")
    parser.add_argument("--api-base", default=DEFAULT_API_BASE, help="API base URL")
    parser.add_argument("--api-key", default=DEFAULT_API_KEY, help="API key")
    parser.add_argument("--model", "-m", default=DEFAULT_MODEL, help="Model name")
    parser.add_argument("--text", required=True, help="Text to synthesize")
    parser.add_argument(
        "--ref-audio",
        default=DEFAULT_REF_AUDIO,
        help="Reference audio for voice cloning (path or URL)",
    )
    parser.add_argument(
        "--ref-text",
        default=DEFAULT_REF_TEXT,
        help="Transcript of the reference audio",
    )
    parser.add_argument("--stream", action="store_true", help="Enable streaming (PCM output)")
    parser.add_argument(
        "--response-format",
        default="wav",
        choices=["wav", "mp3", "flac", "pcm", "aac", "opus"],
        help="Audio format (default: wav)",
    )
    parser.add_argument("--output", "-o", default=None, help="Output file path")
    args = parser.parse_args()
    run_tts(args)


if __name__ == "__main__":
    main()

#!/bin/bash
# Launch Fish Speech S2 Pro server + Gradio demo together.
#
# Usage:
#   ./run_gradio_demo.sh
#   CUDA_VISIBLE_DEVICES=0 PORT=8091 GRADIO_PORT=7860 ./run_gradio_demo.sh

set -e

MODEL="${MODEL:-fishaudio/s2-pro}"
PORT="${PORT:-8091}"
GRADIO_PORT="${GRADIO_PORT:-7860}"
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"

echo "Starting Fish Speech S2 Pro server (port $PORT)..."
FLASHINFER_DISABLE_VERSION_CHECK=1 \
vllm serve "$MODEL" \
    --omni \
    --host 0.0.0.0 \
    --port "$PORT" &
SERVER_PID=$!

cleanup() {
    echo "Stopping server (PID $SERVER_PID)..."
    kill $SERVER_PID 2>/dev/null
    wait $SERVER_PID 2>/dev/null
}
trap cleanup EXIT

# Wait for server to be ready.
echo "Waiting for server to start..."
for i in $(seq 1 120); do
    if curl -s "http://localhost:$PORT/health" > /dev/null 2>&1; then
        echo "Server ready."
        break
    fi
    sleep 2
done

echo "Starting Gradio demo (port $GRADIO_PORT)..."
python "$SCRIPT_DIR/gradio_demo.py" \
    --api-base "http://localhost:$PORT" \
    --port "$GRADIO_PORT"

#!/bin/bash
# Launch vLLM-Omni server for Fish Speech S2 Pro
#
# Usage:
#   ./run_server.sh
#   CUDA_VISIBLE_DEVICES=0 ./run_server.sh

set -e

MODEL="${MODEL:-fishaudio/s2-pro}"
PORT="${PORT:-8091}"

echo "Starting Fish Speech S2 Pro server with model: $MODEL"

FLASHINFER_DISABLE_VERSION_CHECK=1 \
vllm serve "$MODEL" \
    --omni \
    --host 0.0.0.0 \
    --port "$PORT"

"""Client for Fish Speech S2 Pro via /v1/audio/speech endpoint.

Examples:
    # Basic TTS
    python speech_client.py --text "Hello, how are you?"

    # Voice cloning
    python speech_client.py --text "Hello, how are you?" \
        --ref-audio ref.wav --ref-text "This is the reference transcript."

    # Streaming PCM output
    python speech_client.py --text "Hello world" --stream --output output.pcm
"""

import argparse
import base64
import os

import httpx

DEFAULT_API_BASE = "http://localhost:8091"
DEFAULT_API_KEY = "EMPTY"


def encode_audio_to_base64(audio_path: str) -> str:
    """Encode a local audio file to base64 data URL."""
    if not os.path.exists(audio_path):
        raise FileNotFoundError(f"Audio file not found: {audio_path}")
    ext = audio_path.lower().rsplit(".", 1)[-1]
    mime_map = {"wav": "audio/wav", "mp3": "audio/mpeg", "flac": "audio/flac", "ogg": "audio/ogg"}
    mime_type = mime_map.get(ext, "audio/wav")
    with open(audio_path, "rb") as f:
        audio_b64 = base64.b64encode(f.read()).decode("utf-8")
    return f"data:{mime_type};base64,{audio_b64}"


def run_tts(args) -> None:
    """Generate speech via /v1/audio/speech API."""
    payload = {
        "model": args.model,
        "input": args.text,
        "response_format": args.response_format,
    }

    # Voice cloning parameters.
    if args.ref_audio:
        if args.ref_audio.startswith(("http://", "https://")):
            payload["ref_audio"] = args.ref_audio
        else:
            payload["ref_audio"] = encode_audio_to_base64(args.ref_audio)
    if args.ref_text:
        payload["ref_text"] = args.ref_text

    if args.stream:
        payload["stream"] = True
        payload["stream_format"] = "audio"
        payload["response_format"] = "pcm"

    print(f"Model: {args.model}")
    print(f"Text: {args.text}")
    if args.ref_audio:
        print(f"Voice cloning: ref_audio={args.ref_audio}, ref_text={args.ref_text}")
    print("Generating audio...")

    api_url = f"{args.api_base}/v1/audio/speech"
    headers = {
        "Content-Type": "application/json",
        "Authorization": f"Bearer {args.api_key}",
    }

    if args.stream:
        output_path = args.output or "output.pcm"
        with httpx.Client(timeout=300.0) as client:
            with client.stream("POST", api_url, json=payload, headers=headers) as resp:
                if resp.status_code != 200:
                    print(f"Error: {resp.status_code}")
                    print(resp.read().decode())
                    return
                total_bytes = 0
                with open(output_path, "wb") as f:
                    for chunk in resp.iter_bytes():
                        f.write(chunk)
                        total_bytes += len(chunk)
                print(f"Streamed {total_bytes} bytes to: {output_path}")
    else:
        with httpx.Client(timeout=300.0) as client:
            response = client.post(api_url, json=payload, headers=headers)

        if response.status_code != 200:
            print(f"Error: {response.status_code}")
            print(response.text)
            return

        try:
            text = response.content.decode("utf-8")
            if text.startswith('{"error"'):
                print(f"Error: {text}")
                return
        except UnicodeDecodeError:
            pass

        output_path = args.output or "output.wav"
        with open(output_path, "wb") as f:
            f.write(response.content)
        print(f"Audio saved to: {output_path}")


def main():
    parser = argparse.ArgumentParser(description="Fish Speech S2 Pro TTS client")
    parser.add_argument("--api-base", default=DEFAULT_API_BASE, help="API base URL")
    parser.add_argument("--api-key", default=DEFAULT_API_KEY, help="API key")
    parser.add_argument("--model", "-m", default="fishaudio/s2-pro", help="Model name")
    parser.add_argument("--text", required=True, help="Text to synthesize")
    parser.add_argument("--ref-audio", default=None, help="Reference audio for voice cloning (path or URL)")
    parser.add_argument("--ref-text", default=None, help="Transcript of reference audio")
    parser.add_argument("--stream", action="store_true", help="Enable streaming (PCM output)")
    parser.add_argument(
        "--response-format",
        default="wav",
        choices=["wav", "mp3", "flac", "pcm", "aac", "opus"],
        help="Audio format (default: wav)",
    )
    parser.add_argument("--output", "-o", default=None, help="Output file path")
    args = parser.parse_args()
    run_tts(args)


if __name__ == "__main__":
    main()

#!/bin/bash
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project

# Launch vLLM-Omni server for Gepard-1.0 TTS (zero-shot default voice).
#
# Usage:
#   ./run_server.sh
#   CUDA_VISIBLE_DEVICES=0 ./run_server.sh
#
# Packaged deploy config: vllm_omni/deploy/gepard.yaml (async_chunk=false,
# 22.05 kHz mono, default seed 42 until the YAML seed is removed).
# Cold start downloads the talker and NeMo NanoCodec; the serve default of
# --stage-init-timeout 300 is too tight (offline end2end.py uses 900).
#
# On a host whose CUDA toolkit cannot JIT FlashInfer (no nvcc/ninja, or a
# consumer Blackwell sm_120 card), also:
#   export VLLM_USE_FLASHINFER_SAMPLER=0

set -e

MODEL="${MODEL:-nineninesix/gepard-1.0}"
PORT="${PORT:-8091}"

echo "Starting Gepard-1.0 server with model: $MODEL"

vllm-omni serve "$MODEL" \
    --host 0.0.0.0 \
    --port "$PORT" \
    --trust-remote-code \
    --omni \
    --stage-init-timeout 900 \
    --deploy-config vllm_omni/deploy/gepard.yaml

# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project

"""Client for Gepard-1.0 TTS via /v1/audio/speech.

Gepard is zero-shot: omit ``voice`` or pass ``"default"``. Output is 22.05 kHz
mono. ``seed`` is optional; the packaged deploy YAML currently pins ``seed: 42``
until that default is removed, so requests without an explicit seed are still
deterministic.

Examples:
    python speech_client.py --text "Hello, this is Gepard speaking."
    python speech_client.py --text "Hello, this is Gepard speaking." --seed 7
    python speech_client.py --text "Hello, this is Gepard speaking." --stream --output output.pcm
"""

from __future__ import annotations

import argparse

import httpx

DEFAULT_API_BASE = "http://localhost:8091"
DEFAULT_API_KEY = "EMPTY"
DEFAULT_MODEL = "nineninesix/gepard-1.0"


def run_tts(args) -> None:
    payload = {
        "model": args.model,
        "input": args.text,
        "voice": args.voice,
        "response_format": args.response_format,
    }
    if args.seed is not None:
        payload["seed"] = args.seed
    if args.max_new_tokens is not None:
        payload["max_new_tokens"] = args.max_new_tokens
    if args.stream:
        payload["stream"] = True
        payload["stream_format"] = "audio"
        if args.response_format not in ("pcm", "wav"):
            print(f"Note: streaming requires pcm/wav; overriding --response-format {args.response_format} to 'pcm'")
            payload["response_format"] = "pcm"

    print(f"Model: {args.model}")
    print(f"Text: {args.text}")
    print(f"Voice: {args.voice}")
    if args.seed is not None:
        print(f"Seed: {args.seed}")
    print("Generating audio...")

    api_url = f"{args.api_base}/v1/audio/speech"
    headers = {
        "Content-Type": "application/json",
        "Authorization": f"Bearer {args.api_key}",
    }

    if args.stream:
        output_path = args.output or ("output.wav" if payload["response_format"] == "wav" else "output.pcm")
        with httpx.Client(timeout=300.0) as client:
            with client.stream("POST", api_url, json=payload, headers=headers) as resp:
                if resp.status_code != 200:
                    print(f"Error: {resp.status_code}")
                    print(resp.read().decode())
                    return
                total_bytes = 0
                with open(output_path, "wb") as f:
                    for chunk in resp.iter_bytes():
                        f.write(chunk)
                        total_bytes += len(chunk)
                print(f"Streamed {total_bytes} bytes to: {output_path}")
        return

    with httpx.Client(timeout=300.0) as client:
        response = client.post(api_url, json=payload, headers=headers)

    if response.status_code != 200:
        print(f"Error: {response.status_code}")
        print(response.text)
        return

    output_path = args.output or "output.wav"
    with open(output_path, "wb") as f:
        f.write(response.content)
    print(f"Audio saved to: {output_path}")


def main() -> None:
    parser = argparse.ArgumentParser(description="Gepard-1.0 TTS client")
    parser.add_argument("--api-base", default=DEFAULT_API_BASE, help="API base URL")
    parser.add_argument("--api-key", default=DEFAULT_API_KEY, help="API key")
    parser.add_argument("--model", "-m", default=DEFAULT_MODEL, help="Model name")
    parser.add_argument("--text", required=True, help="Text to synthesize")
    parser.add_argument("--voice", default="default", help="Voice name (only 'default' is supported)")
    parser.add_argument("--seed", type=int, default=None, help="Sampling seed")
    parser.add_argument(
        "--max-new-tokens",
        type=int,
        default=None,
        help="Frame budget (1 token = 1 frame ≈ 46.4 ms at 21.5 fps)",
    )
    parser.add_argument("--stream", action="store_true", help="Enable streaming (PCM output)")
    parser.add_argument(
        "--response-format",
        default="wav",
        choices=["wav", "mp3", "flac", "pcm"],
        help="Audio format (default: wav). Streaming is pcm/wav only. Opus is not supported at 22.05 kHz.",
    )
    parser.add_argument("--output", "-o", default=None, help="Output file path")
    args = parser.parse_args()
    run_tts(args)


if __name__ == "__main__":
    main()