vllm_omni.model_executor.models.personaplex.personaplex_mimi ¶

Streaming Mimi codec for PersonaPlex (moshi-free).

Frame-clocked duplex needs a codec that encodes/decodes exactly one 80 ms frame per call with state carried across the conversation. The moshi package provided that; this module removes the dependency by combining:

• transformers MimiModel (kyutai/mimi — the same checkpoint family PersonaPlex ships) for the quantizer and every SEANet conv weight. Its streaming support only covers the encoder convs, so conv streaming is done here instead, uniformly.
• Our own streaming wrappers mirroring the Moshi reference semantics (MIT), verified against recorded reference outputs:
• Conv1d: left-context carry of effective_kernel - stride samples, zero-initialized at stream start (pad_mode="constant").
• ConvTranspose1d: overlap-add tail carry of kernel - stride output samples, with the double-counted bias subtracted on merge.
• transformer: the encoder/decoder transformers use a 250-position sliding context over a ring KV with absolute-offset RoPE. Hugging Face's cache path diverges once the window engages (position 250), so the transformers are reimplemented here on the same ring-KV design as personaplex_temporal.py and loaded directly from the PersonaPlex checkpoint's fused layout (LayerNorm + per-layer LayerScale + GELU FFN).

All per-stream state is [B, ...] with per-row reset (reset_slot), so the codec composes with elastic slot recycling in batched duplex serving.

CODEBOOKS = 8

DEFAULT_HF_REPO = 'kyutai/mimi'

FRAME_SIZE = 1920

decoder_transformer = _MimiStreamingTransformer().to(
    self.device, self.dtype
)

device = torch.device(device)

dtype = next(self.model.parameters()).dtype

encoder_transformer = _MimiStreamingTransformer().to(
    self.device, self.dtype
)

model = self.model.to(self.device).eval()

decode_frame(
    codes: Tensor, active: Tensor | None = None
) -> Tensor

decode_frames(
    codes: Tensor, active: Tensor | None = None
) -> Tensor

encode_frame(
    pcm: Tensor, active: Tensor | None = None
) -> Tensor

reset_slot(b: int) -> None

reset_streaming() -> None

streaming_init(batch_size: int) -> None