Skip to content

vllm_omni.protocol.realtime.capabilities

What a Realtime consumer can serve, and the session check that reads it.

session.update is the one client event whose acceptance depends on who is behind the socket. The audio formats a server can decode, and whether it can do server-side turn detection at all, are properties of the consumer --- the duplex engine, or some other runtime --- not of the protocol. The codec still has to do the checking, because it owns the error envelope and the order the checks run in.

:class:RealtimeProtocolCapabilities is how a consumer states those answers. It is the post-#7413 form of the supports(turn_detection, formats) half of RFC #6592's proposed RealtimeModelAdapter. The other three halves of that proposal --- building a prompt, starting a response, cancelling one --- are not here and must not come here: for a duplex model they are already owned by DuplexModelPlugin together with DuplexSessionRunner, and a second interface over the same responsibility is exactly the fork this layer exists to prevent.

The duplex binding of this object is vllm_omni.engine.duplex.realtime_commands.DUPLEX_REALTIME_CAPABILITIES.

TurnDetectionValidator module-attribute 

TurnDetectionValidator = Callable[
    [Mapping[str, object]], str | None
]

RealtimeProtocolCapabilities dataclass

One consumer's answer to what a session object may ask for.

The defaults are permissive, not restrictive: every format the codec can decode, and no turn-detection validation at all. A consumer that leaves validate_turn_detection at None therefore accepts whatever turn_detection the session object carries --- including values it cannot actually serve, such as semantic_vad. A consumer that supports only some turn-detection modes (or none) must supply its own validator; see vllm_omni.engine.duplex.realtime_commands.DUPLEX_REALTIME_CAPABILITIES.

input_audio_formats class-attribute instance-attribute 

input_audio_formats: frozenset[str] = field(
    default_factory=lambda: frozenset(
        REALTIME_INPUT_AUDIO_FORMATS
    )
)

output_audio_formats class-attribute instance-attribute 

output_audio_formats: frozenset[str] = field(
    default_factory=lambda: frozenset(
        REALTIME_OUTPUT_AUDIO_FORMATS
    )
)

validate_turn_detection class-attribute instance-attribute 

validate_turn_detection: TurnDetectionValidator | None = (
    None
)

RealtimeSessionRejection dataclass

Why a session object was refused: the internal error code and the message.

param names the offending session field where there is one, for the error.param slot of the OpenAI error envelope.

code instance-attribute 

code: str

message instance-attribute 

message: str

param class-attribute instance-attribute 

param: str | None = None

validate_session_payload 

validate_session_payload(
    session_payload: Mapping[str, object],
    *,
    capabilities: RealtimeProtocolCapabilities,
) -> RealtimeSessionRejection | None

Check a session object against one consumer; None when it is acceptable.

Audio formats are checked before turn detection, so a session object that is wrong in both ways reports the format problem --- keep that order, it is what clients already see.