Skip to content

Entrypoints and serving boundaries

Entrypoints translate offline, CLI, and serving requests into stable engine operations and translate engine outputs into public responses.

Contract status

This document describes current entrypoint responsibilities plus the boundary under review in the open roadmap #5227 and helper-move PR #5453. In-flight helper locations are not treated as current paths.

Ownership boundary

This document owns offline API semantics, CLI and serve composition, supported OpenAI-compatible routes, request validation and normalization, response conversion, streaming/session behavior, and engine handoff.

When --api-server-count is greater than one, the serve composition root starts the frontend process manager while a parent-owned StageRuntime launches the shared local stage engines. Each frontend receives only its own stage-client channel configuration and does not launch or retire backend processes. CLI validation rejects unsupported distributed, diffusion, fault-tolerant, elastic-EP, Ray, and runtime-LoRA combinations before startup.

It does not own configuration precedence, cross-stage routing, stage placement, payload implementation, or semantic error classification. entrypoints/openai/errors.py is an explicit primary-path exception owned by error_contracts.md.

It also owns vllm_omni/protocol/**, the shared wire codec promoted by RFC #6592 P0a. It follows the three tiers docs/serving/realtime_duplex_api.md already defines. protocol/realtime/** is Tier 1, identical to OpenAI: 30 server events, 10 client commands, their typed fields and pure wire rendering, audio-format negotiation, conversation-item rules and the error-envelope shape. protocol/duplex/** is Tier 2 (OpenAI names carrying vLLM-Omni extensions, each a subclass of its Tier 1 twin declaring only what it adds) and Tier 3 (ours alone, including our error-code vocabulary), kept separate so a GA-only consumer is not handed a vocabulary its clients never send. That package sits outside entrypoints/ on purpose --- the engine depends on it, so it belongs to neither layer --- but deciding what a Realtime client may put on the wire is this document's subject, and the same reasoning already places vllm_omni/errors.py, inputs/ and outputs/ under contract documents rather than under a consumer.

It is deliberately not owned by ../fullduplex.md, even though duplex is the codec's only consumer today. RFC #6592 asks for a named owner precisely so a second Realtime consumer does not need duplex review to change the codec (#6592 open question 8). Ownership here is review scope, not a dependency: the codec may not import the engine, the entrypoints or the model code, which tests/protocol/realtime/test_protocol_import_boundary.py enforces. When a second consumer lands (#6592 P0b) the package is expected to graduate to its own module document with owners drawn from both consumers.

Candidate invariants

These identifiers are proposals while the document is draft.

ENTRY-INV-001: Entrypoints adapt but do not orchestrate

Rule: Entrypoints MUST NOT implement cross-stage routing or stage lifecycle policy.

ENTRY-INV-002: The shared Realtime codec does not depend on its consumers

Rule: vllm_omni/protocol/** MUST NOT import vllm_omni.engine, vllm_omni.entrypoints, vllm_omni.model_executor, vllm_omni.worker or vllm_omni.clients; protocol/duplex/** MAY depend on protocol/realtime/** but never the reverse, and a Tier 1 class MUST NOT carry a vLLM-Omni extension field; and each protocol/codec behavior MUST have exactly one implementation that every consumer uses. A wire type MUST NOT carry a consumer's internal representation: the duplex mailbox rendering (DuplexCommand.payload(), whose channel differs from the client event for session.update and the conversation.item.* commands) stays engine-side. A duplex consumer (engine/duplex/**, entrypoints/duplex/**, the duplex clients) MUST import vllm_omni.protocol.duplex and MUST NOT import vllm_omni.protocol.realtime directly, so the tier boundary has exactly one extension point.

ENTRY-INV-100: Public requests are normalized once

Rule: Public protocol values MUST be validated and converted to internal request contracts before engine submission.

ENTRY-INV-101: Streaming preserves request identity

Rule: Every streamed response MUST remain associated with the request and output modality that produced it.

The reviewer-proposed rule that model-specific behavior should stay behind a common adapter or processor abstraction remains an unnumbered candidate until the entrypoint refactor makes that boundary enforceable.

Invariant namespace

ENTRY-INV reserves 001-099 for boundary and dependency direction, 100-199 for normalization, sessions, and streaming, 200-299 for disconnect, cancellation, rendering, and cleanup, and 300-399 for upstream route and renderer compatibility. Numbers become append-only after normative promotion.

Safe-change guide

Test request validation, protocol conversion, model-adapter routing, streaming, session identity, disconnect/cancellation, and error mapping for each affected offline or serving entrypoint. Sleep must wait for in-flight generate() admission before EngineCore offload; wake_up does not resume admission — callers must resume_generation(). Sleeping tags are tracked per stage so wake_up(stage_ids=[0]) does not skip a later wake_up(stage_ids=[1]). Streaming input pumps take an admission slot immediately before each EngineCore ADD or update, not while waiting for the next client chunk. Frontend abort keeps request_states until generate() consumes the terminal output. Multi-API serving must preserve one client rank and one channel set per frontend, include every frontend in readiness/failure observation, and keep stage lifecycle ownership in the parent composition root.

Promotion gate

  • Reconcile the page after #5227's P0 ownership work and #5453's final disposition.
  • Publish a supported route/transport matrix rather than claiming blanket OpenAI parity.
  • Verify one normalization and handoff path per route family and one terminal outcome per streaming transport.
  • Promote the api_server.py composition-root rule only after helper moves and import boundaries enforce it.
  • Obtain approval from a technical owner and an independent validation reviewer.