Skip to content

Engine orchestration

Engine orchestration coordinates configured AR and diffusion stages while keeping public clients independent from stage processes and transports.

Contract status

This document is a draft description of current behavior. It also identifies the boundary affected by the in-flight stage client/process refactor in #5441. Names and responsibilities proposed only by that PR are not current contracts.

The orchestration loop also has an opt-in event-driven mode (VLLM_OMNI_EVENT_DRIVEN_ORCH=1, default off except for Qwen3-TTS) proposed in #5221. It changes poll cadence only: the routing, ordering, and terminal-state contracts below hold identically on both loops.

Ownership boundary

This document owns AsyncOmniEngine, Orchestrator, request-state creation, cross-stage routing, output ordering, companion tracking, control/RPC correlation, cancellation propagation, and terminal-state convergence.

It does not own stage placement, replica selection, stage-process startup, payload schemas, public protocol rendering, connector transport, or semantic error classification. Those responsibilities belong to the stage runtime, I/O, entrypoint, connector, and error contracts.

Candidate invariants

These identifiers are proposals while the document is draft.

ORCH-INV-001: The orchestrator owns cross-stage routing

Rule: Entrypoints and stage clients MUST NOT independently forward a request to a downstream logical stage.

ORCH-INV-002: Stage clients do not own routing policy

Rule: Stage clients MUST implement communication and lifecycle operations without selecting the next logical stage.

ORCH-INV-100: Terminal state is monotonic

Rule: Once a request reaches a terminal state, orchestration MUST NOT forward new work for that request.

Invariant namespace

ORCH-INV reserves 001-099 for dependency direction, 100-199 for request state and ordering, 200-299 for failure/cancellation/cleanup, and 300-399 for extension points, shutdown ordering, and upstream alignment. Numbers become append-only after normative promotion.

Safe-change guide

Test routing, output ordering, queue correlation, cancellation, failure propagation, shutdown ordering, and representative multi-stage execution. AR abort of a final-stage LLM request must deliver a terminal output with finish_reason="abort" and the generated prefix so collocated training can resume; diffusion abort remains whole-sample retry. Aborting a parallel-sampling child must drop parent_requests once no children remain. EngineCore control RPCs (sleep / wake_up / pause_scheduler / resume_scheduler) must propagate worker exceptions rather than returning {"supported": False}. When a request has multiple final output stages, only the last abort message is request-terminal. Output-processor abort state is committed only after the physical EngineCore abort succeeds. Changes that cross into stage runtime or public error behavior require review from that contract's owners.

Promotion gate

  • Reconcile current names and boundaries after #5441 merges, closes, or is replaced.
  • Demonstrate one terminal outcome per request across success, abort, request-scoped failure, and fatal engine failure.
  • Verify queue correlation and orchestrator-before-runtime shutdown ordering in the cited validation paths.
  • Obtain approval from a technical owner and an independent validation reviewer.