CI Settings¶

This document describes where Buildkite YAML lives in the repo, how each platform organizes CI, and how to add a new job. It does not document agent queues, GPU types, or container plugin details—those belong in infra / preset files (for example .buildkite/common/ci_mirror_hardwares.yml).

For CI levels (L1–L5) and triggers, see Test System Overview. For test authoring, see Test Writing Guide. For running tests locally or replaying CI jobs, see Test Execution Guide.

Directory layout¶

Canonical layout (prefer these paths for new changes):

.buildkite/
├── common/                          # Shared across platforms
│   ├── scripts/
│   │   ├── skip_ci.py               # skip-ci decision (docs / skip-mark / CI YAML paths)
│   │   ├── upload_pipeline.py       # Bootstrap + test-pipeline uploader (CUDA/NPU)
│   │   └── resolve_skip_ci.sh       # Shell helpers for AMD/Intel bootstrap
│   ├── ci_mirror_hardwares.yml      # CUDA/NPU hardware presets (referenced by name only)
│   └── ci_source_file_dependencies.yml  # L2–L5 path-prefix presets (referenced by name only)
├── cuda/                            # Primary NVIDIA CUDA CI
│   ├── pipeline.yml                 # Bootstrap entry (hook upload)
│   ├── bootstrap-upload-steps.yml   # Bootstrap child steps (upload_pipeline --upload)
│   ├── test-ready.yml               # L2
│   ├── test-merge.yml               # L3
│   ├── test-nightly.yml             # L4
│   ├── test-weekly.yml              # L5
│   └── rebase-pipeline.yml
├── npu/
│   ├── pipeline-npu.yml             # Bootstrap entry (hook upload)
│   ├── bootstrap-upload-steps.yml   # Bootstrap child steps (upload_pipeline --upload)
│   ├── pipeline-npu-a3.yml          # A3 variant (when used)
│   ├── test-npu-ready.yml           # L2
│   ├── test-npu-nightly.yml         # L4
│   └── scripts/
├── amd/
│   ├── test-amd-ready.yml           # L2 job definitions (template input)
│   ├── test-amd-merge.yml           # L3 job definitions
│   ├── test-amd-nightly.yml         # Nightly full-model jobs (experimental)
│   ├── test-template-amd-omni.j2    # Renders final pipeline.yaml
│   └── scripts/
│       ├── bootstrap-amd-omni.sh    # Entry: skip-ci → Jinja → upload
│       └── run-amd-test.sh          # Wraps pytest inside ROCm docker
├── intel/
│   ├── pipeline-intel.yml           # Static Intel XPU pipeline
│   └── scripts/
│       ├── bootstrap-intel-omni.sh
│       └── run-xpu-test.sh
└── release/
    ├── release-pipeline.yml
    └── scripts/

Placement rules¶

Platform comparison¶

Platform configuration style¶

# inferred: H100+B200 + cards_2 → h100_2, or b200_2 when MIRROR_HW=b200
commands:
  - pytest -sv tests/e2e -m "full_model and H100 and B200 and omni and cards_2"
# no mirror_hardwares

# inferred: not cards_1 (no positive cards_*) → that chip's highest pool (l4_4 / h100_4 / b200_4)
commands:
  - pytest -sv tests/e2e -m "full_model and diffusion and L4 and not cards_1"

# forced preset (ignores -m SKU/cards for pool selection)
mirror_hardwares: h100_1

python3 .buildkite/common/scripts/upload_pipeline.py .buildkite/cuda/test-ready.yml

python3 .buildkite/common/scripts/upload_pipeline.py .buildkite/npu/test-npu-ready.yml

Cross-cutting conventions¶

Diff-aware CI¶

PR diffs drive two independent skip layers. Both read changed files from git, but at different pipeline stages and with different granularity:

Changed files (both layers):

Label triggers (ready, merge-test) are unchanged—diff-aware logic only reduces what runs after a pipeline is already scheduled.

Bootstrap skip¶

skip_ci.py classifies the git diff into buckets (docs / qualifying skip-mark / whitelisted L2·L3·L4/L5 YAML / other), then picks one decision path. CUDA/NPU apply that decision by injecting Buildkite if on bootstrap-upload-steps.yml step keys; AMD/Intel call gate_bootstrap_ci <platform> <l2|l3>.

Decision overview¶

skip_all exceptions: For CUDA/NPU, PR labels (nightly-test, merge-test, weekly-test, ready, …) do not revive jobs under skip_all; on main, scheduled NIGHTLY=1 still builds the image and uploads L4, while WEEKLY=1 or NON_CRITICAL=1 still builds the image and uploads L5 (CUDA). WEEKLY=1 on main also uploads L2/L3 with --e2e. AMD applies skip_ci.py gate only to selected ready/merge specs, so an explicitly selected AMD nightly spec (nightly-test, NIGHTLY=1, or a nightly debug override) remains runnable even for a docs-only diff.

Yaml-gated nightly/weekly: L4/L5 upload steps keep their normal label / NIGHTLY / WEEKLY / NON_CRITICAL conditions (for example PR nightly-test still uploads nightly). Only L2/L3 upload steps are matrix-gated.

Whitelisted CI YAML → platform¶

Register new files in L2_YAML_FILES, L3_YAML_FILES, or L45_YAML_FILES in skip_ci.py.

Yaml-gated category branches¶

L2? / L3? / L45? = whether any file in that whitelist changed. Untouched platforms/levels end up off.

Yaml-gated examples (platform-level)¶

Docs/skip-mark mixed with any row below follows the same matrix.

How platforms consume the decision¶

Unit coverage: tests/buildkite/test_skip_ci.py.

Step filtering¶

CUDA L2 (.buildkite/cuda/test-ready.yml), L3 (.buildkite/cuda/test-merge.yml), L4 (.buildkite/cuda/test-nightly.yml), L5 (.buildkite/cuda/test-weekly.yml), and NPU L4 (.buildkite/npu/test-npu-nightly.yml). Bootstrap upload entry: upload_pipeline.py --upload .buildkite/cuda/bootstrap-upload-steps.yml (or npu/bootstrap-upload-steps.yml).

When filtering applies: PR label uploads (ready, merge-test, nightly-test, weekly-test) and post-merge main L3 (test-merge.yml). Scheduled main + env uploads that pass --all (NIGHTLY=1, WEEKLY=1 / NON_CRITICAL=1) or --e2e (WEEKLY=1 on L2/L3) keep every selected job and still strip the key. If no job-key prefix matches the diff, a change to the pipeline YAML being uploaded, or any path under the source_filter_fallback key in ci_source_file_dependencies.yml, keeps every job in that upload. source_filter_fallback is not a job key — do not attach it to a step. When any job-key prefix already matches, normal filtering wins (the fallback does not expand the selection).

Uploader-only keys — removed before Buildkite sees the YAML; never used at runtime on agents:

Policy¶

• Always uploaded (no key): L2/L3 groups outside E2E Test—Simple Test, Diffusion unit tests, Engine/Model Executor, Distributed, Custom Pipeline, Entrypoints (L2), LoRA / Entrypoints (L3).
• Diff-gated: every E2E Test leaf job in L2/L3, L4 nightly leaf jobs (CUDA and NPU), and L5 weekly jobs whose if includes the weekly-test label (Reliability / Perf). Jobs gated only by WEEKLY=1 or NON_CRITICAL=1 have no source_file_dependencies. Registry layout: YAML anchors are model business code only (vllm_omni/model_executor/models/, vllm_omni/diffusion/models/, plus stage_input_processors/ and vllm_omni/deploy/*.yaml when applicable). Job keys alias the anchor and also list the scripts that job runs (pytest files, run_cov_split.sh --offline / --online targets, example README runners). Feature keys stay cross-cutting. A group may define the key instead; the whole group drops if no prefix matches.

YAML examples¶

      - label: "Diffusion · Qwen Image Test"
        source_file_dependencies: diffusion_qwen_image_function
        # diffusion_qwen_image_function: *qwen_image + tests/e2e/online_serving/test_qwen_image.py
        commands:
          - pytest -s -v tests/e2e/online_serving/test_qwen_image.py -m 'core_model' ...
        mirror_hardwares: h100_1

      - label: "TTS · Qwen3-TTS CustomVoice Test"
        source_file_dependencies: tts_qwen3_tts_function
        # tts_qwen3_tts_function: *qwen3_tts + tests/e2e/online_serving/test_qwen3_tts_customvoice.py
        commands:
          - pytest -s -v tests/e2e/online_serving/test_qwen3_tts_customvoice.py ...
        mirror_hardwares: l4_1

Local dry-run¶

python3 .buildkite/common/scripts/upload_pipeline.py .buildkite/cuda/test-ready.yml
python3 .buildkite/common/scripts/upload_pipeline.py .buildkite/cuda/test-merge.yml | grep source_file_dependencies
# (no output expected)

On PR builds, upload_pipeline.py logs skip '…' (no changes under …) for omitted steps.

Labels and grouping¶

• Use groups for dashboard readability; keep E2E Test as the group name CUDA diff filtering expects for --e2e weekly runs on main (WEEKLY=1).
• Prefix labels by model domain: Omni ·, TTS ·, Diffusion ·, Simple ·, etc., matching existing steps.

Per-model coverage¶

Pilot (v1): one Merge-tier (L3) job in .buildkite/cuda/test-merge.yml uploads per-model, per-entry-mode coverage as Buildkite artifacts — "TTS · Qwen3-TTS Base Test". Coverage + artifact upload run only when bootstrap uploaded that job with --e2e (WEEKLY=1 and BUILDKITE_BRANCH=main, via run_cov_split.sh); otherwise the same job runs plain pytest without --cov. It runs on a single GPU so the pilot is cheap to reproduce.

L1 package coverage (CPU): ready/merge Simple Test jobs run without --cov. Scheduled weekly on main with WEEKLY=1 runs Simple · CPU Coverage Test in .buildkite/cuda/test-weekly.yml (pytest -m 'core_model and cpu' + Cobertura XML artifact).

Naming convention: coverage-<model_id>-<mode>-<step_id>.xml.gz, where <model_id> is the model's directory name under vllm_omni/diffusion/models/<model_id>/ or vllm_omni/model_executor/models/<model_id>/ (e.g. z_image, qwen3_omni), <mode> is online or offline, and <step_id> is BUILDKITE_STEP_ID (local outside Buildkite). gunzip before feeding a report to a Cobertura consumer.

Reports are gzipped because --cov=vllm_omni sets coverage's source, which makes it walk the package and emit one <line> element per statement for every file — including the ones a single model's tests never import. That inventory, not the coverage, is the size: a Bagel report is 7.1 MB, of which 96.5% is <line> elements and 77% belongs to the 840 of 1169 files that run never imported. A run covering zero lines produces the same 7 MB. gzip takes it to roughly 410 KB and changes nothing about the data.

<step_id> is what keeps two steps that cover the same model apart. Artifacts are scoped to the build, and a weekly build can carry ready/merge (with --e2e) and weekly tiers at once, so several same-model steps land in one namespace — qwen3_tts already has both a Base and a CustomVoice job, and minicpmo_4_5 a base and a duplex one. Two uploads on one path do not fail loudly: an exact-name download can report the path as ambiguous, and a glob download fetches every match concurrently and renames each onto the same destination, so whichever finishes last wins. The step id remains the logical step identifier across retries, while Buildkite's default artifact lookup selects only the latest attempt — BUILDKITE_JOB_ID would instead give every attempt its own filename, leaving a failed attempt's partial report to be picked up alongside the real one. A step using parallelism/matrix would need BUILDKITE_PARALLEL_JOB added, since its jobs share a step id; neither pilot is one.

Opting in a new model: replace the job's combined pytest command with run_cov_split.sh, which runs each mode, names the reports, uploads them, and fails the step if either half or the upload failed:

commands:
  - |
    .buildkite/common/scripts/run_cov_split.sh \
      --model-id <model_id> \
      --offline <offline test path> \
      --online <online test path> \
      -- <the job's existing pytest flags, e.g. -m '...' --run-level '...'>

Everything after -- is forwarded verbatim to both runs with quoting preserved, so an expression like -m 'advanced_model and cuda' survives intact and a job can pass anything else it needs. --offline/--online are repeatable and only one is required, so a job with tests in a single mode runs just that half. Runtime is bounded solely by the step's timeout_in_minutes — raise it, since the split loads the model once per mode.

Before splitting, confirm each half actually collects at least one test under the job's forwarded filters — pytest exits with code 5 ("no tests collected") if a half is empty, which the script reports as a failure, red-ing a job that used to pass as one combined invocation.

Also check the tests' num_cards against the job's mirror_hardwares preset. _cuda_marks turns num_cards > 1 into skipif(device_count() < num_cards), so a test asking for more GPUs than the preset provides is silently skipped and its coverage artifact comes back empty. Splitting such a job produces a file that looks fine and measures nothing — "Diffusion · Z Image Test" is the current example (online tests want 4 cards, the job runs on l4_1), which is why it is not a coverage pilot.

Online-mode coverage depends on the [tool.coverage.run] patch/sigterm settings in pyproject.toml: online tests launch the server with subprocess.Popen and stop it with SIGTERM, and without those settings the XML reflects only the pytest parent process, not the server's code paths.

The upload depends on buildkite-agent being callable inside the container. CUDA (l4_*, h100_*) and NPU (*_npu_*) presets all use the kubernetes plugin, which provides the agent. l4_* jobs run on the EKS l4-k8s queue (agent-stack-k8s); they no longer use the docker plugin or mount-buildkite-agent. A new docker preset that runs a coverage job still needs mount-buildkite-agent: true.

List both run_cov_split.sh and pyproject.toml in every opted-in job's source_file_dependencies preset — both change what the job measures, so without them a change there is filtered out of normal PR builds and only surfaces in a later nightly. tests/buildkite/test_upload_pipeline.py covers the filter behavior with a synthetic job (it does not pin real merge labels). If no job-key prefix matches, editing the pipeline YAML being uploaded, or any path under the source_filter_fallback registry key, keeps every job in that upload (including coverage jobs) so the wiring can be validated before merge. When a job-key prefix already matches, bypass does not expand the selection. A change to a different pipeline YAML still does not schedule them. When checking a new model's artifacts, compare lines-covered between the online and offline XML rather than just confirming both files exist.

Validation checklist¶

Related documentation¶

• Test System Overview — L1–L5 scope, triggers, test pyramid
• Test Writing Guide — markers, directories, L1–L5 examples
• Test Execution Guide — running CI-aligned jobs locally
• Implementation: .buildkite/common/scripts/upload_pipeline.py, .buildkite/common/scripts/skip_ci.py