Modelplane Modelplane docs
★ Star the Project
Version
main v0.5 Latest v0.4 v0.3 v0.2

ModelDeployment Custom Resource

On this page
This document is for an unreleased version of Modelplane.

This document applies to the Modelplane main branch and not to the latest release v0.5.

Deploy a model to the fleet, from one pod to disaggregated prefill and decode.

Concept guide: Deploy a Model →

#Metadata

API version
modelplane.ai/v1alpha1
Kind
ModelDeployment
Scope
Namespaced
Short names
md

#Example

Manifest
apiVersion: modelplane.ai/v1alpha1
kind: ModelDeployment
metadata:
  name: qwen3-8b
  namespace: ml-team
spec:
  replicas: 2
  template:
    spec:
      engines:
      - name: qwen3-8b
        members:
        - role: Standalone
          nodeSelector:
            devices:
            - name: gpu
              count: 1
              selectors:
              - cel: |
                  device.capacity["gpu.nvidia.com"].memory.compareTo(quantity("20Gi")) >= 0
          template:
            spec:
              containers:
              - name: engine
                image: vllm/vllm-openai:v0.23.0
                args:
                - --model=Qwen/Qwen3-8B
                - --served-model-name=$(MODELPLANE_SERVED_MODEL_NAME)

#Spec

# replicas required integer ≤ 100

How many ModelReplicas to fan out to. Each replica is a complete serving instance scheduled to one InferenceCluster. 0 parks the deployment: its replicas and endpoints are removed but the object and its spec remain, and conditions report NoReplicasDesired.

# template required object

Template for the ModelReplicas this deployment fans out to. Everything but the replica count lives here, mirroring a Kubernetes Deployment’s spec.template.

# metadata optional object
# labels optional map[string]string

Labels to set on the ModelReplicas and ModelEndpoints this deployment composes, alongside the labels Modelplane manages. Use them to organize and select your own resources (kubectl get modelreplica -l tier=prod) or to route to a subset of endpoints from a ModelService. Keys under the modelplane.ai/ prefix are reserved.

# spec required object
# clusterSelector optional object

Optional label selector to filter InferenceClusters. If omitted, all ready clusters are candidates.

# matchLabels optional map[string]string
# engines required object[] 1–8 items
# copies optional integer 1–64 default: 1

How many identical copies of this engine to run per ModelReplica. A fixed number, sized once per deployment; scaling happens by adding ModelReplicas (spec.replicas), never by varying copies. Maps to the composed Deployment’s or LeaderWorkerSet’s replica count, or a Grove PodCliqueSet’s podCliqueScalingGroups[].replicas (its spec.replicas stays 1). Defaults to 1.

# members required EngineMember[] → 1–2 items
# name required string 1–63 chars

Identifies the engine within the deployment. Becomes part of the composed workload names, so it must be a DNS label.

# phase optional enum: Prefill | Decode

The engine’s phase in a PrefillDecode deployment, Prefill or Decode. Set only when serving.mode is PrefillDecode, where exactly one engine takes each phase.

# modelCacheRef optional object

Reference to a ModelCache in the same namespace. Optional for single-node engines; required for any engine that spans multiple nodes (a Leader with one or more Workers), since every pod in the gang mounts it.

# name required string ≥ 1 chars

ModelCache resource name in the same namespace.

# serving optional object

How the deployment is served from the cluster edge to its engines. Modelplane fronts the serving pods with an InferencePool and an endpoint picker in both modes. Unified (the default) serves prefill and decode on each engine. PrefillDecode serves them from the two engines marking those phases, with routing that sequences prefill then decode. Omitted means Unified.

# endpointPicker optional object

The endpoint picker Modelplane runs in front of the serving pods. Required exactly when router is EndpointPicker. Modelplane composes the picker’s InferencePool, Service and RBAC around it.

# config optional object

The picker’s configuration, such as an llm-d EndpointPickerConfig. Modelplane mounts it at $(MODELPLANE_PICKER_CONFIG) without reading it. Omitted for a picker that takes no configuration.

# template required object

Pod template for the picker. A curated subset of PodTemplateSpec, like a member’s. The user writes the picker’s command line, since each picker spells it differently.

# spec required object
# containers required object[] 1–1 items
# args optional string[]
# command optional string[]
# env optional object[]
# name required string
# value optional string
# valueFrom optional object
# configMapKeyRef optional object
# key required string
# name required string
# optional optional boolean
# fieldRef optional object
# apiVersion optional string
# fieldPath required string
# secretKeyRef optional object
# key required string
# name required string
# optional optional boolean
# image required string ≥ 1 chars
# name required string 1–63 chars
# resources optional object
# limits optional map[string]object
# requests optional map[string]object
# imagePullSecrets optional object[]
# name required string
# mode required enum: Unified | PrefillDecode default: Unified

Unified serves prefill and decode on one engine. PrefillDecode splits them across two engines, each marking its phase as Prefill or Decode.

# router optional enum: EndpointPicker

The router the user writes. Required under PrefillDecode. Omitted, under Unified, Modelplane routes the deployment with its default picker configuration. EndpointPicker runs the endpoint picker set in endpointPicker, and the user writes any sidecar the picker needs as a container in the member templates.

# tolerations optional object[]
# effect optional enum: NoSchedule | NoExecute

Taint effect to match. Empty matches all effects.

# key optional string

Taint key to match. Empty with operator Exists tolerates every taint.

# operator optional enum: Exists | Equal default: Equal

Exists matches any taint with the key (ignoring value); Equal matches key and value.

# value optional string

#Status

# replicas optional object
# ready optional integer
# total optional integer

#Subresources

# scale subresource

A scale subresource, so kubectl scale and replica-based autoscalers can drive it.

# specReplicasPath .spec.replicas

JSONPath to the desired replica count in spec.

# statusReplicasPath .status.replicas.total

JSONPath to the observed replica count in status.

#EngineMember object

One member of an engine’s gang — a role (Standalone, Leader, or Worker) with its hardware and pod template.

# nodeSelector optional NodeSelector →

The per-node device request for this member’s pods: what devices each pod needs from its node. The scheduler matches it against a candidate pool’s InferenceClass devices (surfaced on InferenceCluster status.gpuPools) and places the member on a pool that satisfies it, preferring one pool for the whole engine and splitting members across pools only when no pool satisfies them all. claim: DRA requests also become DeviceRequests in the ResourceClaim the member’s pods bind GPUs through. A GPU request’s count is the GPUs per node. Omitted, the member claims no devices and schedules onto its engine’s pool - a coordinator-only leader. At least one member per engine must carry a nodeSelector, and at least one member’s requests must resolve to a claimable (claim: DRA) device; an engine that matches only synthetic devices leaves its pods nothing to claim, so the scheduler treats such a pool as ineligible and the deployment reports InsufficientCapacity.

# role optional enum: Standalone | Leader | Worker default: Standalone

The member’s role in the engine. Standalone is a lone pod; a Leader coordinates and serves while its Workers join it. Defaults to Standalone.

# template required PodTemplate →

Pod template for this member’s engine pods. A curated subset of PodTemplateSpec.

# worker optional object

Settings for a Worker member. Valid only on a member whose role is Worker.

# nodes required integer 1–63

How many nodes this member spans - how big the engine is. Each node runs one worker pod, so the engine’s gang spans 1 (the Leader) plus this many worker pods. Defaults to 1.

#NodeSelector object

The per-node device request for this member’s pods: what devices each pod needs from its node. The scheduler matches it against a candidate pool’s InferenceClass devices (surfaced on InferenceCluster status.gpuPools) and places the member on a pool that satisfies it, preferring one pool for the whole engine and splitting members across pools only when no pool satisfies them all. claim: DRA requests also become DeviceRequests in the ResourceClaim the member’s pods bind GPUs through. A GPU request’s count is the GPUs per node. Omitted, the member claims no devices and schedules onto its engine’s pool - a coordinator-only leader. At least one member per engine must carry a nodeSelector, and at least one member’s requests must resolve to a claimable (claim: DRA) device; an engine that matches only synthetic devices leaves its pods nothing to claim, so the scheduler treats such a pool as ineligible and the deployment reports InsufficientCapacity.

# devices required object[] 1–16 items
# count optional integer 1–64 default: 1

How many matching devices a node must have. For a GPU request this is the per-node GPU count.

# name required string 1–63 chars

Name of this request. Mirrors a DRA DeviceRequest name; carried through to the ResourceClaim.

# selectors required object[] 1–8 items
# cel optional string 1–10240 chars

A DRA CEL expression evaluated against one device. Reads device.driver, device.attributes[""]. (typed), and device.capacity[""]. (a Quantity), with quantity() and semver() helpers, e.g. device.capacity[“gpu.nvidia.com”].memory.compareTo(quantity(“141Gi”)) >= 0.

#PodTemplate object

Pod template for this member’s engine pods. A curated subset of PodTemplateSpec.

# metadata optional object

Metadata applied to the member’s pods. Useful for labels and annotations that control cluster-level features like service mesh injection. Label keys under the modelplane.ai/ prefix are reserved.

# annotations optional map[string]string
# labels optional map[string]string
# spec optional object

Pod spec for this member’s engine pods.

# containers required object[] 1–8 items
# args optional string[]
# command optional string[]
# env optional object[]
# name required string
# value optional string
# valueFrom optional object
# configMapKeyRef optional object
# key required string
# name required string
# optional optional boolean
# fieldRef optional object

Reference a pod field via the downward API, e.g. status.podIP, metadata.name, or metadata.namespace.

# apiVersion optional string
# fieldPath required string
# secretKeyRef optional object
# key required string
# name required string
# optional optional boolean
# image required string ≥ 1 chars

Container image.

# name required string 1–63 chars

Container name. The container named “engine” is the inference engine.

pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?$

# ports optional object[] ≤ 8 items
# containerPort required integer 1–65535

Port number the container listens on.

# name optional string 1–15 chars

Port name. The engine container’s port named “http” is where Modelplane probes the engine and scrapes its metrics.

pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?$

# readinessProbe optional object

How Kubernetes checks that a container beside the engine is ready, such as a sidecar on port 8000. The engine container can’t set one: Modelplane probes the engine on its http port.

# failureThreshold optional integer ≥ 1
# grpc optional object
# port required integer
# service optional string
# httpGet optional object
# path optional string
# port required any of: integer | string
# initialDelaySeconds optional integer
# periodSeconds optional integer ≥ 1
# successThreshold optional integer ≥ 1
# tcpSocket optional object
# port required any of: integer | string
# timeoutSeconds optional integer ≥ 1
# resources optional object

CPU and memory requests and limits. On the engine container, Modelplane adds the member’s device claim beside them.

# limits optional map[string]object
# requests optional map[string]object
# imagePullSecrets optional object[]
# name required string