Set up the most-allocated scheduler on a self-managed control plane

Orka can place VMs two ways. The default scheduler spreads VMs to keep free and used resources balanced across nodes. The most-allocated scheduler fills the busiest node that still has room before it starts on the next one. See VM scheduling. On MacStadium-hosted clusters, the most-allocated scheduler is already set up. On a self-managed Orka Anywhere cluster, the Orka installer doesn’t change your Kubernetes control plane, so you add the scheduler profile yourself. These steps apply to Orka 3.6 and 4.0. On Orka 3.x, use orka3 in place of orka.

Orka accepts --scheduler most-allocated whether or not your control plane has the profile. Without the profile, a VM that asks for it is never scheduled: its pod gets no scheduling events, and orka vm deploy waits, then fails after 10 minutes with a generic timeout. Finish this setup on every control plane node before you use the scheduler in VM configs or pipelines.

Before you begin

A control plane that you manage with kubeadm, running Kubernetes 1.25 or later, where you can edit the kube-scheduler static pod manifest on each control plane node.
SSH access with sudo to every control plane node.
kubectl access to the cluster as a cluster admin.

Managed control planes, such as Amazon EKS, don’t let you change kube-scheduler settings. On those clusters, don’t use --scheduler most-allocated or set vm_scheduler_name to most-allocated-scheduler: VMs that ask for it stay Pending.

Add the scheduler profile

Only the kube-scheduler that holds the leader lease places pods, so every control plane node needs the profile. Change one node at a time, and confirm its kube-scheduler is Running (step 3) before you move to the next.

Create the scheduler configuration

Create /etc/kubernetes/kube-scheduler-config.yaml with the following content. It keeps the standard default-scheduler profile and adds a most-allocated-scheduler profile that scores nodes by how much of their resources are already allocated.

apiVersion: kubescheduler.config.k8s.io/v1
kind: KubeSchedulerConfiguration
clientConnection:
  kubeconfig: /etc/kubernetes/scheduler.conf
profiles:
  - schedulerName: default-scheduler
  - schedulerName: most-allocated-scheduler
    pluginConfig:
      - args:
          scoringStrategy:
            type: MostAllocated
        name: NodeResourcesFit

Keep the default-scheduler profile. Every other pod in your cluster uses it. If your kube-scheduler already runs with a --config file, add the most-allocated-scheduler profile to that file instead of replacing it.

Point kube-scheduler at the file

Edit /etc/kubernetes/manifests/kube-scheduler.yaml. If you keep a backup copy, store it outside /etc/kubernetes/manifests, or the kubelet starts it as a second scheduler. Add the --config flag to the container command, and mount the file into the pod:

spec:
  containers:
    - command:
        - kube-scheduler
        - --config=/etc/kubernetes/kube-scheduler-config.yaml
        # keep the existing flags
      volumeMounts:
        - name: schedulerconfig
          mountPath: /etc/kubernetes/kube-scheduler-config.yaml
          readOnly: true
        # keep the existing mounts
  volumes:
    - name: schedulerconfig
      hostPath:
        path: /etc/kubernetes/kube-scheduler-config.yaml
        type: File
    # keep the existing volumes

The kubelet restarts kube-scheduler when you save the manifest.

Check that kube-scheduler restarted

kubectl get pods -n kube-system -l component=kube-scheduler

The node’s kube-scheduler pod should be Running with a recent start time. If it’s in CrashLoopBackOff, check its logs with kubectl logs -n kube-system <POD_NAME>. A mistake in the configuration file is the usual cause.Repeat steps 1 to 3 on the next control plane node.

Test the scheduler

After every control plane node is done, deploy a VM that asks for the most-allocated scheduler. A short timeout makes a missing profile show up quickly:

orka vm deploy scheduler-test --image <IMAGE> --scheduler most-allocated --timeout 2

If the VM starts, the profile works. Delete it with orka vm delete scheduler-test. If the deploy times out, check the preceding steps on every control plane node.

Keep the setup through Kubernetes upgrades

kubeadm upgrade regenerates the static pod manifests from the cluster configuration in the kubeadm-config ConfigMap, which removes the --config flag and the mount. To keep them, add them to the scheduler section of that ConfigMap:

kubectl -n kube-system edit cm kubeadm-config

For kubeadm’s v1beta4 configuration (Kubernetes 1.31 and later):

scheduler:
  extraArgs:
    - name: config
      value: /etc/kubernetes/kube-scheduler-config.yaml
  extraVolumes:
    - name: schedulerconfig
      hostPath: /etc/kubernetes/kube-scheduler-config.yaml
      mountPath: /etc/kubernetes/kube-scheduler-config.yaml
      readOnly: true
      pathType: File

On v1beta3, extraArgs is a map instead, for example config: /etc/kubernetes/kube-scheduler-config.yaml. After a Kubernetes upgrade, repeat the check in step 3.

Use the scheduler

After setup, choose the scheduler per VM config or per VM with --scheduler most-allocated. See orka vm-config create. You can also make it the cluster default, for every VM that doesn’t name a scheduler. Talk to MacStadium before you do, because a mistake here affects every VM:

Do it only after the setup succeeds on every control plane node. If the profile is missing or the name is misspelled, every VM that doesn’t name a scheduler stays Pending.
Pass -e vm_scheduler_name=most-allocated-scheduler to the Orka installer. Setting it in cluster.yml doesn’t work, because the installer’s own settings win. See Override installer settings.
Pass it on every installer run, including upgrades. A run without it sets the default back to default-scheduler.

MacStadium

Orka

MacStadium VDI

IaaS

Set up the most-allocated scheduler on a self-managed control plane

Before you begin

Add the scheduler profile

Keep the setup through Kubernetes upgrades

Use the scheduler

​Before you begin

​Add the scheduler profile

​Keep the setup through Kubernetes upgrades

​Use the scheduler

​Related

Before you begin

Add the scheduler profile

Keep the setup through Kubernetes upgrades

Use the scheduler

Related