> ## Documentation Index
> Fetch the complete documentation index at: https://docs.macstadium.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Orka Upgrade Service
> Run guided, observable Orka cluster upgrades from the orka3 CLI: how to install Upgrade Service, how the upgrade flow works, and how to check status and history.
Orka Upgrade Service upgrades an Orka cluster to a selected release through a durable, observable upgrade run started from the `orka3` CLI. It coordinates the control plane and every eligible Apple silicon node, records progress in Kubernetes, and contains failures so that a problem on one node does not silently spread across the fleet.
Orka 3.6.0 introduced an earlier version of the Upgrade Service that let MacStadium push cluster upgrades to your environment. This page describes an expanded version that lets you start, monitor, and review upgrades yourself.
## When to use it
Upgrade Service is for a planned, manually initiated Orka release upgrade, after MacStadium has published and made the target release available. It replaces manually sequencing service, agent, and node-component upgrades one at a time.
It is not a scheduler, a general node-repair tool, a background auto-update service, or a way to install unpublished builds. Scheduled and automatic upgrades aren't available yet; they're a possible future direction, not something you can rely on today.
It also does not upgrade the macOS version running on a node. "Node components" throughout this page means Orka's own node-local software (Orka Engine and Virtual Kubelet), not the guest operating system.
If Upgrade Service isn't yet installed or finalized in your environment, see [Installing the Upgrade Service](#installing-the-upgrade-service) below before continuing.
## What it does
Upgrade Service has two parts:
* An operator running on the control plane validates the requested release, plans the upgrade, updates in-cluster services, dispatches node work, and records status.
* An upgrade agent running on each eligible Mac node installs node-local components, restarts them, verifies their health, and rolls back the local change if verification fails.
Only one upgrade run executes at a time. Each run follows the same ordered workflow:
1. Validate the target release and the current cluster state.
2. Upgrade Upgrade Service itself, if required.
3. Upgrade the per-node upgrade agents.
4. Upgrade node components, using a canary node first and then a bounded rolling window across the rest of the fleet.
5. Upgrade core control-plane components.
6. Upgrade the remaining Orka services.
If a node upgrade fails, Upgrade Service stops releasing new node work, lets already-started work finish, and marks the run failed. The control plane stays on its previous version if the node stage doesn't complete. Every intended node ends the run as succeeded, failed, or skipped with a reason, so you can see exactly what happened rather than an opaque pass/fail.
Successful runs update the cluster's declared Orka version. Completed runs, including failed ones, are retained as history so you can review them later. Until a cluster completes its first successful v2 run, `orka3 upgrade versions` reports its current version as `unknown`.
## Installing the Upgrade Service
How you get Upgrade Service running depends on your deployment.
Ansible images for both on-prem and AWS-hosted Orka Anywhere are released.
**Orka in the MacStadium Cloud (MSDC).** Requires Orka 3.6. All 3.6+ MSDC clusters are fully configured with Upgrade Service v2, already installed and accessible, no action needed. Customers can use it today to upgrade to the current release (3.6.4).
**Orka Anywhere, on-prem.** Requires Orka 3.6.2+. Upgrade Service is already running on your cluster (as v1), but needs to finish migrating to v2. Before running the migration, confirm Upgrade Service has already completed its own v1-to-v2 self-upgrade — the migration script below does not perform that step itself. This happens on its own once v2 is available to your cluster; confirm with `orka3 upgrade service status`, which should show an operator version starting with `2.0.0-`. If it doesn't, contact MacStadium support before running the migration script.
The script requires `kubectl` (its current context pointing at the cluster, with admin permissions) and `jq` on the machine running it, plus outbound HTTPS access to distribution.macstadium.com; it exits early if any of these aren't available.
```shell theme={null}
curl -fsSLO https://distribution.macstadium.com/orka-releases/helpers/migrate-upgrade-service.sh
chmod +x migrate-upgrade-service.sh
./migrate-upgrade-service.sh
```
With no argument, this migrates to the latest released build. To target a specific released version instead, pass it as the first argument:
```shell theme={null}
./migrate-upgrade-service.sh
```
The script installs the v2 CRDs, applies the updated permissions, removes v1 leftovers, and waits for every node's upgrade agent to check back in, up to seven minutes. Seeing repeated "waiting for agent leases" messages during that window is expected, not a failure. If your cluster labels its Mac nodes in a way that doesn't expose a standard `arm64` architecture label, set the `ORKA_EXPECTED_AGENTS` environment variable to your actual node count before running the script, so it knows how many agents to wait for.
**Orka Anywhere, AWS-hosted.** Requires Orka 3.6+. Upgrade Service is not present by default and must be installed. Run it via CodeBuild the same way you run [ARM node tooling upgrades](/orka/orka-upgrades-and-release-notes/upgrading-orka-on-aws#over-ssh), using the 3.6.4 Ansible image (`ghcr.io/macstadium/orka-ansible-aws:3.6.4`):
```shell theme={null}
AWS_DEFAULT_REGION= ansible-playbook -i arm.ssh.aws_ec2.yml /ansible/deploy-upgrade-service.yml --private-key ~/.ssh/id_rsa
```
## Before you start an upgrade
Confirm Upgrade Service and every expected node agent are healthy:
```shell theme={null}
orka3 upgrade versions
orka3 upgrade service status
orka3 upgrade service agents
```
`upgrade service status` compares the number of eligible nodes against agents that are live and agents that have ever been seen. `upgrade service agents` lists each eligible node's agent version, status, and lease age — resolve any node reporting `STALE` or `MISSING` before starting a run. If your OrkaNode roster isn't in the default namespace, pass `-n ` to point these commands at it.
## Starting and monitoring a run
Starting a run requires administrator access; checking status, history, and versions only needs read access.
Start a run targeting a specific release:
```shell theme={null}
orka3 upgrade run --version
```
Or request the highest available release:
```shell theme={null}
orka3 upgrade run --version latest
```
`latest` resolves once when the run is created and stays fixed for that run, even if a newer release becomes available while it's in progress. Use an explicit version for production upgrades; reserve `latest` for environments where your change policy allows the release index to pick the target.
The command returns a timestamped run name. Use it to check on the run specifically, or check the most recent run without it:
```shell theme={null}
orka3 upgrade status
orka3 upgrade status
```
List retained runs, newest first:
```shell theme={null}
orka3 upgrade history
```
`upgrade run`, `upgrade status`, `upgrade history`, and `upgrade versions` all support `--output`/`-o`; use `json` for scripting.
## Troubleshooting
Upgrade Service is restart-safe and idempotent, but it isn't a transactional, whole-cluster operation:
* **A run stays `Pending` and then fails.** Upgrade Service allows up to 30 minutes for a run to be admitted. A message about exceeding that limit usually means the target couldn't be resolved or another run was still active; check `orka3 upgrade history`.
* **The requested version isn't found or is unavailable.** The target release may be missing from the release index or temporarily quarantined. Check the current and available releases with `orka3 upgrade versions`, then choose an available release or use `--version latest`.
* **The cluster is already on the requested version.** Check `orka3 upgrade versions` to confirm. Repeated run requests are safe (timestamped run names never collide), so requesting the same version again is not an error. If you're retrying a previously failed run, fix the cause of that failure and start a new run at the same target; failed runs can't be resumed, but already-converged components are handled idempotently, so retrying is safe.
* **A node did not upgrade.** Run `orka3 upgrade service status`, then `orka3 upgrade service agents`. An agent that hasn't renewed its lease in 90 seconds reports `STALE`; one that's never been seen reports `MISSING`. Recover the agent, then start a new run.
* **A run fails during the node stage.** This is expected behavior, not a bug: Upgrade Service stops releasing new node work, lets already-started work finish, and marks the run failed, leaving the control plane on its previous version. Fix the reported node or agent issue, then create a new run for the same target — already-converged components are handled idempotently, so retrying is safe.
* **A node was skipped, or a node deadline expired.** Check the node's skip reason in `orka3 upgrade status -o json` — common causes include no live agent, the node already being on the target version, the node being drained, or the node roster changing while the run was active.
* Completed work is not automatically undone. There is no full-cluster rollback.
* Don't delete a run to try to cancel it. Deletion removes the run's history while its work may still be active.
* Upgrade Service retains the 10 most recent completed runs. If you need older run details for long-term records, export them (`orka3 upgrade history` or a specific run's status) before they age out.
## Getting help
If a run doesn't complete as expected, [contact MacStadium support](mailto:support@macstadium.com) with the run name, target release, and the output of `orka3 upgrade status -o json` and `orka3 upgrade service agents -o json`.
## Related topics
- [Upgrading Orka on AWS](/orka/orka-upgrades-and-release-notes/upgrading-orka-on-aws.md)
- [Orka release notes](/orka/orka-upgrades-and-release-notes/orka-release-notes.md)
- [Kubernetes upgrade guide](/orka/orka-upgrades-and-release-notes/kubernetes-upgrade-guide.md)
- [How to upgrade your Orka cluster](/orka/orka-upgrades-and-release-notes/orka-upgrades.md)
- [Orka VM Tools for guest services on ARM VMs](/orka/orka-resources/vm-tools.md)