> ## Documentation Index
> Fetch the complete documentation index at: https://docs.macstadium.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Consuming Metrics from Prometheus

> Scrape Prometheus metrics from your Orka cluster: server, operator, and node exporter endpoints. Covers Prometheus config, the web console, and Grafana.

How to aggregate data from Prometheus node exporters and access the Prometheus web console

It is possible to collect Prometheus metrics directly from your Orka cluster, including host metrics for each node in the cluster as well as metrics for the Orka API server and operator.

For firewall and network-level metrics (throughput, CPU, sessions, IPsec tunnels), see [Consuming Firewall and Network Metrics](https://docs.macstadium.com/orka/orka-resources/consuming-firewall-metrics.md) instead. That's a separate collection pipeline from what's covered here.

> [!NOTE]
>
> Prometheus metrics support has been available since Orka 3.0. Starting with Orka 3.5, clusters using Harbor OCI storage also have Prometheus metrics available for the Harbor registry. Contact support to confirm your Harbor Prometheus endpoint if you're on Orka 3.5 or later.

## Integrating Orka metrics with an existing Prometheus instance

If your organization is already using Prometheus and you wish to collect additional data from your Orka cluster, follow the instructions in this section.

### Orka Server and Operator Metrics

#### A Note About the Orka Server

Starting with Orka 3.0, the API server is no longer primarily responsible for managing Orka VMs. Instead, the operator is primarily responsible for managing the lifecycle of Orka VMs and other resources. Clients such as the orka CLI speak directly to the Kubernetes API instead of the Orka API.

As a consequence, metrics for the Orka API Server are generally less relevant, unless you are using workflows and integrations that still rely on the legacy API server.

Metrics for the Orka server and operator are available from the Orka load balancer address. For most environments this is typically 10.221.188.20 or 10.221.188.100, but check the IP Allocation table in [My Cloud](https://docs.macstadium.com/macstadium/macstadium-overview/ip-plan.md) if you aren't sure.

* **Orka server:** `/api/v1/metrics` on the load balancer address. The server exposes only Go runtime and process metrics, such as `go_goroutines` and `process_resident_memory_bytes`.
* **Orka operator:** `/metrics` on port 8080. VM deployment metrics, such as deployment counts, failures, and durations, come from the operator.

To verify you are able to access the Orka server and operator metrics, try cURLing the endpoints:

**curl**

```
curl -s -o /dev/null -w '%{http_code}\n' 10.221.188.20/api/v1/metrics
curl -I 10.221.188.20:8080/metrics
```

You should receive a response of 200 OK if all is well.

> [!NOTE]
>
> The Orka server check uses a GET request rather than `curl -I`. The server only answers the HTTP methods its routes register, so a HEAD request to `/api/v1/metrics` returns 404 even when the endpoint is working and Prometheus can scrape it. The operator's endpoint answers HEAD, so `-I` is fine there.

> [!TIP]
>
> To view descriptions and additional information on the available Orka server metrics, try `curl 10.221.188.20/api/v1/metrics`. For Orka operator metrics, try `curl 10.221.188.20:8080/metrics`.

In order to scrape the Orka server and operator metrics, add jobs to your Prometheus configuration:

**prometheus.yml**

```
scrape_configs:
  # ...

  - job_name: 'orka/orka-server'
    scrape_interval: 30s
    scrape_timeout: 30s
    metrics_path: /api/v1/metrics
    scheme: http
    static_configs:
      - targets: ['10.221.188.20']
        labels:
          service: 'orka-server'

  - job_name: 'orka/orka-operator'
    scrape_interval: 30s
    scrape_timeout: 30s
    metrics_path: /metrics
    scheme: http
    static_configs:
      - targets: ['10.221.188.20:8080']
        labels:
          service: 'orka-operator'
```

### Prometheus node exporters

Every node in the Orka cluster runs a Prometheus Node Exporter on port 9100. This includes both Intel and ARM nodes. To ensure you are able to scrape the node exporter data, try cURLing the metrics endpoint from any node:

**curl**

```
curl -I http://10.221.188.31:9100/metrics
```

If the node exporter is running and reachable from outside the cluster, you should receive a response of 200 OK.

> [!NOTE]
>
> **Outside access:** If your Prometheus instance is hosted outside of the cluster, make sure that TCP traffic is permitted to port 9100 for all hosts on the Orka private network. For more information, see [My Cloud](https://docs.macstadium.com/macstadium/macstadium-overview/ip-plan.md).

In order to scrape the node exporter data, add a job to your Prometheus configuration. For example:

**prometheus.yml**

```
scrape_configs:
  # ...

  - job_name: 'orka/node-exporter'
    scrape_interval: 5s
    static_configs:
      - targets: ['10.221.188.5:9100', '10.221.188.6:9100', '10.221.188.7:9100']
        labels:
          group: 'master'
      # Intel nodes. Orka 4.0 is Apple silicon only, so this group doesn't apply there.
      - targets: ['10.221.188.31:9100', '10.221.188.32:9100', '10.221.188.33:9100']
        labels:
          group: 'x86'
      - targets: ['10.221.188.34:9100']
        labels:
          group: 'arm'
```

> [!TIP]
>
> As an alternative to the static config, you can use [file-based service discovery](https://prometheus.io/docs/guides/file-sd/) to scrape the targets.

## GitHub Actions integration metrics

The [Orka GitHub Actions integration](https://github.com/macstadium/orka-github-actions-integration) can optionally expose Prometheus metrics for runner scale set statistics. This is opt-in and disabled by default.

### Enable metrics

Set the following environment variables when running the integration:

| Variable | Default | Description |
| - | - | - |
| `ENABLE_METRICS` | `false` | Set to `true` to expose a `/metrics` endpoint. |
| `METRICS_ADDR` | `:8080` | Address where the metrics endpoint listens. |
| `METRICS_POLL_INTERVAL` | `30s` | How often runner scale set statistics are polled (e.g., `30s`, `1m`). |

### Available metrics

All metrics carry a `runner_name` label corresponding to the runner scale set name.

| Metric | Description |
| - | - |
| `runner_scale_set_total_available_jobs` | Jobs available to be assigned to a runner |
| `runner_scale_set_total_acquired_jobs` | Jobs acquired by a runner but not yet assigned |
| `runner_scale_set_total_assigned_jobs` | Jobs assigned to a specific runner |
| `runner_scale_set_total_running_jobs` | Jobs currently executing |
| `runner_scale_set_total_registered_runners` | Runners registered with GitHub |
| `runner_scale_set_total_busy_runners` | Runners currently processing a job |
| `runner_scale_set_total_idle_runners` | Runners registered but not processing a job |

### Scrape configuration

Add a job to your `prometheus.yml` to scrape the integration:

```yaml theme={null}
scrape_configs:
  # ...

  - job_name: 'orka/gha-runner'
    scrape_interval: 30s
    metrics_path: /metrics
    scheme: http
    static_configs:
    - targets: ['<GHA_RUNNER_HOST>:8080']
      labels:
        service: 'orka-gha-runner'
```

Replace `<GHA_RUNNER_HOST>` with the hostname or IP of the machine running the GitHub Actions integration. The port should match `METRICS_ADDR` (default `8080`).


## Related topics

- [Consuming Firewall and Network Metrics](https://docs.macstadium.com/orka/orka-resources/consuming-firewall-metrics.md)
- [Shared VM storage for Orka CI artifacts](https://docs.macstadium.com/orka/orka-resources/shared-vm-storage.md)
- [Orka 3.x release notes](https://docs.macstadium.com/orka/orka-upgrades-and-release-notes/orka-3-release-notes.md)
- [Orka release notes](https://docs.macstadium.com/orka/orka-upgrades-and-release-notes/orka-release-notes.md)
- [Orka on AWS architecture and deployment overview](https://docs.macstadium.com/orka/orka-on-aws-and-on-prem/orka-on-aws-getting-started.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.