When runs-on Won't Fail Over: A Hybrid CI Runner Strategy

Our backend test suite is around 12,000 tests, and a lot of them spin up real infrastructure — databases, caches, an event store — in throwaway containers. That’s the kind of suite that tells you the truth about your code. It’s also the kind of suite that doesn’t fit on a small, shared CI runner.

We found that out the way everyone does: the build started dying. Not failing — dying. Mid-run, the test JVM would vanish, killed for using too much memory on a runner that didn’t have enough to give. The fix sounds obvious — “use a bigger runner” — but the route there turned out to be more interesting than picking a machine. It led us through a sharp edge in how GitHub Actions schedules work, and out the other side with a small routing pattern worth sharing.

This is a map of the terrain we crossed. None of it is specific to our product, or to us — if you run a heavy test suite on GitHub Actions, you’ll recognise the ground.

The wall we hit

A test that boots a database container needs headroom: the container, the JVM under test, the test framework’s own forks, all resident at once. Run enough of those in parallel and the memory ceiling on a small shared runner becomes a hard wall. The symptom is the operating system quietly killing the biggest process to save itself, and your build reporting a process that “exited unexpectedly” with no stack trace to chase.

The honest reading of that symptom is not “flaky test.” It’s “this work doesn’t fit here.” We needed more memory and more cores. The question was where to get them without making CI slower, more expensive, or more fragile than it already was.

The trap in runs-on

The instinct is to reach for a bigger machine you already own — a self-hosted runner — and fall back to a cloud runner when the self-hosted one is occupied. GitHub Actions makes that look easy. You write:

runs-on: [self-hosted, big]

and assume that if the self-hosted box is busy, the job goes elsewhere.

It doesn’t. runs-on is a label match, not a scheduler with failover. If the runner that matches your labels is busy, the job queues — and waits, however long that takes. There is no “…and if that’s not free, use the cloud” built in. List a label, and you’ve married that label. This is documented GitHub behaviour, but it’s the kind of thing you usually discover at 9am when a long job is sitting in a queue behind another long job.

So the real problem wasn’t “which runner.” It was: how do you choose the runner at the moment the job starts, based on what’s actually available right now?

The map: three tiers

Before solving that, it helped to name the terrain. We had three kinds of runner available, each good at something different:

• A self-hosted runner — effectively free to run, and warm: its caches and container images are already on local disk, so a build that lands here starts fast. The catch is capacity. One box, one job at a time.
• A cloud runner — fast, generously sized, and always available. It costs per-minute, but it never makes you wait in a queue. We use Blacksmith here; it’s one of several vendors in this space, and the pattern doesn’t care which you pick.
• The standard GitHub-hosted runner — the universal fallback. Smaller, but always there even if every account, credit, or integration above it has evaporated.

The shape we wanted: prefer the free warm box when it’s idle, reach for the fast cloud runner the instant it isn’t, and keep the standard runner as a cold last resort that can never leave us stranded. A clear preference order — and a guarantee that we always land somewhere.

The route through: pick the label at runtime

Since runs-on won’t choose for us, we choose for it. The key realisation is that runs-on can read its value from another job’s output — so we add a tiny job that runs first, decides which runner to use, and hands the answer down.

jobs:
  route:
    runs-on: ubuntu-latest
    outputs:
      runner: ${{ steps.pick.outputs.runner }}
    steps:
      - id: pick
        run: |
          # Default to the always-available cloud runner.
          runner='"fast-cloud-runner"'
          # If the self-hosted box is online AND idle right now, prefer it.
          if self_hosted_is_free; then
            runner='["self-hosted","big"]'
          fi
          echo "runner=$runner" >> "$GITHUB_OUTPUT"

  build:
    needs: route
    runs-on: ${{ fromJSON(needs.route.outputs.runner) }}
    steps:
      - run: ./run-the-heavy-suite

Two details make this work, and both are easy to miss:

Live availability, not configuration. self_hosted_is_free isn’t a static flag — it’s a question asked at runtime against the live list of runners: is the box online, and is it not currently busy? Routing on a stale config value would just recreate the queueing problem; routing on live state is what lets us dodge it.

fromJSON lets one output be either a string or a list. A cloud runner is named by a single label string ("fast-cloud-runner"); a self-hosted runner is matched by an array of labels (["self-hosted","big"]). runs-on accepts both forms — but a job output is always a string. So we emit JSON text — sometimes a quoted string, sometimes a bracketed array — and let fromJSON(...) parse it back into whichever shape runs-on needs. One channel, two types.

The last principle is the one that earns its keep: fail open. If anything in the routing step goes sideways — the API call errors, the check is inconclusive, a credential is missing — the default value is already set to the always-available cloud runner before any check runs. A routing layer should never be able to strand a build. The worst case is “we paid for a cloud runner we didn’t strictly need,” which is a cost, not an outage.

A caching detour worth taking

Moving the heavy job onto a cloud runner surfaced a second question: caching. A large dependency-and-build cache is expensive to move around. The default CI caching model — compress it, upload it to remote storage, download and extract it on the next run — spends real wall-clock time on a multi-gigabyte cache every single run, on both ends.

Many cloud-runner vendors offer a better primitive: a persistent disk that re-attaches across runs. Instead of shipping the cache back and forth, it simply stays put. The build mounts it and the cache is already there. For a big cache, re-attaching beats download-extract-reupload by a wide margin.

One footgun to flag, because we hit it: if you adopt persistent-disk caching, make sure your language or build tooling isn’t also configured to push that same directory back to the old remote-tarball cache. Leave both on and every run pays the slow cost on top of the fast one — you’ve added a cache, not replaced one. Pick one mechanism per directory.

Where we landed

The OOM-kills are gone, because the heavy suite now runs where it has room. The queue stalls are gone, because a busy self-hosted box no longer blocks anything — the next job routes straight to the cloud. And the bill stays modest, because the free warm runner still takes the work whenever it’s idle, which is most of the time. Three tiers, a clear preference order, and no single point that can leave a build waiting or stranded.

None of the pieces are exotic. The insight is just that runs-on is a match, not a scheduler — so if you want failover, you compute the label yourself, from live state, and you fail open. The three tiers happen to be ours; yours might be two, or four, or a different cloud entirely. The pattern is portable, which is the point: it’s built from standard GitHub Actions primitives and a vendor you can swap, so nothing here locks you to the map we happened to draw.

If your CI is queueing behind a busy box, or dying on a runner that’s too small, that’s the path we found through. Go chart your own.

Cover photo by pavel ondera on Unsplash.