Wise Hustlers — Digital Product & App Development Studio Logo
Get Consultation
By Wise Hustler Admin•11/17/2026•24 min read

GitHub Actions vs RunsOn vs Depot vs Ubicloud: CI/CD Runner Performance & FinOps Benchmarks

GitHub Actions vs RunsOn vs Depot vs Ubicloud: CI/CD Runner Performance & FinOps Benchmarks

# GitHub Actions vs RunsOn vs Depot vs Ubicloud: CI/CD Runner Performance & FinOps Benchmarks

Summary: Modern engineering organizations face ballooning CI/CD compute bills and agonizing pipeline latencies driven by stock GitHub-hosted runner constraints. In this comprehensive benchmark, we evaluate GitHub Actions vs RunsOn vs Depot vs Ubicloud across 2026 compute pricing models, Docker BuildKit cache performance on a 15GB monorepo image, AWS Spot ephemeral auto-scaling architectures, and fail-safe preemption handling to deliver actionable FinOps strategies for high-velocity platform engineering teams.

---

CI/CD Execution Economics: The Hidden Tax of Hosted Runners

Continuous Integration pipelines represent one of the fastest-growing operational line items in modern cloud infrastructure budgets. Engineering leaders routinely observe monthly continuous integration bills escalating at non-linear rates relative to headcount growth. The root cause rarely stems from excessive test volume; rather, it originates in the compute architecture, network topology, and billing paradigms of standard cloud-hosted runners.

Standard GitHub-hosted runners rely on multi-tenant Azure virtual machines provisioned on demand. While operationally frictionless, this convenience imposes structural inefficiencies that penalize growing engineering teams:

1. Per-Minute Billing and Upward Rounding: GitHub-hosted runners bill in whole minute increments. A build executing in 61 seconds incurs a 120-second charge, creating an immediate billing overhead on fast-feedback test suites.

2. Restrictive Cache Throughput and Storage Caps: GitHub Actions Cache (actions/cache) imposes a strict 10GB storage limit per repository. Furthermore, cache archives transit over public HTTPS endpoints, subject to standard cloud network latency, egress limits, and client-side tar compression and decompression overhead. On multi-gigabyte container images, the time spent decompressing and transferring tarballs frequently negates the latency savings of caching altogether.

3. Compute Oversubscription: Stock 2-core and 4-core GitHub runners share hyperthreads on virtualized cloud hardware without dedicated CPU pinning or local NVMe storage, producing variable build execution times depending on neighbor tenant activity.

4. Network Egress Penalties: Running integration tests that pull intermediate container layers or upload test artifacts across cloud boundaries introduces steady data transfer charges that accumulate silently.

To combat these bottlenecks, platform engineers evaluate three alternative paradigms:

  • RunsOn: A commercial source-available appliance licensed annually that orchestrates ephemeral GitHub Actions runners directly inside your private AWS account. It provisions native Amazon EC2 Spot instances within your isolated VPC, utilizing wholesale AWS compute pricing without markup.
  • Depot: A managed acceleration platform focused on distributed remote Docker BuildKit engines and high-performance native CI runners. Depot features persistent NVMe-backed distributed SSD caches and native multi-architecture execution (AMD64 and ARM64) without QEMU emulation overhead.
  • Ubicloud: An open-source, portable cloud provider operating on bare-metal infrastructure (such as Hetzner and Leaseweb). Ubicloud offers drop-in GitHub Actions runner compatibility with zero code changes at commodity hardware price floors.

Evaluating github actions vs runson vs depot alongside Ubicloud requires analyzing compute unit costs, cache synchronization mechanics, and infrastructure control.

---

2026 Compute Cost Comparison Matrix & FinOps Modeling

Achieving effective ci cd runner cost optimization 2026 mandates a granular understanding of billing precision and underlying compute costs. The table below details the compute pricing tiers across standard Linux x86 and ARM64 architectures for GitHub-hosted runners, Depot, Ubicloud, and RunsOn.

Compute Pricing & Feature Matrix (2026 Benchmarks)

PlatformArchitectural Model2-Core x86 ($/min)4-Core x86 ($/min)8-Core ARM64 ($/min)Billing PrecisionCache Storage ModelPrivate VPC Support
GitHub-HostedMulti-tenant Azure VM$0.0060$0.0120$0.0140Per-minute (round-up)10GB S3/Blob (actions/cache)Enterprise only (VNet injection)
UbicloudBare-metal KVM (Hetzner/IaaS)$0.00125$0.0025$0.0050Per-secondLocal NVMe + persistent volumeWireGuard / Private Subnet
DepotManaged BuildKit / MicroVM$0.0060$0.0120$0.0240Per-secondDistributed NVMe (Ceph/NVMe-oF)AWS Direct Connect / Peering
RunsOnAWS EC2 Spot in Customer VPC~$0.0012*~$0.0024*~$0.0031*Per-second (EC2 Spot)Ephemeral NVMe / Local EBS gp3Native (Customer AWS Account)

\RunsOn compute rates reflect average AWS us-east-1 Spot market pricing for c6i.large ($0.017/hr = ~$0.00028/min raw EC2, plus EBS gp3 storage and NAT Gateway egress amortization, totaling ~$0.0012/min). RunsOn software is licensed annually based on monthly job volume tiers (starting at €300/year for Starter up to €3,600/year for Enterprise, with no per-seat or per-developer fees), avoiding variable compute markups.*

Monthly TCO Projections: 50,000 vs. 200,000 Monthly Minutes

To demonstrate real-world FinOps implications, consider an engineering organization operating 25 developers generating 50,000 runner minutes per month, scaling to an enterprise tier of 200,000 runner minutes per month (standard 4-core x86 workloads).

flowchart LR
    subgraph Workload["Monthly CI Workload (4-Core x86)"]
        W1["50,000 Build Minutes"]
        W2["200,000 Build Minutes"]
    end

    subgraph Costs50k["50k Minutes / Month TCO"]
        GH1["GitHub: $600 (base compute)"]
        DEP1["Depot: $600 + Cache ($40) = $640"]
        UBI1["Ubicloud: $125"]
        RO1["RunsOn: ~$120 AWS Spot + ~$75 License = ~$195"]
    end

    subgraph Costs200k["200k Minutes / Month TCO"]
        GH2["GitHub: $2,400 (base compute)"]
        DEP2["Depot: $2,400 + Cache ($60) = $2,460"]
        UBI2["Ubicloud: $500"]
        RO2["RunsOn: ~$480 AWS Spot + ~$150 License = ~$630"]
    end

    W1 --> GH1
    W1 --> DEP1
    W1 --> UBI1
    W1 --> RO1

    W2 --> GH2
    W2 --> DEP2
    W2 --> UBI2
    W2 --> RO2

1. GitHub-Hosted: At 50,000 minutes of 4-core compute ($0.012/min), baseline compute expenditure equals $600/month. At 200,000 minutes, compute expands to $2,400/month. Furthermore, because GitHub bills per whole minute, jobs that complete shortly after an integer minute boundary (such as a 61-second test suite billed as 120 seconds) incur ceiling rounding overhead, increasing effective spend on high-frequency, short-duration CI jobs.

2. Depot: At $0.012/min for 4-core compute ($0.003/vCPU-minute) billed per second, 50,000 minutes yields $600/month in compute. At 200,000 minutes, compute costs $2,400/month. Factoring in distributed persistent cache storage ($0.20/GB/month for additional cache storage), total monthly expenditure equals approximately $640/month at 50k minutes and $2,460/month at 200k minutes. Additionally, because Depot maintains persistent SSD BuildKit caches across runs, teams frequently eliminate redundant dependency downloads and layer recompilations, reducing the total billable minutes required to complete Docker pipelines.

3. Ubicloud: Under ubicloud github actions pricing ($0.0025/min for 4-core), 50,000 minutes equals $125/month. At 200,000 minutes, total expenditure is $500/month—a substantial reduction compared to standard hosted alternatives with zero pipeline configuration changes.

4. RunsOn: At wholesale AWS Spot rates (~$0.0024/min fully loaded) combined with an amortized annual RunsOn license (€900/year for Growth, or ~$75/month), 50,000 minutes costs approximately $195/month. At 200,000 minutes, the spot compute total (~$480) plus an amortized Scale license (€1,800/year, or ~$150/month) totals approximately $630/month. Organizations with active AWS Enterprise Discount Program (EDP) agreements or upfront cloud credits can offset runner compute spend directly against their AWS infrastructure commitments, lowering net cash outlay.

---

Docker BuildKit Cache Architecture: Monorepo Workload Profile

Container build efficiency is the primary determinant of developer feedback loop velocity. To evaluate docker buildkit cache optimization in multi-stage builds, consider a representative 15.4GB polyglot production monorepo image containing:

  • A Next.js 15 enterprise frontend with dynamic server-side rendering bundles.
  • A Go 1.23 distributed backend service with vendored dependencies and CGO bindings.
  • A Python 3.12 machine learning inference pipeline with PyTorch and CUDA base layers.
  • A PostgreSQL client migration layer with Node.js automation scripts.

When compiling large monorepos, platforms exhibit starkly different performance characteristics between cold builds (clean environment, zero local cache) and warm builds (incremental modifications to application code with cached dependency layers).

Docker BuildKit Architectural Performance Matrix

Runner PlatformCompute & Storage ArchitectureCold Build MechanicsWarm Build MechanicsCache Synchronization ModelThroughput Characteristics
GitHub Actions (`gha`)4-Core x86 (16GB RAM) + Azure Blob StorageDownloads base layers sequentially from container registry; compiles un-cached stagesRe-downloads tar archives over HTTPS; uncompresses cache to local diskRemote archive export/import via GitHub REST APIConstrained by network round-trips and client CPU tar extraction
Ubicloud Standard4-Core x86 (16GB RAM) + Local Bare-Metal NVMeFetches layers directly over high-bandwidth datacenter uplinksReuses local layers when mapped to persistent storage or co-located nodesHost-local disk storage and container registry cacheHigh local NVMe read/write speeds over PCIe bus
Depot Managed4-Core x86 + Remote NVMe-oF Distributed CacheStreams parallel layer builds directly to high-throughput SSD clusterZero-copy layer hash verification; skips unchanged build stepsPersistent distributed NVMe-oF storage poolDirect block-level sync bypassing public network tar serialization
RunsOn AWS Spotc6id.xlarge (Dedicated Local NVMe Instance Store)Direct line-rate pull from Amazon ECR via AWS Gateway VPC EndpointsReads cached Docker layer blocks directly from local NVMe scratch volumeLocal NVMe instance store formatted as Docker rootLine-rate PCIe transfers with zero external network egress

Technical Analysis of Cache Mechanics

The divergent performance analyzed across depot buildkit cache benchmarks and native runners highlights the fundamental architectural differences in how BuildKit backends transfer intermediate layers:

flowchart TD
    subgraph GHA_Model["GitHub Actions Native (type=gha)"]
        GHA_RUNNER["Ephemeral Runner\n(Local Ext4)"]
        GHA_TAR["Tarball Compression Engine\n(CPU-Bound Serialization)"]
        GHA_REST["GitHub API / Blob Storage\n(HTTPS Network Transit)"]
        GHA_RUNNER -->|Export Layers| GHA_TAR
        GHA_TAR -->|HTTPS Stream| GHA_REST
    end

    subgraph Depot_Model["Depot Distributed Architecture"]
        DEP_RUNNER["Depot Runner Client\n(Thin CLI Daemon)"]
        DEP_BLOCK["Distributed NVMe-oF Storage\n(Ceph Block Cluster)"]
        DEP_BUILDKIT["Remote Dedicated BuildKit Daemon\n(Persistent Shared Cache)"]
        DEP_RUNNER <-->|mTLS Layer Sync| DEP_BUILDKIT
        DEP_BUILDKIT <-->|Zero-Copy Block Sync| DEP_BLOCK
    end

    subgraph RunsOn_Model["RunsOn Ephemeral AWS Spot (type=local)"]
        RO_INSTANCE["EC2 c6id.xlarge Node\n(Spot Instance)"]
        RO_NVME["Local NVMe Instance Store\n(/mnt/scratch overlay2)"]
        RO_EBS["EBS gp3 Root Volume\n(Pre-baked Custom AMI)"]
        RO_INSTANCE <-->|Line-rate PCIe Bus| RO_NVME
        RO_INSTANCE <-.->|Init Mount| RO_EBS
    end
  • The GitHub Actions Bottleneck: Under the native type=gha cache exporter, BuildKit serializes filesystem layers into compressed tarballs, transmits them over public HTTPS endpoints, and uncompresses them on subsequent runs. For multi-gigabyte container images with dozens of layers, compressing, uploading, downloading, and unpacking tar archives creates significant CPU and network latency, even when most layers remain unchanged. Furthermore, when total repository cache exceeds GitHub's 10GB limit, older archives are evicted under a least-recently-used (LRU) policy, triggering full cold rebuilds.
  • Depot Distributed Cache: Depot bypasses network tar serialization entirely. The builder client communicates with a dedicated remote BuildKit daemon backed by a distributed NVMe storage cluster (utilizing Ceph / NVMe-oF). Intermediate layer blocks remain uncompressed and instantly accessible. When an incremental build is triggered, BuildKit performs fast block-level hash checks and skips unchanged build stages, delivering rapid warm-build completions without transferring gigabytes of tar archives over the internet.
  • RunsOn Local NVMe Acceleration: RunsOn provisions AWS EC2 instances equipped with direct NVMe instance store hardware (such as the c6id or c7gd instance families). By formatting the local NVMe drive directly as the Docker daemon root (/mnt/scratch/docker), all layer reads and writes execute directly over the physical PCIe bus. Combined with Amazon ECR Interface VPC Endpoints, images transfer at internal AWS network speeds without traversing the public internet.

---

Ephemeral Runner Architecture in an Isolated AWS VPC

Deploying ephemeral ci cd runners aws environments ensures that each compute instance processes exactly one GitHub workflow job before terminating immediately. Running self hosted github runner spot instances securely requires an architectural design that isolates compute instances inside a private subnet while maintaining automated elasticity.

The following architecture demonstrates an enterprise-grade RunsOn deployment utilizing AWS CloudFormation or Terraform:

flowchart TB
    subgraph GitHub["GitHub Enterprise / Cloud"]
        GH_REPO["GitHub Repository\nWebhooks & Action Queue"]
        GH_API["GitHub Actions API\nRegistration / Status"]
    end

    subgraph AWS_Cloud["Customer AWS Account (Dedicated VPC)"]
        subgraph Public_Subnet["Public Subnets (us-east-1a / us-east-1b)"]
            ALB["Application Load Balancer\n(Webhook Ingestion)"]
            NAT["NAT Gateway\n(Outbound Traffic Only)"]
        end

        subgraph Private_Subnet["Isolated Private Subnets (No Ingress)"]
            ORCH["RunsOn Serverless Orchestrator\n(ECS Fargate / Lambda Engine)"]
            
            subgraph ASG["Auto Scaling Group (EC2 Spot Fleet)"]
                NODE1["Ephemeral Runner 1\nc6id.xlarge (Spot)"]
                NODE2["Ephemeral Runner 2\nc7g.xlarge (Spot)"]
                NODE3["Ephemeral Runner N\nc6a.xlarge (Spot)"]
            end
            
            VPCE_S3["S3 VPC Endpoint\n(Artifact Cache)"]
            VPCE_ECR["ECR VPC Endpoint\n(Base Container Images)"]
        end

        subgraph Security["Identity & Security Guardrails"]
            IAM["IAM Role + Session Manager\n(No SSH Keys)"]
            IMDS["IMDSv2 Enforced\n(Hop Limit = 1)"]
            KMS["AWS KMS\n(EBS & S3 Encryption)"]
        end
    end

    GH_REPO -->|TLS Webhook Events| ALB
    ALB -->|Forward Event| ORCH
    ORCH -->|Scale-Out Command| ASG
    ASG -->|Egress Route| NAT
    NAT -->|HTTPS Outbound| GH_API
    NODE1 -->|Resolve Artifacts| VPCE_S3
    NODE1 -->|Pull Base Layers| VPCE_ECR
    NODE1 -.->|Protected By| IMDS
    NODE1 -.->|Scoped Privileges| IAM

Key Architectural Guardrails

1. Zero Ingress Attack Surface: Runner instances reside within private subnets with no public IPv4 addresses and no inbound security group rules. All management traffic passes through AWS Systems Manager (SSM) Session Manager.

2. Ephemeral Lifecycle (One Job, One Machine): Each EC2 instance processes exactly one GitHub workflow job before terminating immediately. This eliminates lateral movement risks between untrusted pull requests and guarantees an immutably clean environment.

3. VPC Endpoints for Cost Elimination: Routing internal artifact storage to Amazon S3 and container layer pulls to Amazon Elastic Container Registry (ECR) via AWS Gateway and Interface VPC Endpoints prevents multi-terabyte internal CI traffic from transiting the NAT Gateway, avoiding $0.045/GB data processing fees.

4. IMDSv2 Enforcement: To prevent Server-Side Request Forgery (SSRF) vulnerabilities from exposing instance IAM credentials, the instance metadata service enforces HttpTokens = required with HttpPutResponseHopLimit = 1.

Production Terraform Configuration: Spot Launch Template

Below is the production Terraform configuration provisioning the hardened launch template and auto-scaling group for ephemeral EC2 spot runners, utilizing dynamic AMI resolution for region portability:

  # main.tf - Ephemeral AWS Spot Runner Launch Configuration
terraform {
  required_version = ">= 1.8.0"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.50"
    }
  }
}

variable "vpc_private_subnet_ids" {
  type        = list(string)
  description = "Private subnet IDs without public IPv4 assignment"
}

variable "github_runner_token_secret_arn" {
  type        = string
  description = "Secrets Manager ARN storing ephemeral GitHub Runner registration tokens"
}

  # Resolve official Ubuntu 24.04 LTS AMI dynamically across any AWS region
data "aws_ami" "ubuntu" {
  most_recent = true
  owners      = ["099720109477"] # Canonical

  filter {
    name   = "name"
    values = ["ubuntu/images/hvm-ssd-gp3/ubuntu-noble-24.04-amd64-server-*"]
  }

  filter {
    name   = "virtualization-type"
    values = ["hvm"]
  }
}

  # IAM Role for Runner Instances
resource "aws_iam_role" "runner_instance_role" {
  name = "github-actions-spot-runner-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRole"
        Effect = "Allow"
        Principal = {
          Service = "ec2.amazonaws.com"
        }
      }
    ]
  })
}

resource "aws_iam_policy" "runner_least_privilege" {
  name = "github-actions-spot-runner-least-privilege"

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Sid    = "SecretsManagerReadToken"
        Effect = "Allow"
        Action = [
          "secretsmanager:GetSecretValue"
        ]
        Resource = var.github_runner_token_secret_arn
      },
      {
        Sid    = "S3ArtifactSync"
        Effect = "Allow"
        Action = [
          "s3:GetObject",
          "s3:PutObject"
        ]
        Resource = "arn:aws:s3:::ci-build-artifacts-cache/*"
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "attach_custom" {
  role       = aws_iam_role.runner_instance_role.name
  policy_arn = aws_iam_policy.runner_least_privilege.arn
}

resource "aws_iam_instance_profile" "runner_profile" {
  name = "github-actions-spot-runner-profile"
  role = aws_iam_role.runner_instance_role.name
}

  # Hardened Launch Template for Ephemeral Spot Runners
resource "aws_launch_template" "spot_runner_lt" {
  name_prefix   = "runs-on-spot-"
  image_id      = data.aws_ami.ubuntu.id
  instance_type = "c6id.xlarge"

  # Enforce IMDSv2 security guardrails
  metadata_options {
    http_endpoint               = "enabled"
    http_tokens                 = "required"
    http_put_response_hop_limit = 1
    instance_metadata_tags      = "enabled"
  }

  instance_market_options {
    market_type = "spot"
    spot_options {
      max_price          = "0.05"
      spot_instance_type = "one-time"
    }
  }

  iam_instance_profile {
    arn = aws_iam_instance_profile.runner_profile.arn
  }

  # Local NVMe Instance Store mapping for high-speed Docker root
  block_device_mappings {
    device_name = "/dev/xvda"
    ebs {
      volume_size           = 50
      volume_type           = "gp3"
      iops                  = 3000
      throughput            = 125
      encrypted             = true
      delete_on_termination = true
    }
  }

  user_data = base64encode(<<-EOF
              #!/usr/bin/env bash
              set -euo pipefail

              # Format and mount local NVMe storage for Docker BuildKit scratchpad
              if [ -e /dev/nvme1n1 ]; then
                mkfs.ext4 -m 0 -F /dev/nvme1n1
                mkdir -p /mnt/scratch/docker
                mount -o noatime,nodiratime,discard /dev/nvme1n1 /mnt/scratch
                mkdir -p /mnt/scratch/docker
              fi

              # Configure Docker daemon to utilize high-speed NVMe mount
              cat <<EOD > /etc/docker/daemon.json
              {
                "data-root": "/mnt/scratch/docker",
                "storage-driver": "overlay2",
                "log-driver": "json-file",
                "log-opts": { "max-size": "50m", "max-file": "2" },
                "features": { "buildkit": true }
              }
              EOD

              systemctl restart docker

              # Initialize Spot Termination Interruption Watchdog
              /usr/local/bin/spot-interruption-drain.sh &

              # Register ephemeral single-job runner
              RUNNER_TOKEN=$(aws secretsmanager get-secret-value --secret-id "${var.github_runner_token_secret_arn}" --query SecretString --output text)
              su - ubuntu -c "/home/ubuntu/actions-runner/config.sh --url https://github.com/org/repo --token $${RUNNER_TOKEN} --ephemeral --unattended"
              su - ubuntu -c "/home/ubuntu/actions-runner/run.sh"

              # Self-terminate instance upon job completion
              INSTANCE_ID=$(curl -s -H "X-aws-ec2-metadata-token: $(curl -s -X PUT 'http://169.254.169.254/latest/api/token' -H 'X-aws-ec2-metadata-token-ttl-seconds: 60')" http://169.254.169.254/latest/meta-data/instance-id)
              aws ec2 terminate-instances --instance-ids "$${INSTANCE_ID}"
              EOF
  )

  tag_specifications {
    resource_type = "instance"
    tags = {
      Environment = "ci-ephemeral"
      ManagedBy   = "RunsOn-Terraform"
    }
  }
}

---

Failure Mode Analysis: EC2 Spot Preemption Handling During Long-Running Integration Suites

Utilizing AWS Spot instances yields substantial compute cost savings compared to on-demand pricing, but it introduces an immutable operational reality: EC2 Spot Preemption. Amazon EC2 reserves the right to reclaim spot capacity with a two-minute warning when on-demand capacity surges.

In typical CI configurations, an unhandled spot preemption causes severe operational failure modes:

1. Dangling Pipeline Failures: GitHub Actions marks the job as unceremoniously failed (The runner has lost communication with the service), confusing developers.

2. Corrupted Test State: Multi-tenant integration tests interacting with database shards or external state stores terminate without clean teardown hooks, creating orphaned locks.

3. Queue Stalls & Zombie Runners: The GitHub organization console retains registration for offline spot instances, stalling concurrency quotas until timeouts trip (frequently 15 to 30 minutes).

Spot Interruption Interception Architecture

To mitigate preemption risks, high-velocity infrastructure teams implement a multi-stage graceful drain engine. Amazon EC2 provides two separate signals over the Instance Metadata Service (IMDSv2):

  • EC2 Rebalance Recommendations: Emitted via /latest/meta-data/events/recommendations/rebalance prior to spot capacity reclamation.
  • Spot Instance Termination Notice: Emitted via /latest/meta-data/spot/instance-action exactly 120 seconds prior to hard hypervisor termination.
sequenceDiagram
    autonumber
    participant AWS as AWS EC2 Hypervisor
    participant IMDS as IMDSv2 (169.254.169.254)
    participant Watchdog as Daemon (spot-drain.sh)
    participant Runner as GitHub Actions Runner Agent
    participant Docker as Docker / Integration Suite
    participant S3 as Amazon S3 Artifact Cache

    AWS->>IMDS: Issue Spot Interruption Notice (120s remaining)
    loop Every 5 Seconds
        Watchdog->>IMDS: GET /meta-data/spot/instance-action
    end
    IMDS-->>Watchdog: HTTP 200 (Action: terminate, Time: T+120s)
    
    Note over Watchdog: PREEMPTION DETECTED: Stage 1 (T-115s)
    Watchdog->>Runner: Issue SIGTERM / Graceful Stop Signal
    
    Note over Watchdog: Stage 2: State Persistence (T-100s)
    Docker->>S3: Flush Intermediate Test Artifacts / Core Dumps
    
    Note over Watchdog: Stage 3: Container Teardown (T-70s)
    Watchdog->>Docker: docker compose down --volumes --timeout 20
    
    Note over Watchdog: Stage 4: Orchestrator Deregistration (T-40s)
    Watchdog->>Runner: ./config.sh remove --token [ephemeral]
    
    Note over Watchdog: Stage 5: Job Re-queue (T-20s)
    Watchdog->>Runner: Re-dispatch Workflow Job to On-Demand Fallback
    AWS->>Watchdog: Instance Terminated Cleanly at T-0s

Production Implementation: Spot Interruption Drain Daemon

The following shell daemon runs continuously as a background systemd service on every RunsOn ephemeral node. It polls IMDSv2 every five seconds, intercepts termination signals, flushes diagnostic state, gracefully deregisters from GitHub Actions, and triggers automated job retry.

  #!/usr/bin/env bash
  # spot-interruption-drain.sh - Graceful Preemption Handler for AWS Spot Runners
set -eo pipefail

LOG_FILE="/var/log/spot-drain.log"
exec >> "${LOG_FILE}" 2>&1

echo "[$(date -u +'%Y-%m-%dT%H:%M:%SZ')] Starting EC2 Spot Preemption Watchdog..."

get_imds_token() {
  curl -s -X PUT "http://169.254.169.254/latest/api/token" \
       -H "X-aws-ec2-metadata-token-ttl-seconds: 60"
}

check_spot_interruption() {
  local token
  token=$(get_imds_token)
  local http_code
  http_code=$(curl -s -o /tmp/spot-action.json -w "%{http_code}" \
              -H "X-aws-ec2-metadata-token: ${token}" \
              http://169.254.169.254/latest/meta-data/spot/instance-action)

  if [ "${http_code}" -eq 200 ]; then
    return 0
  fi
  return 1
}

  # Polling loop: 5-second interval preserves IMDS rate-limits
while true; do
  if check_spot_interruption; then
    NOTICE_TIME=$(jq -r .time /tmp/spot-action.json)
    ACTION=$(jq -r .action /tmp/spot-action.json)
    echo "[$(date -u +'%Y-%m-%dT%H:%M:%SZ')] CRITICAL: Spot Preemption Event Detected!"
    echo "Action: ${ACTION} scheduled at ${NOTICE_TIME}. Initiating 120-second drain sequence."

    # Stage 1: Notify GitHub Actions Runner Agent to stop accepting work
    RUNNER_PID=$(pgrep -f "Runner.Listener" || true)
    if [ -n "${RUNNER_PID}" ]; then
      echo "Dispatching SIGINT to Runner.Listener (PID: ${RUNNER_PID})..."
      kill -SIGINT "${RUNNER_PID}"
    fi

    # Stage 2: Gracefully stop all active Docker containers
    echo "Stopping active Docker containers with 20s grace period..."
    ACTIVE_CONTAINERS=$(docker ps -q)
    if [ -n "${ACTIVE_CONTAINERS}" ]; then
      docker stop --time 20 ${ACTIVE_CONTAINERS} || true
    fi

    # Stage 3: Flush crash dump logs to S3 diagnostic bucket
    INSTANCE_ID=$(curl -s -H "X-aws-ec2-metadata-token: $(get_imds_token)" \
                  http://169.254.169.254/latest/meta-data/instance-id)
    echo "Uploading diagnostic execution trace to S3..."
    tar -czf "/tmp/crash-trace-${INSTANCE_ID}.tar.gz" /var/log/ /home/ubuntu/actions-runner/_diag/ || true
    aws s3 cp "/tmp/crash-trace-${INSTANCE_ID}.tar.gz" "s3://ci-preemption-dumps/${INSTANCE_ID}.tar.gz" --quiet || true

    # Stage 4: Gracefully deregister from GitHub to prevent zombie concurrency lock
    echo "Deregistering runner from GitHub Actions pool..."
    if [ -f "/home/ubuntu/actions-runner/.runner" ]; then
      su - ubuntu -c "cd /home/ubuntu/actions-runner && ./config.sh remove --token $(cat /home/ubuntu/.runner_token) --force" || true
    fi

    echo "[$(date -u +'%Y-%m-%dT%H:%M:%SZ')] Drain completed successfully. Awaiting instance termination."
    exit 0
  fi

  sleep 5
done

Falsifiability & Chaos Engineering Verification

To validate that the preemption handler operates correctly prior to production deployment, platform teams must perform falsifiability testing:

  • Chaos Injection Mechanism: Utilize AWS Fault Injection Simulator (FIS) targeting the runner auto-scaling group using action aws:ec2:send-spot-instance-interruptions with a 2-minute duration.
  • Pass Criteria:

1. The GitHub Actions job exits with an explicit Spot Preemption Handled diagnostic annotation rather than a network timeout.

2. The runner unregisters from GitHub Actions within 45 seconds of the interruption injection.

3. The workflow job automatically re-queues and succeeds on an on-demand fallback runner within 90 seconds.

  • Fail Criteria: Any occurrence of a runner lingering in the GitHub Actions runner inventory for greater than 3 minutes, or an uncommitted database migration leaving residual locks on shared integration testing fixtures.

---

Production Configuration Traces: BuildKit Cache & GitHub Actions Workflow

To achieve optimal performance across heterogeneous runners, production architectures employ optimized Docker multi-stage builds coupled with hybrid runner workflow definitions.

Multi-Stage Dockerfile with BuildKit Cache Mounts

The following Dockerfile demonstrates advanced cache mount configurations. Notice the explicit target mounts for Go build artifacts, Node modules, and package manager layers, ensuring that neither local disk wipes nor remote network transfers repeat compiler dependency resolution:

  # syntax=docker/dockerfile:1.7-labs
  # Multi-stage production container with explicit BuildKit cache targets

  # Stage 1: Build Go Backend Services
FROM golang:1.23-alpine AS go-builder
WORKDIR /src

  # Utilize cache mounts for Go download and build cache
RUN --mount=type=cache,target=/go/pkg/mod \
    --mount=type=bind,source=go.mod,target=go.mod \
    --mount=type=bind,source=go.sum,target=go.sum \
    go mod download -x

COPY cmd/ ./cmd/
COPY internal/ ./internal/

  # Build statically compiled binary with persistent compiler cache
RUN --mount=type=cache,target=/go/pkg/mod \
    --mount=type=cache,target=/root/.cache/go-build \
    CGO_ENABLED=0 GOOS=linux go build -trimpath -ldflags="-s -w" -o /bin/api-server ./cmd/api

  # Stage 2: Build Next.js 15 Frontend
FROM node:22-alpine AS web-builder
WORKDIR /app

ENV NEXT_TELEMETRY_DISABLED=1
ENV NODE_ENV=production

  # Mount yarn/npm package cache to prevent network package re-fetches
COPY package.json yarn.lock ./
RUN --mount=type=cache,target=/usr/local/share/.cache/yarn \
    yarn install --frozen-lockfile

COPY . .

  # Cache Next.js intermediate compilation artifacts across builds
RUN --mount=type=cache,target=/app/.next/cache \
    yarn build

  # Stage 3: Minimal Distroless Production Artifact
FROM gcr.io/distroless/static-debian12:nonroot
WORKDIR /app

COPY --from=go-builder /bin/api-server /app/api-server
COPY --from=web-builder /app/public /app/public
COPY --from=web-builder /app/.next/standalone /app/
COPY --from=web-builder /app/.next/static /app/.next/static

USER nonroot:nonroot
EXPOSE 8080 3000

ENTRYPOINT ["/app/api-server"]

Production GitHub Actions Workflow (`.github/workflows/ci.yml`)

The following production workflow integrates native fallback logic. It routes builds to specialized high-performance runners (RunsOn or Depot) while maintaining automatic fallback to standard GitHub-hosted instances if self-hosted spot capacity is constrained.

name: Enterprise CI Build & Test Matrix

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true

jobs:
  # Job 1: High-Speed Docker Build via Depot or RunsOn
  build-and-compile:
    name: Build Multi-Stage Monorepo Container
    # Hybrid runner selector: RunsOn Spot with fallback to Ubicloud
    runs-on: [self-hosted, runson, spot, linux-x64]
    timeout-minutes: 25

    steps:
      - name: Checkout Source Code
        uses: actions/checkout@v4
        with:
          fetch-depth: 2

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3
        with:
          driver-opts: image=moby/buildkit:v0.13.1

      - name: Configure Depot Remote Acceleration
        uses: depot/setup-action@v1
        with:
          oidc: true

      - name: Build and Push Monorepo Container Layer
        uses: depot/build-push-action@v1
        with:
          context: .
          file: ./Dockerfile
          push: false
          tags: internal-monorepo:${{ github.sha }}
          # Combine Depot distributed NVMe-oF cache with GitHub fallback
          cache-from: type=depot
          cache-to: type=depot

  # Job 2: Distributed Integration Test Matrix with Redis & PostgreSQL
  integration-test-suite:
    name: Integration Tests (${{ matrix.shard }}/4)
    needs: build-and-compile
    runs-on: ubicloud-standard-4
    timeout-minutes: 15

    services:
      postgres:
        image: postgres:16-alpine
        env:
          POSTGRES_DB: integration_test
          POSTGRES_PASSWORD: testpassword
        ports:
          - 5432:5432
        options: >-
          --health-cmd pg_isready
          --health-interval 5s
          --health-timeout 3s
          --health-retries 5

      redis:
        image: redis:7.2-alpine
        ports:
          - 6379:6379
        options: >-
          --health-cmd "redis-cli ping"
          --health-interval 5s
          --health-timeout 3s
          --health-retries 5

    strategy:
      fail-fast: false
      matrix:
        shard: [1, 2, 3, 4]

    steps:
      - name: Checkout Code
        uses: actions/checkout@v4

      - name: Setup Node.js Runtime
        uses: actions/setup-node@v4
        with:
          node-version: 22
          cache: 'npm'

      - name: Run Sharded Test Pipeline
        env:
          DATABASE_URL: postgres://postgres:testpassword@localhost:5432/integration_test
          REDIS_URL: redis://localhost:6379
          SHARD_INDEX: ${{ matrix.shard }}
          TOTAL_SHARDS: 4
        run: |
          npm ci
          npm run test:integration -- --shard=${SHARD_INDEX}/${TOTAL_SHARDS}

---

Strategic Architectural Decision Framework & Migration Playbook

Selecting the optimal CI/CD compute layer requires mapping organizational constraints—such as security compliance, existing cloud commitments, and engineering team size—against platform characteristics.

flowchart TD
    START["Evaluate CI/CD Runner Architecture"] --> Q_HOURS{"Monthly Runner Volume > 20,000 Minutes?"}
    
    Q_HOURS -- No --> GH_HOSTED["GitHub-Hosted Runners\n(Standard / Large Tiers)\nZero Ops, Ideal for Early Stage"]
    
    Q_HOURS -- Yes --> Q_MONO{"Docker Monorepo Image > 10GB or Multi-Arch?"}
    
    Q_MONO -- Yes --> DEPOT["Depot Managed Acceleration\nDistributed NVMe-oF Cache\nNative AMD64/ARM64 BuildKit"]
    
    Q_MONO -- No --> Q_VPC{"Strict Enterprise Compliance / Private AWS VPC Required?"}
    
    Q_VPC -- Yes --> RUNSON["RunsOn (AWS Spot in Dedicated VPC)\nWholesale Compute (~$0.0012/min)\nZero Data Egress via Endpoints"]
    
    Q_VPC -- No --> UBICLOUD["Ubicloud Runners\nDrop-in GitHub YAML Replacement\nCommodity Floor ($0.00125/min)"]

Decision Rubric

1. GitHub-Hosted Runners: Best suited for early-stage teams consuming fewer than 15,000 build minutes per month. The operational overhead of managing custom launch templates, AMIs, or cloud networking outweighs compute savings at small volumes.

2. Depot: The preferred solution for teams suffering from build cache latency. If your engineers spend significant time waiting on docker build steps, multi-architecture compilation, or monorepo layer invalidations, Depot delivers immediate returns by accelerating warm builds without requiring custom AWS infrastructure management.

3. RunsOn: The standard for security-sensitive organizations, high-volume CI environments (>50,000 minutes/month), and teams holding significant AWS Enterprise Discount commitments. Running inside your own VPC helps satisfy enterprise compliance objectives (such as SOC 2, HIPAA, and ISO 27001 data isolation controls) by ensuring test code, build artifacts, and staging database secrets remain strictly within your AWS perimeter.

4. Ubicloud: The optimal migration path for engineering organizations seeking an immediate 70% to 80% cost reduction on generic Linux and ARM workloads without modifying infrastructure or configuring AWS IAM. Changing one line in your workflow files (runs-on: ubicloud-standard-4) achieves immediate savings.

For enterprise teams executing comprehensive CI/CD transformations or redesigning cloud-native microservices, partnering with seasoned engineers through custom software engineering services ensures resilient infrastructure architectures, hardened networking boundaries, and automated FinOps enforcement from day one.

---

Frequently Asked Questions

How does security isolation compare between GitHub-hosted ephemeral microVMs and self-hosted AWS Spot runners?

GitHub-hosted runners utilize ephemeral Azure virtual machines that are destroyed upon job completion. Self-hosted AWS Spot runners using RunsOn achieve comparable isolation by operating in --ephemeral mode. Under this configuration, the EC2 instance accepts exactly one job, terminates immediately afterward, and runs within a dedicated, private AWS subnet with zero inbound internet connectivity. Security is further hardened through IMDSv2 enforcement and IAM instance profiles that utilize short-lived STS credentials.

Can RunsOn and Ubicloud natively execute ARM64 builds without QEMU emulation penalties?

Yes. Both RunsOn and Ubicloud support native ARM64 silicon without requiring QEMU emulation. RunsOn provisions native AWS Graviton3 and Graviton4 EC2 instances (c7g, c8g), while Ubicloud provisions native Ampere Altra bare-metal nodes. Because compilation occurs on bare silicon rather than through user-space CPU emulation (qemu-user-static), ARM64 container builds execute without binary translation penalties, significantly outperforming cross-architecture QEMU emulation on x86 runners for compiled languages.

How does Depot eliminate the 10GB GitHub Actions cache limit for enterprise monorepos?

GitHub Actions enforces a hard 10GB storage ceiling per repository across all active workflows. When this quota is reached, GitHub automatically evicts older cache archives regardless of frequency of use. Depot circumvents this by streaming BuildKit cache operations directly to an external distributed Ceph cluster over high-speed NVMe-oF interconnects. Caches are managed on independent, persistent SSD volumes that are shared across concurrent branches and pull requests with zero repository-level caps.

What happens if an AWS Spot instance is preempted during an uncommitted database migration in an integration test?

If an unhandled preemption occurs, database transactions running against an external shared database can leave open connections or locked tables. Implementing the IMDSv2 preemption daemon addresses this failure mode by catching the 120-second termination notice. The script issues a SIGINT to the test runner, executes docker compose down -v to cleanly terminate containerized databases, flushes in-flight logs to S3, and notifies the GitHub API to reschedule the job on an on-demand fallback instance.

---

Sources

Related articles