# GitHub Actions vs RunsOn vs Depot vs Ubicloud: CI/CD Runner Performance & FinOps Benchmarks
Summary: Modern engineering organizations face ballooning CI/CD compute bills and agonizing pipeline latencies driven by stock GitHub-hosted runner constraints. In this comprehensive benchmark, we evaluate GitHub Actions vs RunsOn vs Depot vs Ubicloud across 2026 compute pricing models, Docker BuildKit cache performance on a 15GB monorepo image, AWS Spot ephemeral auto-scaling architectures, and fail-safe preemption handling to deliver actionable FinOps strategies for high-velocity platform engineering teams.
---
CI/CD Execution Economics: The Hidden Tax of Hosted Runners
Continuous Integration pipelines represent one of the fastest-growing operational line items in modern cloud infrastructure budgets. Engineering leaders routinely observe monthly continuous integration bills escalating at non-linear rates relative to headcount growth. The root cause rarely stems from excessive test volume; rather, it originates in the compute architecture, network topology, and billing paradigms of standard cloud-hosted runners.
Standard GitHub-hosted runners rely on multi-tenant Azure virtual machines provisioned on demand. While operationally frictionless, this convenience imposes structural inefficiencies that penalize growing engineering teams:
1. Per-Minute Billing and Upward Rounding: GitHub-hosted runners bill in whole minute increments. A build executing in 61 seconds incurs a 120-second charge, creating an immediate billing overhead on fast-feedback test suites.
2. Restrictive Cache Throughput and Storage Caps: GitHub Actions Cache (actions/cache) imposes a strict 10GB storage limit per repository. Furthermore, cache archives transit over public HTTPS endpoints, subject to standard cloud network latency, egress limits, and client-side tar compression and decompression overhead. On multi-gigabyte container images, the time spent decompressing and transferring tarballs frequently negates the latency savings of caching altogether.
3. Compute Oversubscription: Stock 2-core and 4-core GitHub runners share hyperthreads on virtualized cloud hardware without dedicated CPU pinning or local NVMe storage, producing variable build execution times depending on neighbor tenant activity.
4. Network Egress Penalties: Running integration tests that pull intermediate container layers or upload test artifacts across cloud boundaries introduces steady data transfer charges that accumulate silently.
To combat these bottlenecks, platform engineers evaluate three alternative paradigms:
- RunsOn: A commercial source-available appliance licensed annually that orchestrates ephemeral GitHub Actions runners directly inside your private AWS account. It provisions native Amazon EC2 Spot instances within your isolated VPC, utilizing wholesale AWS compute pricing without markup.
- Depot: A managed acceleration platform focused on distributed remote Docker BuildKit engines and high-performance native CI runners. Depot features persistent NVMe-backed distributed SSD caches and native multi-architecture execution (AMD64 and ARM64) without QEMU emulation overhead.
- Ubicloud: An open-source, portable cloud provider operating on bare-metal infrastructure (such as Hetzner and Leaseweb). Ubicloud offers drop-in GitHub Actions runner compatibility with zero code changes at commodity hardware price floors.
Evaluating github actions vs runson vs depot alongside Ubicloud requires analyzing compute unit costs, cache synchronization mechanics, and infrastructure control.
---
2026 Compute Cost Comparison Matrix & FinOps Modeling
Achieving effective ci cd runner cost optimization 2026 mandates a granular understanding of billing precision and underlying compute costs. The table below details the compute pricing tiers across standard Linux x86 and ARM64 architectures for GitHub-hosted runners, Depot, Ubicloud, and RunsOn.
Compute Pricing & Feature Matrix (2026 Benchmarks)
| Platform | Architectural Model | 2-Core x86 ($/min) | 4-Core x86 ($/min) | 8-Core ARM64 ($/min) | Billing Precision | Cache Storage Model | Private VPC Support |
|---|---|---|---|---|---|---|---|
| GitHub-Hosted | Multi-tenant Azure VM | $0.0060 | $0.0120 | $0.0140 | Per-minute (round-up) | 10GB S3/Blob (actions/cache) | Enterprise only (VNet injection) |
| Ubicloud | Bare-metal KVM (Hetzner/IaaS) | $0.00125 | $0.0025 | $0.0050 | Per-second | Local NVMe + persistent volume | WireGuard / Private Subnet |
| Depot | Managed BuildKit / MicroVM | $0.0060 | $0.0120 | $0.0240 | Per-second | Distributed NVMe (Ceph/NVMe-oF) | AWS Direct Connect / Peering |
| RunsOn | AWS EC2 Spot in Customer VPC | ~$0.0012* | ~$0.0024* | ~$0.0031* | Per-second (EC2 Spot) | Ephemeral NVMe / Local EBS gp3 | Native (Customer AWS Account) |
\RunsOn compute rates reflect average AWS us-east-1 Spot market pricing for c6i.large ($0.017/hr = ~$0.00028/min raw EC2, plus EBS gp3 storage and NAT Gateway egress amortization, totaling ~$0.0012/min). RunsOn software is licensed annually based on monthly job volume tiers (starting at €300/year for Starter up to €3,600/year for Enterprise, with no per-seat or per-developer fees), avoiding variable compute markups.*
Monthly TCO Projections: 50,000 vs. 200,000 Monthly Minutes
To demonstrate real-world FinOps implications, consider an engineering organization operating 25 developers generating 50,000 runner minutes per month, scaling to an enterprise tier of 200,000 runner minutes per month (standard 4-core x86 workloads).
flowchart LR
subgraph Workload["Monthly CI Workload (4-Core x86)"]
W1["50,000 Build Minutes"]
W2["200,000 Build Minutes"]
end
subgraph Costs50k["50k Minutes / Month TCO"]
GH1["GitHub: $600 (base compute)"]
DEP1["Depot: $600 + Cache ($40) = $640"]
UBI1["Ubicloud: $125"]
RO1["RunsOn: ~$120 AWS Spot + ~$75 License = ~$195"]
end
subgraph Costs200k["200k Minutes / Month TCO"]
GH2["GitHub: $2,400 (base compute)"]
DEP2["Depot: $2,400 + Cache ($60) = $2,460"]
UBI2["Ubicloud: $500"]
RO2["RunsOn: ~$480 AWS Spot + ~$150 License = ~$630"]
end
W1 --> GH1
W1 --> DEP1
W1 --> UBI1
W1 --> RO1
W2 --> GH2
W2 --> DEP2
W2 --> UBI2
W2 --> RO21. GitHub-Hosted: At 50,000 minutes of 4-core compute ($0.012/min), baseline compute expenditure equals $600/month. At 200,000 minutes, compute expands to $2,400/month. Furthermore, because GitHub bills per whole minute, jobs that complete shortly after an integer minute boundary (such as a 61-second test suite billed as 120 seconds) incur ceiling rounding overhead, increasing effective spend on high-frequency, short-duration CI jobs.
2. Depot: At $0.012/min for 4-core compute ($0.003/vCPU-minute) billed per second, 50,000 minutes yields $600/month in compute. At 200,000 minutes, compute costs $2,400/month. Factoring in distributed persistent cache storage ($0.20/GB/month for additional cache storage), total monthly expenditure equals approximately $640/month at 50k minutes and $2,460/month at 200k minutes. Additionally, because Depot maintains persistent SSD BuildKit caches across runs, teams frequently eliminate redundant dependency downloads and layer recompilations, reducing the total billable minutes required to complete Docker pipelines.
3. Ubicloud: Under ubicloud github actions pricing ($0.0025/min for 4-core), 50,000 minutes equals $125/month. At 200,000 minutes, total expenditure is $500/month—a substantial reduction compared to standard hosted alternatives with zero pipeline configuration changes.
4. RunsOn: At wholesale AWS Spot rates (~$0.0024/min fully loaded) combined with an amortized annual RunsOn license (€900/year for Growth, or ~$75/month), 50,000 minutes costs approximately $195/month. At 200,000 minutes, the spot compute total (~$480) plus an amortized Scale license (€1,800/year, or ~$150/month) totals approximately $630/month. Organizations with active AWS Enterprise Discount Program (EDP) agreements or upfront cloud credits can offset runner compute spend directly against their AWS infrastructure commitments, lowering net cash outlay.
---
Docker BuildKit Cache Architecture: Monorepo Workload Profile
Container build efficiency is the primary determinant of developer feedback loop velocity. To evaluate docker buildkit cache optimization in multi-stage builds, consider a representative 15.4GB polyglot production monorepo image containing:
- A Next.js 15 enterprise frontend with dynamic server-side rendering bundles.
- A Go 1.23 distributed backend service with vendored dependencies and CGO bindings.
- A Python 3.12 machine learning inference pipeline with PyTorch and CUDA base layers.
- A PostgreSQL client migration layer with Node.js automation scripts.
When compiling large monorepos, platforms exhibit starkly different performance characteristics between cold builds (clean environment, zero local cache) and warm builds (incremental modifications to application code with cached dependency layers).
Docker BuildKit Architectural Performance Matrix
| Runner Platform | Compute & Storage Architecture | Cold Build Mechanics | Warm Build Mechanics | Cache Synchronization Model | Throughput Characteristics |
|---|---|---|---|---|---|
| GitHub Actions (`gha`) | 4-Core x86 (16GB RAM) + Azure Blob Storage | Downloads base layers sequentially from container registry; compiles un-cached stages | Re-downloads tar archives over HTTPS; uncompresses cache to local disk | Remote archive export/import via GitHub REST API | Constrained by network round-trips and client CPU tar extraction |
| Ubicloud Standard | 4-Core x86 (16GB RAM) + Local Bare-Metal NVMe | Fetches layers directly over high-bandwidth datacenter uplinks | Reuses local layers when mapped to persistent storage or co-located nodes | Host-local disk storage and container registry cache | High local NVMe read/write speeds over PCIe bus |
| Depot Managed | 4-Core x86 + Remote NVMe-oF Distributed Cache | Streams parallel layer builds directly to high-throughput SSD cluster | Zero-copy layer hash verification; skips unchanged build steps | Persistent distributed NVMe-oF storage pool | Direct block-level sync bypassing public network tar serialization |
| RunsOn AWS Spot | c6id.xlarge (Dedicated Local NVMe Instance Store) | Direct line-rate pull from Amazon ECR via AWS Gateway VPC Endpoints | Reads cached Docker layer blocks directly from local NVMe scratch volume | Local NVMe instance store formatted as Docker root | Line-rate PCIe transfers with zero external network egress |
Technical Analysis of Cache Mechanics
The divergent performance analyzed across depot buildkit cache benchmarks and native runners highlights the fundamental architectural differences in how BuildKit backends transfer intermediate layers:
flowchart TD
subgraph GHA_Model["GitHub Actions Native (type=gha)"]
GHA_RUNNER["Ephemeral Runner\n(Local Ext4)"]
GHA_TAR["Tarball Compression Engine\n(CPU-Bound Serialization)"]
GHA_REST["GitHub API / Blob Storage\n(HTTPS Network Transit)"]
GHA_RUNNER -->|Export Layers| GHA_TAR
GHA_TAR -->|HTTPS Stream| GHA_REST
end
subgraph Depot_Model["Depot Distributed Architecture"]
DEP_RUNNER["Depot Runner Client\n(Thin CLI Daemon)"]
DEP_BLOCK["Distributed NVMe-oF Storage\n(Ceph Block Cluster)"]
DEP_BUILDKIT["Remote Dedicated BuildKit Daemon\n(Persistent Shared Cache)"]
DEP_RUNNER <-->|mTLS Layer Sync| DEP_BUILDKIT
DEP_BUILDKIT <-->|Zero-Copy Block Sync| DEP_BLOCK
end
subgraph RunsOn_Model["RunsOn Ephemeral AWS Spot (type=local)"]
RO_INSTANCE["EC2 c6id.xlarge Node\n(Spot Instance)"]
RO_NVME["Local NVMe Instance Store\n(/mnt/scratch overlay2)"]
RO_EBS["EBS gp3 Root Volume\n(Pre-baked Custom AMI)"]
RO_INSTANCE <-->|Line-rate PCIe Bus| RO_NVME
RO_INSTANCE <-.->|Init Mount| RO_EBS
end- The GitHub Actions Bottleneck: Under the native
type=ghacache exporter, BuildKit serializes filesystem layers into compressed tarballs, transmits them over public HTTPS endpoints, and uncompresses them on subsequent runs. For multi-gigabyte container images with dozens of layers, compressing, uploading, downloading, and unpacking tar archives creates significant CPU and network latency, even when most layers remain unchanged. Furthermore, when total repository cache exceeds GitHub's 10GB limit, older archives are evicted under a least-recently-used (LRU) policy, triggering full cold rebuilds. - Depot Distributed Cache: Depot bypasses network tar serialization entirely. The builder client communicates with a dedicated remote BuildKit daemon backed by a distributed NVMe storage cluster (utilizing Ceph / NVMe-oF). Intermediate layer blocks remain uncompressed and instantly accessible. When an incremental build is triggered, BuildKit performs fast block-level hash checks and skips unchanged build stages, delivering rapid warm-build completions without transferring gigabytes of tar archives over the internet.
- RunsOn Local NVMe Acceleration: RunsOn provisions AWS EC2 instances equipped with direct NVMe instance store hardware (such as the
c6idorc7gdinstance families). By formatting the local NVMe drive directly as the Docker daemon root (/mnt/scratch/docker), all layer reads and writes execute directly over the physical PCIe bus. Combined with Amazon ECR Interface VPC Endpoints, images transfer at internal AWS network speeds without traversing the public internet.
---
Ephemeral Runner Architecture in an Isolated AWS VPC
Deploying ephemeral ci cd runners aws environments ensures that each compute instance processes exactly one GitHub workflow job before terminating immediately. Running self hosted github runner spot instances securely requires an architectural design that isolates compute instances inside a private subnet while maintaining automated elasticity.
The following architecture demonstrates an enterprise-grade RunsOn deployment utilizing AWS CloudFormation or Terraform:
flowchart TB
subgraph GitHub["GitHub Enterprise / Cloud"]
GH_REPO["GitHub Repository\nWebhooks & Action Queue"]
GH_API["GitHub Actions API\nRegistration / Status"]
end
subgraph AWS_Cloud["Customer AWS Account (Dedicated VPC)"]
subgraph Public_Subnet["Public Subnets (us-east-1a / us-east-1b)"]
ALB["Application Load Balancer\n(Webhook Ingestion)"]
NAT["NAT Gateway\n(Outbound Traffic Only)"]
end
subgraph Private_Subnet["Isolated Private Subnets (No Ingress)"]
ORCH["RunsOn Serverless Orchestrator\n(ECS Fargate / Lambda Engine)"]
subgraph ASG["Auto Scaling Group (EC2 Spot Fleet)"]
NODE1["Ephemeral Runner 1\nc6id.xlarge (Spot)"]
NODE2["Ephemeral Runner 2\nc7g.xlarge (Spot)"]
NODE3["Ephemeral Runner N\nc6a.xlarge (Spot)"]
end
VPCE_S3["S3 VPC Endpoint\n(Artifact Cache)"]
VPCE_ECR["ECR VPC Endpoint\n(Base Container Images)"]
end
subgraph Security["Identity & Security Guardrails"]
IAM["IAM Role + Session Manager\n(No SSH Keys)"]
IMDS["IMDSv2 Enforced\n(Hop Limit = 1)"]
KMS["AWS KMS\n(EBS & S3 Encryption)"]
end
end
GH_REPO -->|TLS Webhook Events| ALB
ALB -->|Forward Event| ORCH
ORCH -->|Scale-Out Command| ASG
ASG -->|Egress Route| NAT
NAT -->|HTTPS Outbound| GH_API
NODE1 -->|Resolve Artifacts| VPCE_S3
NODE1 -->|Pull Base Layers| VPCE_ECR
NODE1 -.->|Protected By| IMDS
NODE1 -.->|Scoped Privileges| IAMKey Architectural Guardrails
1. Zero Ingress Attack Surface: Runner instances reside within private subnets with no public IPv4 addresses and no inbound security group rules. All management traffic passes through AWS Systems Manager (SSM) Session Manager.
2. Ephemeral Lifecycle (One Job, One Machine): Each EC2 instance processes exactly one GitHub workflow job before terminating immediately. This eliminates lateral movement risks between untrusted pull requests and guarantees an immutably clean environment.
3. VPC Endpoints for Cost Elimination: Routing internal artifact storage to Amazon S3 and container layer pulls to Amazon Elastic Container Registry (ECR) via AWS Gateway and Interface VPC Endpoints prevents multi-terabyte internal CI traffic from transiting the NAT Gateway, avoiding $0.045/GB data processing fees.
4. IMDSv2 Enforcement: To prevent Server-Side Request Forgery (SSRF) vulnerabilities from exposing instance IAM credentials, the instance metadata service enforces HttpTokens = required with HttpPutResponseHopLimit = 1.
Production Terraform Configuration: Spot Launch Template
Below is the production Terraform configuration provisioning the hardened launch template and auto-scaling group for ephemeral EC2 spot runners, utilizing dynamic AMI resolution for region portability:
# main.tf - Ephemeral AWS Spot Runner Launch Configuration
terraform {
required_version = ">= 1.8.0"
required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 5.50"
}
}
}
variable "vpc_private_subnet_ids" {
type = list(string)
description = "Private subnet IDs without public IPv4 assignment"
}
variable "github_runner_token_secret_arn" {
type = string
description = "Secrets Manager ARN storing ephemeral GitHub Runner registration tokens"
}
# Resolve official Ubuntu 24.04 LTS AMI dynamically across any AWS region
data "aws_ami" "ubuntu" {
most_recent = true
owners = ["099720109477"] # Canonical
filter {
name = "name"
values = ["ubuntu/images/hvm-ssd-gp3/ubuntu-noble-24.04-amd64-server-*"]
}
filter {
name = "virtualization-type"
values = ["hvm"]
}
}
# IAM Role for Runner Instances
resource "aws_iam_role" "runner_instance_role" {
name = "github-actions-spot-runner-role"
assume_role_policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
Action = "sts:AssumeRole"
Effect = "Allow"
Principal = {
Service = "ec2.amazonaws.com"
}
}
]
})
}
resource "aws_iam_policy" "runner_least_privilege" {
name = "github-actions-spot-runner-least-privilege"
policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
Sid = "SecretsManagerReadToken"
Effect = "Allow"
Action = [
"secretsmanager:GetSecretValue"
]
Resource = var.github_runner_token_secret_arn
},
{
Sid = "S3ArtifactSync"
Effect = "Allow"
Action = [
"s3:GetObject",
"s3:PutObject"
]
Resource = "arn:aws:s3:::ci-build-artifacts-cache/*"
}
]
})
}
resource "aws_iam_role_policy_attachment" "attach_custom" {
role = aws_iam_role.runner_instance_role.name
policy_arn = aws_iam_policy.runner_least_privilege.arn
}
resource "aws_iam_instance_profile" "runner_profile" {
name = "github-actions-spot-runner-profile"
role = aws_iam_role.runner_instance_role.name
}
# Hardened Launch Template for Ephemeral Spot Runners
resource "aws_launch_template" "spot_runner_lt" {
name_prefix = "runs-on-spot-"
image_id = data.aws_ami.ubuntu.id
instance_type = "c6id.xlarge"
# Enforce IMDSv2 security guardrails
metadata_options {
http_endpoint = "enabled"
http_tokens = "required"
http_put_response_hop_limit = 1
instance_metadata_tags = "enabled"
}
instance_market_options {
market_type = "spot"
spot_options {
max_price = "0.05"
spot_instance_type = "one-time"
}
}
iam_instance_profile {
arn = aws_iam_instance_profile.runner_profile.arn
}
# Local NVMe Instance Store mapping for high-speed Docker root
block_device_mappings {
device_name = "/dev/xvda"
ebs {
volume_size = 50
volume_type = "gp3"
iops = 3000
throughput = 125
encrypted = true
delete_on_termination = true
}
}
user_data = base64encode(<<-EOF
#!/usr/bin/env bash
set -euo pipefail
# Format and mount local NVMe storage for Docker BuildKit scratchpad
if [ -e /dev/nvme1n1 ]; then
mkfs.ext4 -m 0 -F /dev/nvme1n1
mkdir -p /mnt/scratch/docker
mount -o noatime,nodiratime,discard /dev/nvme1n1 /mnt/scratch
mkdir -p /mnt/scratch/docker
fi
# Configure Docker daemon to utilize high-speed NVMe mount
cat <<EOD > /etc/docker/daemon.json
{
"data-root": "/mnt/scratch/docker",
"storage-driver": "overlay2",
"log-driver": "json-file",
"log-opts": { "max-size": "50m", "max-file": "2" },
"features": { "buildkit": true }
}
EOD
systemctl restart docker
# Initialize Spot Termination Interruption Watchdog
/usr/local/bin/spot-interruption-drain.sh &
# Register ephemeral single-job runner
RUNNER_TOKEN=$(aws secretsmanager get-secret-value --secret-id "${var.github_runner_token_secret_arn}" --query SecretString --output text)
su - ubuntu -c "/home/ubuntu/actions-runner/config.sh --url https://github.com/org/repo --token $${RUNNER_TOKEN} --ephemeral --unattended"
su - ubuntu -c "/home/ubuntu/actions-runner/run.sh"
# Self-terminate instance upon job completion
INSTANCE_ID=$(curl -s -H "X-aws-ec2-metadata-token: $(curl -s -X PUT 'http://169.254.169.254/latest/api/token' -H 'X-aws-ec2-metadata-token-ttl-seconds: 60')" http://169.254.169.254/latest/meta-data/instance-id)
aws ec2 terminate-instances --instance-ids "$${INSTANCE_ID}"
EOF
)
tag_specifications {
resource_type = "instance"
tags = {
Environment = "ci-ephemeral"
ManagedBy = "RunsOn-Terraform"
}
}
}---
Failure Mode Analysis: EC2 Spot Preemption Handling During Long-Running Integration Suites
Utilizing AWS Spot instances yields substantial compute cost savings compared to on-demand pricing, but it introduces an immutable operational reality: EC2 Spot Preemption. Amazon EC2 reserves the right to reclaim spot capacity with a two-minute warning when on-demand capacity surges.
In typical CI configurations, an unhandled spot preemption causes severe operational failure modes:
1. Dangling Pipeline Failures: GitHub Actions marks the job as unceremoniously failed (The runner has lost communication with the service), confusing developers.
2. Corrupted Test State: Multi-tenant integration tests interacting with database shards or external state stores terminate without clean teardown hooks, creating orphaned locks.
3. Queue Stalls & Zombie Runners: The GitHub organization console retains registration for offline spot instances, stalling concurrency quotas until timeouts trip (frequently 15 to 30 minutes).
Spot Interruption Interception Architecture
To mitigate preemption risks, high-velocity infrastructure teams implement a multi-stage graceful drain engine. Amazon EC2 provides two separate signals over the Instance Metadata Service (IMDSv2):
- EC2 Rebalance Recommendations: Emitted via
/latest/meta-data/events/recommendations/rebalanceprior to spot capacity reclamation. - Spot Instance Termination Notice: Emitted via
/latest/meta-data/spot/instance-actionexactly 120 seconds prior to hard hypervisor termination.
sequenceDiagram
autonumber
participant AWS as AWS EC2 Hypervisor
participant IMDS as IMDSv2 (169.254.169.254)
participant Watchdog as Daemon (spot-drain.sh)
participant Runner as GitHub Actions Runner Agent
participant Docker as Docker / Integration Suite
participant S3 as Amazon S3 Artifact Cache
AWS->>IMDS: Issue Spot Interruption Notice (120s remaining)
loop Every 5 Seconds
Watchdog->>IMDS: GET /meta-data/spot/instance-action
end
IMDS-->>Watchdog: HTTP 200 (Action: terminate, Time: T+120s)
Note over Watchdog: PREEMPTION DETECTED: Stage 1 (T-115s)
Watchdog->>Runner: Issue SIGTERM / Graceful Stop Signal
Note over Watchdog: Stage 2: State Persistence (T-100s)
Docker->>S3: Flush Intermediate Test Artifacts / Core Dumps
Note over Watchdog: Stage 3: Container Teardown (T-70s)
Watchdog->>Docker: docker compose down --volumes --timeout 20
Note over Watchdog: Stage 4: Orchestrator Deregistration (T-40s)
Watchdog->>Runner: ./config.sh remove --token [ephemeral]
Note over Watchdog: Stage 5: Job Re-queue (T-20s)
Watchdog->>Runner: Re-dispatch Workflow Job to On-Demand Fallback
AWS->>Watchdog: Instance Terminated Cleanly at T-0sProduction Implementation: Spot Interruption Drain Daemon
The following shell daemon runs continuously as a background systemd service on every RunsOn ephemeral node. It polls IMDSv2 every five seconds, intercepts termination signals, flushes diagnostic state, gracefully deregisters from GitHub Actions, and triggers automated job retry.
#!/usr/bin/env bash
# spot-interruption-drain.sh - Graceful Preemption Handler for AWS Spot Runners
set -eo pipefail
LOG_FILE="/var/log/spot-drain.log"
exec >> "${LOG_FILE}" 2>&1
echo "[$(date -u +'%Y-%m-%dT%H:%M:%SZ')] Starting EC2 Spot Preemption Watchdog..."
get_imds_token() {
curl -s -X PUT "http://169.254.169.254/latest/api/token" \
-H "X-aws-ec2-metadata-token-ttl-seconds: 60"
}
check_spot_interruption() {
local token
token=$(get_imds_token)
local http_code
http_code=$(curl -s -o /tmp/spot-action.json -w "%{http_code}" \
-H "X-aws-ec2-metadata-token: ${token}" \
http://169.254.169.254/latest/meta-data/spot/instance-action)
if [ "${http_code}" -eq 200 ]; then
return 0
fi
return 1
}
# Polling loop: 5-second interval preserves IMDS rate-limits
while true; do
if check_spot_interruption; then
NOTICE_TIME=$(jq -r .time /tmp/spot-action.json)
ACTION=$(jq -r .action /tmp/spot-action.json)
echo "[$(date -u +'%Y-%m-%dT%H:%M:%SZ')] CRITICAL: Spot Preemption Event Detected!"
echo "Action: ${ACTION} scheduled at ${NOTICE_TIME}. Initiating 120-second drain sequence."
# Stage 1: Notify GitHub Actions Runner Agent to stop accepting work
RUNNER_PID=$(pgrep -f "Runner.Listener" || true)
if [ -n "${RUNNER_PID}" ]; then
echo "Dispatching SIGINT to Runner.Listener (PID: ${RUNNER_PID})..."
kill -SIGINT "${RUNNER_PID}"
fi
# Stage 2: Gracefully stop all active Docker containers
echo "Stopping active Docker containers with 20s grace period..."
ACTIVE_CONTAINERS=$(docker ps -q)
if [ -n "${ACTIVE_CONTAINERS}" ]; then
docker stop --time 20 ${ACTIVE_CONTAINERS} || true
fi
# Stage 3: Flush crash dump logs to S3 diagnostic bucket
INSTANCE_ID=$(curl -s -H "X-aws-ec2-metadata-token: $(get_imds_token)" \
http://169.254.169.254/latest/meta-data/instance-id)
echo "Uploading diagnostic execution trace to S3..."
tar -czf "/tmp/crash-trace-${INSTANCE_ID}.tar.gz" /var/log/ /home/ubuntu/actions-runner/_diag/ || true
aws s3 cp "/tmp/crash-trace-${INSTANCE_ID}.tar.gz" "s3://ci-preemption-dumps/${INSTANCE_ID}.tar.gz" --quiet || true
# Stage 4: Gracefully deregister from GitHub to prevent zombie concurrency lock
echo "Deregistering runner from GitHub Actions pool..."
if [ -f "/home/ubuntu/actions-runner/.runner" ]; then
su - ubuntu -c "cd /home/ubuntu/actions-runner && ./config.sh remove --token $(cat /home/ubuntu/.runner_token) --force" || true
fi
echo "[$(date -u +'%Y-%m-%dT%H:%M:%SZ')] Drain completed successfully. Awaiting instance termination."
exit 0
fi
sleep 5
doneFalsifiability & Chaos Engineering Verification
To validate that the preemption handler operates correctly prior to production deployment, platform teams must perform falsifiability testing:
- Chaos Injection Mechanism: Utilize AWS Fault Injection Simulator (FIS) targeting the runner auto-scaling group using action
aws:ec2:send-spot-instance-interruptionswith a 2-minute duration. - Pass Criteria:
1. The GitHub Actions job exits with an explicit Spot Preemption Handled diagnostic annotation rather than a network timeout.
2. The runner unregisters from GitHub Actions within 45 seconds of the interruption injection.
3. The workflow job automatically re-queues and succeeds on an on-demand fallback runner within 90 seconds.
- Fail Criteria: Any occurrence of a runner lingering in the GitHub Actions runner inventory for greater than 3 minutes, or an uncommitted database migration leaving residual locks on shared integration testing fixtures.
---
Production Configuration Traces: BuildKit Cache & GitHub Actions Workflow
To achieve optimal performance across heterogeneous runners, production architectures employ optimized Docker multi-stage builds coupled with hybrid runner workflow definitions.
Multi-Stage Dockerfile with BuildKit Cache Mounts
The following Dockerfile demonstrates advanced cache mount configurations. Notice the explicit target mounts for Go build artifacts, Node modules, and package manager layers, ensuring that neither local disk wipes nor remote network transfers repeat compiler dependency resolution:
# syntax=docker/dockerfile:1.7-labs
# Multi-stage production container with explicit BuildKit cache targets
# Stage 1: Build Go Backend Services
FROM golang:1.23-alpine AS go-builder
WORKDIR /src
# Utilize cache mounts for Go download and build cache
RUN --mount=type=cache,target=/go/pkg/mod \
--mount=type=bind,source=go.mod,target=go.mod \
--mount=type=bind,source=go.sum,target=go.sum \
go mod download -x
COPY cmd/ ./cmd/
COPY internal/ ./internal/
# Build statically compiled binary with persistent compiler cache
RUN --mount=type=cache,target=/go/pkg/mod \
--mount=type=cache,target=/root/.cache/go-build \
CGO_ENABLED=0 GOOS=linux go build -trimpath -ldflags="-s -w" -o /bin/api-server ./cmd/api
# Stage 2: Build Next.js 15 Frontend
FROM node:22-alpine AS web-builder
WORKDIR /app
ENV NEXT_TELEMETRY_DISABLED=1
ENV NODE_ENV=production
# Mount yarn/npm package cache to prevent network package re-fetches
COPY package.json yarn.lock ./
RUN --mount=type=cache,target=/usr/local/share/.cache/yarn \
yarn install --frozen-lockfile
COPY . .
# Cache Next.js intermediate compilation artifacts across builds
RUN --mount=type=cache,target=/app/.next/cache \
yarn build
# Stage 3: Minimal Distroless Production Artifact
FROM gcr.io/distroless/static-debian12:nonroot
WORKDIR /app
COPY --from=go-builder /bin/api-server /app/api-server
COPY --from=web-builder /app/public /app/public
COPY --from=web-builder /app/.next/standalone /app/
COPY --from=web-builder /app/.next/static /app/.next/static
USER nonroot:nonroot
EXPOSE 8080 3000
ENTRYPOINT ["/app/api-server"]Production GitHub Actions Workflow (`.github/workflows/ci.yml`)
The following production workflow integrates native fallback logic. It routes builds to specialized high-performance runners (RunsOn or Depot) while maintaining automatic fallback to standard GitHub-hosted instances if self-hosted spot capacity is constrained.
name: Enterprise CI Build & Test Matrix
on:
push:
branches: [main]
pull_request:
branches: [main]
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
jobs:
# Job 1: High-Speed Docker Build via Depot or RunsOn
build-and-compile:
name: Build Multi-Stage Monorepo Container
# Hybrid runner selector: RunsOn Spot with fallback to Ubicloud
runs-on: [self-hosted, runson, spot, linux-x64]
timeout-minutes: 25
steps:
- name: Checkout Source Code
uses: actions/checkout@v4
with:
fetch-depth: 2
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
with:
driver-opts: image=moby/buildkit:v0.13.1
- name: Configure Depot Remote Acceleration
uses: depot/setup-action@v1
with:
oidc: true
- name: Build and Push Monorepo Container Layer
uses: depot/build-push-action@v1
with:
context: .
file: ./Dockerfile
push: false
tags: internal-monorepo:${{ github.sha }}
# Combine Depot distributed NVMe-oF cache with GitHub fallback
cache-from: type=depot
cache-to: type=depot
# Job 2: Distributed Integration Test Matrix with Redis & PostgreSQL
integration-test-suite:
name: Integration Tests (${{ matrix.shard }}/4)
needs: build-and-compile
runs-on: ubicloud-standard-4
timeout-minutes: 15
services:
postgres:
image: postgres:16-alpine
env:
POSTGRES_DB: integration_test
POSTGRES_PASSWORD: testpassword
ports:
- 5432:5432
options: >-
--health-cmd pg_isready
--health-interval 5s
--health-timeout 3s
--health-retries 5
redis:
image: redis:7.2-alpine
ports:
- 6379:6379
options: >-
--health-cmd "redis-cli ping"
--health-interval 5s
--health-timeout 3s
--health-retries 5
strategy:
fail-fast: false
matrix:
shard: [1, 2, 3, 4]
steps:
- name: Checkout Code
uses: actions/checkout@v4
- name: Setup Node.js Runtime
uses: actions/setup-node@v4
with:
node-version: 22
cache: 'npm'
- name: Run Sharded Test Pipeline
env:
DATABASE_URL: postgres://postgres:testpassword@localhost:5432/integration_test
REDIS_URL: redis://localhost:6379
SHARD_INDEX: ${{ matrix.shard }}
TOTAL_SHARDS: 4
run: |
npm ci
npm run test:integration -- --shard=${SHARD_INDEX}/${TOTAL_SHARDS}---
Strategic Architectural Decision Framework & Migration Playbook
Selecting the optimal CI/CD compute layer requires mapping organizational constraints—such as security compliance, existing cloud commitments, and engineering team size—against platform characteristics.
flowchart TD
START["Evaluate CI/CD Runner Architecture"] --> Q_HOURS{"Monthly Runner Volume > 20,000 Minutes?"}
Q_HOURS -- No --> GH_HOSTED["GitHub-Hosted Runners\n(Standard / Large Tiers)\nZero Ops, Ideal for Early Stage"]
Q_HOURS -- Yes --> Q_MONO{"Docker Monorepo Image > 10GB or Multi-Arch?"}
Q_MONO -- Yes --> DEPOT["Depot Managed Acceleration\nDistributed NVMe-oF Cache\nNative AMD64/ARM64 BuildKit"]
Q_MONO -- No --> Q_VPC{"Strict Enterprise Compliance / Private AWS VPC Required?"}
Q_VPC -- Yes --> RUNSON["RunsOn (AWS Spot in Dedicated VPC)\nWholesale Compute (~$0.0012/min)\nZero Data Egress via Endpoints"]
Q_VPC -- No --> UBICLOUD["Ubicloud Runners\nDrop-in GitHub YAML Replacement\nCommodity Floor ($0.00125/min)"]Decision Rubric
1. GitHub-Hosted Runners: Best suited for early-stage teams consuming fewer than 15,000 build minutes per month. The operational overhead of managing custom launch templates, AMIs, or cloud networking outweighs compute savings at small volumes.
2. Depot: The preferred solution for teams suffering from build cache latency. If your engineers spend significant time waiting on docker build steps, multi-architecture compilation, or monorepo layer invalidations, Depot delivers immediate returns by accelerating warm builds without requiring custom AWS infrastructure management.
3. RunsOn: The standard for security-sensitive organizations, high-volume CI environments (>50,000 minutes/month), and teams holding significant AWS Enterprise Discount commitments. Running inside your own VPC helps satisfy enterprise compliance objectives (such as SOC 2, HIPAA, and ISO 27001 data isolation controls) by ensuring test code, build artifacts, and staging database secrets remain strictly within your AWS perimeter.
4. Ubicloud: The optimal migration path for engineering organizations seeking an immediate 70% to 80% cost reduction on generic Linux and ARM workloads without modifying infrastructure or configuring AWS IAM. Changing one line in your workflow files (runs-on: ubicloud-standard-4) achieves immediate savings.
For enterprise teams executing comprehensive CI/CD transformations or redesigning cloud-native microservices, partnering with seasoned engineers through custom software engineering services ensures resilient infrastructure architectures, hardened networking boundaries, and automated FinOps enforcement from day one.
---
Frequently Asked Questions
How does security isolation compare between GitHub-hosted ephemeral microVMs and self-hosted AWS Spot runners?
GitHub-hosted runners utilize ephemeral Azure virtual machines that are destroyed upon job completion. Self-hosted AWS Spot runners using RunsOn achieve comparable isolation by operating in --ephemeral mode. Under this configuration, the EC2 instance accepts exactly one job, terminates immediately afterward, and runs within a dedicated, private AWS subnet with zero inbound internet connectivity. Security is further hardened through IMDSv2 enforcement and IAM instance profiles that utilize short-lived STS credentials.
Can RunsOn and Ubicloud natively execute ARM64 builds without QEMU emulation penalties?
Yes. Both RunsOn and Ubicloud support native ARM64 silicon without requiring QEMU emulation. RunsOn provisions native AWS Graviton3 and Graviton4 EC2 instances (c7g, c8g), while Ubicloud provisions native Ampere Altra bare-metal nodes. Because compilation occurs on bare silicon rather than through user-space CPU emulation (qemu-user-static), ARM64 container builds execute without binary translation penalties, significantly outperforming cross-architecture QEMU emulation on x86 runners for compiled languages.
How does Depot eliminate the 10GB GitHub Actions cache limit for enterprise monorepos?
GitHub Actions enforces a hard 10GB storage ceiling per repository across all active workflows. When this quota is reached, GitHub automatically evicts older cache archives regardless of frequency of use. Depot circumvents this by streaming BuildKit cache operations directly to an external distributed Ceph cluster over high-speed NVMe-oF interconnects. Caches are managed on independent, persistent SSD volumes that are shared across concurrent branches and pull requests with zero repository-level caps.
What happens if an AWS Spot instance is preempted during an uncommitted database migration in an integration test?
If an unhandled preemption occurs, database transactions running against an external shared database can leave open connections or locked tables. Implementing the IMDSv2 preemption daemon addresses this failure mode by catching the 120-second termination notice. The script issues a SIGINT to the test runner, executes docker compose down -v to cleanly terminate containerized databases, flushes in-flight logs to S3, and notifies the GitHub API to reschedule the job on an on-demand fallback instance.
---