My Notes

GitHub Actions Runner Controller Troubleshooting Guide

4 min read

Overview

This guide covers common issues with GitHub Actions Runner Controller in Kubernetes and how to diagnose and fix them.

Understanding GitHub Actions Runner Architecture

Key Concepts

  • AutoscalingRunnerSet: Manages the overall runner configuration
  • EphemeralRunner: Individual runner registrations with GitHub
  • Pods: Created on-demand when jobs need to run
  • minRunners: Number of runners registered with GitHub (NOT running pods)

Important: Pod Behavior

  • Normal: No pods exist when no jobs are queued
  • On-demand: Pods are created only when GitHub has jobs to run
  • Ephemeral: Pods are destroyed after job completion
  • minRunners ≠ running pods: minRunners: 4 means 4 registered runners, not 4 running pods

Common Issues and Troubleshooting

Issue 1: Stuck/Terminating Pods

Symptoms:

  • Pod stuck in “Terminating” status for extended time
  • FailedKillPod errors in events
  • Timeout errors during container termination

Diagnosis:

# Check pod status
kubectl get pods -n gh-actions-runners-set
 
# Describe the stuck pod
kubectl describe pod <pod-name> -n gh-actions-runners-set
 
# Check logs
kubectl logs <pod-name> -n gh-actions-runners-set --tail=50

Solution:

# Force delete the stuck pod
kubectl delete pod <pod-name> -n gh-actions-runners-set --force --grace-period=0

Issue 2: Failed EphemeralRunners

Symptoms:

  • No pods being created for new jobs
  • failedEphemeralRunners > 0 in AutoscalingRunnerSet status
  • EphemeralRunners stuck in “Failed” status
  • Message: “Pod has failed to start more than 5 times”

Diagnosis:

# Check AutoscalingRunnerSet status
kubectl get autoscalingrunnerset mktech224-github-actions-runner-generic -n gh-actions-runners-set -o json | jq '.status'
 
# List all EphemeralRunners
kubectl get ephemeralrunner -n gh-actions-runners-set -o wide
 
# Check specific EphemeralRunner
kubectl describe ephemeralrunner <runner-name> -n gh-actions-runners-set

Solution:

# Delete failed EphemeralRunners (they will be recreated automatically)
kubectl delete ephemeralrunner -l actions.github.com/scale-set-name=mktech224-github-actions-runner-generic -n gh-actions-runners-set
 
# Verify new ones are being created
kubectl get ephemeralrunner -n gh-actions-runners-set -o wide

if deletion is stuck :

kubectl get ephemeralrunners -n gh-actions-runners-set -o name | xargs -I {} kubectl patch {} -n gh-actions-runners-set -p '{"metadata":{"finalizers":null}}' --type=merge

Issue 3: No Runners Available

Symptoms:

  • GitHub Actions jobs stuck in “Queued” status
  • No EphemeralRunners exist
  • currentRunners: 0 in AutoscalingRunnerSet

Diagnosis:

# Check if AutoscalingRunnerSet exists and is healthy
kubectl get autoscalingrunnerset -n gh-actions-runners-set
 
# Check runner controller logs
kubectl logs -l app.kubernetes.io/name=gha-rs-controller -n gh-actions-runners
 
# Verify GitHub token secret
kubectl get secret github-actions-runner-token -n gh-actions-runners-set -o yaml

Diagnostic Commands Reference

Quick Health Check

# Check overall runner set status
kubectl get autoscalingrunnerset -n gh-actions-runners-set
 
# Check current runners
kubectl get ephemeralrunner -n gh-actions-runners-set -o wide
 
# Check for any running pods
kubectl get pods -n gh-actions-runners-set
 
# Check recent events
kubectl get events -n gh-actions-runners-set --sort-by='.lastTimestamp' | tail -20

Detailed Investigation

# Get AutoscalingRunnerSet details
kubectl describe autoscalingrunnerset mktech224-github-actions-runner-generic -n gh-actions-runners-set
 
# Check runner controller deployment
kubectl get deployment -n gh-actions-runners
# View controller logs
kubectl logs -l app.kubernetes.io/name=gha-rs-controller -n gh-actions-runners --tail=100
 
# Check for resource constraints
kubectl top nodes
kubectl top pods -n gh-actions-runners-set

Step-by-Step Troubleshooting Process

Step 1: Identify the Problem

  1. Check if jobs are stuck in GitHub Actions UI
  2. Verify AutoscalingRunnerSet status
  3. List EphemeralRunners and their states
  4. Look for stuck or failed pods

Step 2: Gather Information

# Quick status overview
kubectl get autoscalingrunnerset,ephemeralrunner,pods -n gh-actions-runners-set
 
# Check for errors in events
kubectl get events -n gh-actions-runners-set --field-selector type=Warning
 
# Review controller logs
kubectl logs -l app.kubernetes.io/name=gha-rs-controller -n gh-actions-runners --since=1h

Step 3: Apply Appropriate Fix

  • Stuck pods: Force delete them
  • Failed EphemeralRunners: Delete them (auto-recreated)
  • Missing runners: Check controller and GitHub token
  • Resource issues: Scale cluster or adjust resource requests

Step 4: Verify Resolution

# Wait for new runners to be created
kubectl get ephemeralrunner -n gh-actions-runners-set -w
 
# Test with a simple workflow
# Trigger a GitHub Actions job and verify pod creation
 
# Monitor pod lifecycle
kubectl get pods -n gh-actions-runners-set -w

Prevention and Best Practices

Monitoring

  • Set up alerts for failed EphemeralRunners
  • Monitor pod termination times
  • Track job queue times in GitHub

Maintenance

  • Regularly clean up old failed runners
  • Update runner images periodically
  • Monitor cluster resource usage

Configuration

  • Set appropriate resource requests/limits
  • Configure proper storage classes for ephemeral volumes
  • Use node selectors if needed for runner placement

Expected Behavior vs Issues

✅ Normal Behavior

  • No pods when no jobs are queued
  • Pods created quickly when jobs start
  • EphemeralRunners in “Pending” status when idle
  • Clean pod termination after job completion

❌ Problem Indicators

  • Pods stuck in “Terminating” for >5 minutes
  • EphemeralRunners in “Failed” status
  • Jobs queued but no pods created
  • Frequent pod creation failures

Emergency Recovery

If runners are completely broken:

# 1. Delete all failed runners
kubectl delete ephemeralrunner --all -n gh-actions-runners-set
 
# 2. Restart the runner controller
kubectl rollout restart deployment -l app.kubernetes.io/name=gha-runner-scale-set-controller -n actions-runner-system
 
# 3. If needed, recreate the runner set
helm uninstall <runner-set-name> -n gh-actions-runners-set
# Then reinstall with helm/flux

Remember: GitHub Actions Runner Controller is designed to be ephemeral and self-healing. Most issues resolve automatically when problematic resources are deleted and allowed to recreate.

Graph View