Kubeshark Tutorial for Talos Kubernetes Cluster

Introduction

Kubeshark is a powerful observability and debugging tool for Kubernetes, often described as “Wireshark for Kubernetes”. This tutorial will guide you through using Kubeshark effectively with your Talos Kubernetes cluster, which includes services like Longhorn, Woodpecker CI, External-DNS, Cert-Manager, and various other components.

Installation

If you haven’t already installed Kubeshark, you can install it using:

# For macOS
brew tap kubeshark/kubeshark
brew install kubeshark
 
# For Linux
curl -Lo kubeshark https://github.com/kubeshark/kubeshark/releases/latest/download/kubeshark-linux-amd64
chmod +x kubeshark
sudo mv kubeshark /usr/local/bin/
 
# For Windows (using PowerShell)
Invoke-WebRequest -Uri https://github.com/kubeshark/kubeshark/releases/latest/download/kubeshark-windows-amd64.exe -OutFile kubeshark.exe

Basic Operations

Starting Kubeshark

Start Kubeshark to monitor all namespaces in your cluster:

kubeshark tap

This will deploy Kubeshark components to your cluster and open the web UI (typically at http://localhost:8899).

Targeting Specific Namespaces

Monitor only specific namespaces in your Talos cluster:

# Monitor Longhorn system
kubeshark tap -n longhorn-system
 
# Monitor multiple namespaces
kubeshark tap -n longhorn-system,external-dns,cert-manager,woodpecker

Targeting Specific Services or Pods

Target specific services in your cluster:

# Monitor Longhorn manager traffic
kubeshark tap --target service=longhorn-manager
 
# Monitor Woodpecker server
kubeshark tap --target service=woodpecker-server
 
# Monitor External DNS
kubeshark tap --target service=external-dns

Target pods with specific labels:

# Monitor Goldilocks dashboard pods
kubeshark tap --target "app.kubernetes.io/name=goldilocks,app.kubernetes.io/component=dashboard"

Working with the Kubeshark UI

Main UI Components

When the Kubeshark UI opens, you’ll see:

Top Bar: Search functionality and filtering options
Left Panel: List of captured traffic entries
Main Panel: Traffic details with multiple tabs
Status Bar: Connection status and statistics

Traffic List

The left panel shows all captured traffic with:

Source and destination services
HTTP method and status code
Timestamp and response time

Detailed Traffic Analysis

Click on any traffic entry to see detailed information:

Overview Tab: Summary of the request and response with service connection path
Request Tab: HTTP method, URL, headers, and request body
Response Tab: Status code, headers, and response body
Headers Tab: All request and response headers
Timeline Tab: Visualization of the request/response cycle timing

Search and Filtering with KFL

Use Kubeshark Filter Language (KFL) in the search bar:

# Filter traffic to the Longhorn backend
dst.name == "longhorn-backend"

# Show only error responses
http.status >= 400

# Filter by HTTP method
http.method == "POST"

# Combine filters
dst.namespace == "woodpecker" && http.status != 200

Practical Examples for Your Cluster

Example 1: Monitoring Longhorn Storage Operations

kubeshark tap -n longhorn-system

In the UI, use these filters to analyze different aspects:

# Volume operations
http.path.contains("/v1/volumes")

# Replica operations
http.path.contains("/v1/replicas")

# Error tracking
dst.name == "longhorn-backend" && http.status >= 400

Example 2: Debugging Cert-Manager and Let’s Encrypt Communication

kubeshark tap -n cert-manager

Look for these patterns in the UI:

# Let's Encrypt ACME communication
dst.ip.regex(".*acme-v02.*")

# Certificate issuance operations
http.path.contains("certificates")

# DNS-01 challenge operations with Cloudflare
dst.ip.contains("cloudflare") && http.path.contains("dns-01")

Example 3: Analyzing Woodpecker CI Pipeline Execution

kubeshark tap -n woodpecker

Filter interesting traffic:

# GitHub webhook traffic
src.ip.external == true && dst.name == "woodpecker-server"

# Pipeline execution
dst.name == "woodpecker-server" && http.path.contains("/api/repos")

# Agent communication
src.name.contains("woodpecker-agent") && dst.name == "woodpecker-server"

Example 4: Examining External-DNS Updates to Cloudflare

kubeshark tap -n external-dns

Use these filters:

# Cloudflare API calls
dst.ip.external == true

# DNS record updates
http.method == "POST" || http.method == "PATCH"

# API token usage
request.headers["Authorization"].exists == true

Advanced Use Cases

Recording and Exporting Traffic

Record traffic for later analysis:

kubeshark tap --recordings.enabled=true

Access recordings from the UI under the “Recordings” section.

Service Map Visualization

Use the Service Map view to visualize service communications:

Start Kubeshark: kubeshark tap
In the UI, click on “Map View” to see visual service connections
Identify high-traffic paths and dependencies

Continuous Monitoring

Set up continuous monitoring with limited resources:

kubeshark tap --resources.limits.worker.cpu=100m --resources.limits.worker.memory=200Mi --hub.persistence.enabled=true

Automating with CLI Scripts

Extract specific information using scripts:

# Extract all URLs from Longhorn API calls
kubeshark tap -n longhorn-system --script 'if dst.name == "longhorn-backend" then return request.url end'
 
# Count requests by status code
kubeshark tap --script 'counts = {}; return function(data) if data.protocol == "http" then counts[data.http.status] = (counts[data.http.status] or 0) + 1; return counts end end'

Troubleshooting Your Cluster

Diagnosing Service Connectivity Issues

If services can’t communicate:

# Monitor both source and destination namespaces
kubeshark tap -n namespace1,namespace2
 
# Look for connection timeouts or errors
http.status >= 500 || response.latency > 5000

Identifying Authentication Problems

For auth issues between services:

# Look for 401/403 responses
http.status == 401 || http.status == 403
 
# Examine auth headers
request.headers["Authorization"].exists == true

Debugging Ingress and TLS Issues

For Traefik ingress or TLS problems:

# Monitor Traefik traffic
kubeshark tap --target "app.kubernetes.io/name=traefik"
 
# Filter for TLS handshake issues
tls.handshake_error == true

Cleaning Up

When you’re done using Kubeshark, clean up the resources:

kubeshark clean

Specific Tips for Talos K8s Cluster

Resource Considerations

Talos nodes can be resource-constrained, so limit Kubeshark resource usage:

kubeshark tap --resources.limits.worker.cpu=100m --resources.limits.worker.memory=200Mi

Security Context

Talos has strict security policies, so you might need to run with elevated privileges:

kubeshark tap --privileged

Node Selection

If your Talos cluster has mixed node types, target specific nodes:

kubeshark tap --selector kubernetes.io/hostname=node1

Going Further

Regular Security Auditing

Run periodic security checks:

# Check for unencrypted credentials
kubeshark tap --query "request.headers['Authorization'].type == 'Basic' && !dst.ip.internal"
 
# Monitor sensitive API operations
kubeshark tap --query "http.path.contains('/secrets') || http.path.contains('/tokens')"

Performance Analysis

Analyze service performance:

# Find slow API endpoints
kubeshark tap --query "response.latency > 1000" --script 'return {url=request.url, latency=response.latency}'
 
# Identify chatty services
kubeshark tap --script 'counts = {}; return function(data) if data.src and data.dst then local pair = data.src.name .. "->" .. data.dst.name; counts[pair] = (counts[pair] or 0) + 1; return counts end end'

Conclusion

Kubeshark is an invaluable tool for monitoring, troubleshooting, and securing your Talos Kubernetes cluster. By leveraging its powerful filtering capabilities and intuitive UI, you can gain deep insights into your cluster’s communication patterns and quickly identify issues.

Use this guide as a reference when working with Kubeshark in your environment, adapting the examples to your specific needs and services.

My Notes

Explorer