Implement eBPF to Prevent Circular Dependencies in Deployments: A Step-by-Step Guide

Introduction

Every deployment system is vulnerable to circular dependencies—situations where a deployment script inadvertently relies on a service that it's supposed to be fixing. At GitHub, we discovered that our own deployment scripts could create dependencies that would fail during an outage, preventing us from restoring service. By using eBPF (extended Berkeley Packet Filter), we were able to monitor and selectively block these dangerous calls without affecting legitimate operations. This guide walks you through the same approach, from identifying risky patterns to writing and deploying your own eBPF programs.

Implement eBPF to Prevent Circular Dependencies in Deployments: A Step-by-Step Guide — Source: github.blog

What You Need

A Linux system with kernel version 4.18 or later (eBPF support required)
The BCC (BPF Compiler Collection) toolkit or libbpf with a compatible C compiler
Access to your deployment scripts and infrastructure (for testing)
Basic understanding of Linux system calls and network filtering
A non-production environment for initial testing
Optional: An IDE or editor for writing eBPF programs, and bpftrace for debugging

Step-by-Step Instructions

Step 1: Identify Circular Dependency Types

Before writing any code, you need to understand the types of circular dependencies that could affect your deployments. Based on GitHub’s experience, these fall into three categories:

Direct dependencies: Your deployment script calls an external service (like fetching a binary from GitHub) that is currently broken or unavailable during the incident.
Hidden dependencies: A tool on the machine checks for updates or contacts a remote service before completing its task, even if that check isn’t strictly required.
Transient dependencies: Your script calls an internal API (e.g., a migration service) that in turn fetches something from an external source, creating a chain of failure.

Analyze your current deployment scripts and list every external call—HTTP requests, binary downloads, API invocations—that could fail. Mark those that are not essential for the core operation. This list will form the basis of your eBPF policy.

Step 2: Define a Deployment Safety Policy

Create a clear policy that separates allowed and blocked calls during a deployment. For example:

Allow connections to internal databases and config servers.
Block any HTTP requests to external hosts (like GitHub.com) that are not part of the deployment’s mandatory path.
Allow local disk reads and writes from pre-vetted tools only.
Block any attempt to download new binaries from the internet during a critical rollback.

Document these rules in a simple text file or YAML format. You will translate them into eBPF filters later.

Step 3: Write an eBPF Program to Monitor System Calls

eBPF attaches to system calls like connect, sendto, and open. Use the BCC Python bindings to write a monitor that captures all outbound network connections and file opens from your deployment process. Here’s a minimal example in Python that watches connect syscalls:

from bcc import BPF
bpf_text = """
int kprobe__sys_connect(struct pt_regs *ctx, int fd, struct sockaddr __user *usockaddr, int addrlen) {
    // Extract destination IP and port
    // Log to trace_pipe
    return 0;
}
"""
b = BPF(text=bpf_text)
b.trace_print()

Replace the comment with actual logic to read the socket address structure. Test this monitor on your deployment script in a sandbox. Log every outbound connection and note which ones are suspicious.

Step 4: Implement Blocking Logic for Unauthorized Calls

Once you have a list of dangerous patterns, add blocking actions to your eBPF program. For connect syscalls, you can return a negative error code (-EPERM or -EACCES) to deny the connection. For example:

int kprobe__sys_connect(struct pt_regs *ctx) {
    // Retrieve destination address
    // If address matches blocked list (e.g., GitHub API)
    if (blocked) {
        return -EPERM;  // Permission denied
    }
    return 0;
}

For file opens, you can also block attempts to download new files from specific directories (like /tmp/github_downloads). Use a lookup table of blocked IPs and hostnames, updated from your policy file. Test that legitimate calls (to internal services) still succeed.

Step 5: Integrate the eBPF Program into Your Deployment System

Attach the eBPF program to the deployment process using exec_event or perf_event to filter only the deployment script’s PID. For example, in BCC:

b.attach_kprobe(event="sys_connect", fn_name="kprobe__sys_connect", pid=\)

Wrap your deployment command with a wrapper script that loads the eBPF program before executing the actual deploy logic. Use bpf_get_current_pid_tgid() to ensure only the deployment process is affected. This integration should be seamless—team members don’t need to change their tools.

Step 6: Test in a Non-Critical Environment

Run your deployment in a staging environment that mimics production (including simulated outages). Verify that:

The deployment succeeds when all dependencies are available.
The deployment fails gracefully when a blocked call is attempted (returns an error, but doesn’t hang).
Legitimate operations like reading config files or connecting to internal databases are not blocked.
Logs clearly show which calls were blocked and why.

Iterate on the blocked/allow list based on observed false positives.

Step 7: Deploy to Production with Monitoring

Gradually roll out the eBPF-based protection to a small subset of production hosts first. Monitor system performance (CPU overhead from eBPF is usually minimal, below 1%). Use the trace output to detect any new circular dependencies that were missed during testing. Set up alerts for any blocked calls that should have been allowed. After a week, expand to all hosts.

Tips for Success

Start with monitoring only: Before blocking any calls, run your eBPF program in monitor mode for a few days to understand the full dependency graph of your deployments.
Use bpftrace for quick verification: If you need to test a single filter rule quickly, bpftrace is faster than writing a full BCC program.
Handle transient failures gracefully: Your deployment scripts should have retry logic. eBPF blocking returns an immediate error, so ensure your scripts don’t hang on retry.
Keep your allowed list lean: Only whitelist absolutely essential endpoints. Anything else should be blocked by default. You can always add more later.
Document the eBPF policy: Share the blocked/allowed list with your team so everyone understands what is protected. Update it whenever deployment scripts change.
Monitor kernel compatibility: eBPF features evolve. Use tools like bpftool to check your kernel’s capabilities before deploying.

By following these steps, you can eliminate the most dangerous circular dependencies from your deployment system—just as GitHub did. eBPF gives you fine-grained control without modifying your scripts, making your infrastructure more resilient to real-world outages.