10 Critical Insights into How GitHub Leverages eBPF for Safer Deployments
When you run the world’s largest code hosting platform, every deployment carries immense risk. GitHub not only stores millions of repositories but also hosts its own source code on the very platform it serves. This creates a unique circular dependency: if GitHub goes down, deploying fixes becomes impossible without external backups. To break this loop and enhance deployment safety, GitHub turned to eBPF—a powerful kernel technology that monitors and controls system calls. This article dives into ten key things you need to know about their approach, from identifying dependency types to implementing custom eBPF programs.
1. The Self-Hosting Dilemma
GitHub uses its own platform to host its source code—a decision that makes them their own biggest customer. This internal dogfooding ensures that features are tested thoroughly before reaching users. However, it introduces a critical circular dependency: if github.com experiences downtime, developers cannot access the very repositories needed to deploy a fix. To mitigate this, GitHub maintains a code mirror for rapid forward fixes and pre-built assets for rollbacks. Yet, this only solves the first layer of circularity.
2. Beyond the Mirror: Deeper Circular Dependencies
Even with a code mirror, deployment scripts can introduce new circular dependencies. For instance, a script might inadvertently depend on an internal service or download a binary from GitHub itself. During an outage, such actions fail or hang, preventing recovery. These subtle dependencies are often hard to predict and review manually. GitHub’s new host-based deployment system needed to detect and block them in real time, without adding complexity to scripts.
3. Three Types of Circular Dependencies in Deployments
To tackle the problem, GitHub categorized circular dependencies into three types. Direct dependencies occur when a deploy script explicitly fetches something from GitHub (e.g., pulling the latest release of a tool). Hidden dependencies arise from tools already on the machine that check for updates—if the check fails during an outage, the tool may hang. Transient dependencies happen when a script calls an internal API, which in turn depends on GitHub. Understanding these categories helped design a targeted eBPF solution.
4. Previous Manual Review: Pain Points
Before eBPF, GitHub relied on teams owning stateful hosts to manually review their deployment scripts for circular dependencies. This approach was error-prone and unscalable. Engineers had to trace every network call and file access, often missing transient or hidden dependencies. As the number of services grew, the review process became a bottleneck, delaying deployments and increasing risk. A more automated, systematic method was needed.
5. Enter eBPF: A Kernel-Level Guardian
eBPF (extended Berkeley Packet Filter) allows sandboxed programs to run inside the Linux kernel without changing kernel source code or loading kernel modules. GitHub realized they could use eBPF to intercept system calls made by deployment scripts—specifically network connections, file reads, and binary executions. By writing custom eBPF programs, they could selectively monitor and block calls that would create circular dependencies, all with minimal performance overhead.
6. How eBPF Catches Circular Calls
GitHub’s eBPF programs attach to tracepoints and kprobes that trigger on key syscalls like connect(), open(), and execve(). When a deployment script attempts to reach out to github.com or download a binary, the eBPF program checks a set of rules defined for the current deployment context. If the call matches a known circular dependency pattern, the program can block it or log a warning. This happens in kernel space, making it efficient and secure.
7. Implementation on Host-Based Deployments
Within their new host-based deployment system, GitHub integrated eBPF as a safety layer. Each deploy script runs inside a controlled environment where an eBPF program is loaded. The program maintains a map of allowed destinations and file paths. For example, during a MySQL fix, the script is permitted to contact only local services and pre-approved internal mirrors. Any attempt to reach public endpoints or unapproved binaries is instantly blocked, breaking the circular chain.
8. Handling False Positives and Exceptions
No safety system is perfect. GitHub had to handle cases where legitimate calls might be blocked. They implemented a fallback mechanism: if an eBPF program blocks a call, the script receives a clear error message and an option to temporarily override the block via an authenticated API. This allows engineers to approve necessary external dependencies during planned maintenance windows. Over time, these overrides help refine the eBPF ruleset.
9. Performance and Security Considerations
eBPF is designed for high performance, but GitHub still tested its impact carefully. The deployed eBPF programs use minimal instructions and avoid expensive operations like loops. Benchmarking showed less than 1% CPU overhead on deployment hosts. Security is also tight: eBPF programs are verified by the kernel before loading, preventing malicious or buggy code from crashing the system. GitHub also restricts eBPF capabilities using BPF Type Format (BTF) and CAP_BPF.
10. Results and Future Directions
Since rolling out eBPF protection, GitHub has seen a significant reduction in deployment failures caused by circular dependencies. The automated blocking catches issues that would have been missed by manual reviews. Engineers now trust the deployment system to prevent self-inflicted outages. Looking ahead, GitHub plans to extend eBPF monitoring to other parts of their infrastructure, such as container startup and network policies, further hardening their deployment safety.
GitHub’s innovative use of eBPF turns a fundamental architectural challenge into a managed, runtime-supervised process. By embedding safety checks at the kernel level, they can break circular dependencies without slowing down deployments or burdening developers. This approach offers a blueprint for any organization running critical infrastructure that depends on itself. Whether you're at a startup or a large enterprise, eBPF provides a powerful tool to make deployments safer and more resilient.