By Faiyaz Rahman
Last week, forker managed to run an isolated process. It had its own hostname, its own list of processes, and its own view of the filesystem. But when I tried to reach it from the host using curl, I got nothing. The network was a sealed box with no way in or out.
This week, I finally built the door.
What is the actual problem?
When the Linux kernel creates a new network namespace, it gives that process a completely blank slate. There is one "loopback" interface (the standard 127.0.0.1 you use for local testing), but even that is turned off by default. There are no other ways to talk to the machine. No routes to the outside world. No way for anything on your host to find the process.
Think of it like having a brand new laptop in a room with no WiFi and no ethernet port. The machine works perfectly fine on its own, but it cannot talk to anyone.
I needed to find a way to plug that laptop into the rest of the world. In the world of containers, we do this using two main tools: a bridge and a veth pair.
The bridge: A virtual network switch
A bridge is essentially a software-only network switch. It lives entirely inside the kernel, so there is no physical hardware involved. You create it on your host machine and give it an IP address. From that point on, any virtual cable you plug into it can talk to any other cable on that same bridge. It is very similar to how all the devices in your house connect to your home router.
I called my bridge forker0. It starts up automatically whenever forker runs.
// runtime/network.go
const (
bridgeName = "forker0"
subnetCIDR = "10.200.0.1/16" // The host's address on our virtual network
subnet = "10.200.0.0/16" // The range of addresses available for sandboxes
)
func initNetwork() error {
// If the bridge doesn't exist, we create it
if !linkExists(bridgeName) {
if err := run("ip", "link", "add", bridgeName, "type", "bridge"); err != nil {
return err
}
}
// Turn the bridge "on"
if err := run("ip", "link", "set", bridgeName, "up"); err != nil {
return err
}
// Assign the host its identity on this network
_ = runQuietly("ip", "addr", "add", subnetCIDR, "dev", bridgeName)
}After this runs, the host machine is sitting at address 10.200.0.1 on this new virtual switch. Now we just need to plug some things into it.
The veth pair: A virtual ethernet cable
A "veth pair" is exactly what it sounds like: two virtual network interfaces that are permanently wired together. You always create them as a pair. Whatever data you send into one end immediately comes out the other end. It is like an invisible ethernet cable that exists only in your computer's memory.
Here is the plan: we keep one end of the cable on the host and plug it into our forker0 bridge. We then take the other end and "push" it into the sandbox's private world. Inside the sandbox, that end shows up as eth0, which becomes the sandbox's primary way to talk to the world.
Host Side Sandbox Side
──────────────────────────────────────────────────
forker0 (Bridge, 10.200.0.1) eth0 (10.200.0.60)
│ │
veth-a ─────── kernel ────── [veth peer]
Once that is wired up, we assign the sandbox side an IP address like 10.200.0.60. Now the host and the sandbox are on the same virtual network. They can finally see each other, ping each other, and send data back and forth. It is just like two laptops sharing the same WiFi.
func setupVeth(sandboxID string, pid int) error {
vethHost := "veth-" + sandboxID[len(sandboxID)-4:]
vethPeer := "eth0"
ip := allocateIP() // Grabs an available address like 10.200.0.60
// 1. Create the pair on the host
run("ip", "link", "add", vethHost, "type", "veth", "peer", "name", vethPeer)
// 2. Plug the host end into our bridge and turn it on
run("ip", "link", "set", vethHost, "master", bridgeName)
run("ip", "link", "set", vethHost, "up")
// 3. Hand the other end to the sandbox (using its Process ID)
run("ip", "link", "set", vethPeer, "netns", fmt.Sprintf("%d", pid))
// 4. Set up the connection inside the sandbox
execInSandbox(sandboxID, "ip", []string{"addr", "add", ip + "/16", "dev", "eth0"})
execInSandbox(sandboxID, "ip", []string{"link", "set", "eth0", "up"})
// 5. Tell the sandbox: "If you don't know where to send a packet, send it to 10.200.0.1"
execInSandbox(sandboxID, "ip", []string{"route", "add", "default", "via", "10.200.0.1"})
return nil
}That last step is crucial. It sets the "default route," which tells the sandbox that the host is its gateway to the rest of the universe.
The "Chicken and Egg" problem
I ran into a tricky timing issue here. You have to set up the cable from the host side, but you can only do that after the sandbox process already exists. This is because you need the Process ID (PID) of the sandbox to know where to send the cable end.
However, the sandbox cannot actually start its work (like running a web server) until the cable is plugged in and the network is ready. If it starts too early, it will try to use a network that doesn't exist yet and crash.
So, I had to create a simple handshake:
- The parent spawns the child process.
- The child does its initial setup and then writes a "ready" file to the disk.
- The parent waits for that file, then plugs in the network cables.
- The parent then signals the child that the network is ready.
- The child finally starts the actual application.
It is a simple back-and-forth using files on the disk. It might not be the most elegant solution, but it is reliable and easy to understand.
Reaching the internet with NAT
Even with the bridge working, a sandbox still cannot reach google.com easily. This is because its internal address (like 10.200.0.60) is private. If it sends a request to the internet, the internet won't know how to send a reply back to that specific address.
We fix this with NAT (Network Address Translation). It works like a mail room in a large building. When a sandbox sends a request to the outside world, the host replaces the sandbox's private return address with the host's own public address. When the reply comes back, the host remembers who asked for it and forwards it to the right sandbox.
To make this work, we need to tell the host two things. First, it needs to be willing to forward data that isn't meant for it.
os.WriteFile("/proc/sys/net/ipv4/ip_forward", []byte("1"), 0644)Second, we add a rule to the host's firewall to handle the address translation (this is called "masquerading").
runQuietly("iptables", "-t", "nat", "-A", "POSTROUTING",
"-s", subnet, "!", "-o", bridgeName, "-j", "MASQUERADE")This rule essentially says: "If you see data leaving this machine that came from our virtual network, give it my return address so the internet can reply."
What this actually looks like
no output yet
Now we have two sandboxes on the same bridge. They can talk to the host, they can talk to each other, and they can both reach the internet.
From the host, I can now do this:
$ curl 10.200.0.60:8080
Hello from sandbox-aAnd from inside sandbox-a, I can reach its neighbor:
$ ping 10.200.0.106
64 bytes from 10.200.0.106: time=0.4 msThis was the moment the sandbox stopped feeling like a toy and started feeling like a real tool.
What is missing?
While this is great, it is still very manual. I am reaching sandboxes by their IP addresses directly. In a real Kubernetes cluster, you almost never do that because sandboxes (Pods) are constantly being created and destroyed. Instead, you use a Service that provides one stable address for a group of Pods.
There is also no "port forwarding" yet. I cannot just go to localhost:8080 on my host and reach a sandbox. I have to use the specific 10.200.0.x sandbox ip address.
What is next?
Right now, forker is a bit tied up. When I run a sandbox, the process stays in my terminal until I stop it. If I want to run three different servers, I need three different terminal windows.
Next week, I am going to build a daemon. This will be a background process that manages all our sandboxes for us. You will be able to run a simple command like forker ps to see everything that is running, and the daemon will handle all the networking and life cycles in the background.
See you in week 4!