# udptunnel – UdK Berlin ⇄ Folkwang Essen UDP broadcast bridge

```
 Folkwang LAN 192.168.1.0/24                 amsel.udk.ai                 UdK LAN 192.168.1.0/24
 sensors ──bcast :3000-5999──▶ edge(site 2) ══UDP══▶ broker :40000 ══UDP══▶ edge(site 1) ──bcast :3000-5999──▶ receivers
 receivers ◀──bcast :6000-8999── edge(site 2) ◀══════ broker ◀══════ edge(site 1) ◀──bcast :6000-8999── sensors
```

In the default **P2P mode** the broker is only the meeting point. It tells each edge the other's public address,
the edges punch through their NATs, and data then flows **directly between Berlin and Essen**:

```
 edge(site 2) ──ping──▶ broker ◀──ping── edge(site 1)      1. rendezvous: broker reports each edge's public ip:port
 edge(site 2) ◀════════ direct UDP ═══════▶ edge(site 1)   2. hole punch, then all data goes direct
```

If no direct path can be made (e.g. a symmetric NAT or a strict firewall), or it breaks later, traffic falls back
to the broker relay automatically, and switches back to direct once punching succeeds again.

* **One binary**, three roles: `udptunnel broker`, `udptunnel edge`, `udptunnel genkey`.
* **Lowest latency:** one sensor datagram becomes one tunnel datagram, sent the moment it arrives. There's no batching,
  queueing or TCP, and no extra threads: each process is a single epoll loop. The edge reads directly
  into the send buffer (zero-copy), and the broker forwards the bytes unchanged. Overhead is 30 bytes per datagram.
  Tunnel packets are marked DSCP EF. Measured locally: about 0.1 ms p50 for the full sensor→edge→broker→edge→receiver path,
  0 loss for 200k packets at 25k pkt/s. In practice the Berlin–Essen network RTT (~10 ms) is the only latency that matters.
* **Robust:**
  * Edges sit behind NAT. They keep the mapping alive with pings every 500 ms, and the broker learns each
    edge's current public address from authenticated packets. If an edge's address or port changes, traffic follows it at once.
  * No pong for 3 s → the edge opens a fresh socket (new NAT mapping) and re-resolves DNS every 3 s until the broker is back.
  * Edges start even when the network or DNS isn't up yet. systemd restarts any crashed process within 1 s.
  * The capture and inject ranges must not overlap; the edge refuses to start if they do. This makes broadcast loops impossible.
  * Ports are shared with SO_REUSEADDR, so other apps on the edge machine can still listen on the same ports.
  * The fd limit is raised automatically (1 socket per port, 3000 sockets per edge).
  * `--dup N` on edge and broker sends every packet N times per leg, for lossy links. Duplicates are discarded.
* **Broker-independent once connected (P2P):** an established direct path keeps working if the broker goes down.
* **Secure:** every packet carries an HMAC-SHA256 tag from a shared key. A sliding replay window blocks
  replayed or forged packets, so nobody on the internet can inject broadcasts into the campus LANs or hijack
  an edge's address. The payload is authenticated but **not encrypted**.

## Install from packages (recommended)

`dist/` contains ready-made packages (static binaries, no dependencies):

| host | file |
|---|---|
| Debian / Ubuntu, x86-64 | `udptunnel_1.1.0_amd64.deb` |
| Debian / Raspberry Pi OS 64-bit | `udptunnel_1.1.0_arm64.deb` |
| Raspberry Pi OS 32-bit | `udptunnel_1.1.0_armhf.deb` |
| macOS (Intel + Apple Silicon) | `udptunnel-1.1.0-macos-universal.tar.gz` |

Every host also needs the shared key. It's on amsel at `~/.config/udptunnel/key`; copy it with `scp`, not by e-mail.

**Debian edge:**
```sh
sudo apt install ./udptunnel_1.1.0_amd64.deb
sudo udptunnel-setup folkwang --key-file udptunnel.key      # or: udk
sudo udptunnel-setup status          # or: logs | stop | start
```
Options: `--mode p2p|relay|both`, `--bcast 192.168.1.255`, `--dup N`, `--broker host:port`.
It runs as systemd service `udptunnel-edge` under its own unprivileged user. Config lives in `/etc/udptunnel/`.

**macOS edge:**
```sh
tar xzf udptunnel-1.1.0-macos-universal.tar.gz && cd udptunnel-1.1.0-macos
sudo ./install.sh udk --key-file ../udptunnel.key            # or: folkwang
sudo ./install.sh status             # or: logs | stop | start | uninstall
```
It installs a launchd daemon (`ai.udk.udptunnel.edge`) that starts at boot and restarts on failure. The daemon runs at
interactive priority (no App Nap throttling) and keeps the Mac from sleeping (`--allow-sleep` turns that off).
If the macOS firewall is on, the installer adds an exception. The log is `/var/log/udptunnel-edge.log`, rotated automatically.

Rebuild all packages with `packaging/build.sh`. It needs the rustup targets `x86_64/aarch64-unknown-linux-musl`,
`armv7-unknown-linux-musleabihf`, `x86_64/aarch64-apple-darwin`, plus `cargo-zigbuild` 0.20 and zig 0.13
(`pip install ziglang`).

## Build

```sh
cargo build --release            # -> target/release/udptunnel   (Rust ≥ 1.70)
```
Build on each edge machine, or copy the x86-64 Linux binary. It runs on Linux and macOS; Windows builds should work but are untested.

## Install

```sh
udptunnel genkey > key           # once; copy the SAME key to the broker and both edges
sudo install -m755 target/release/udptunnel /usr/local/bin/
sudo install -d /etc/udptunnel && sudo install -m600 key /etc/udptunnel/key
```

**Broker (amsel.udk.ai):**
```sh
sudo cp deploy/udptunnel-broker.service /etc/systemd/system/
sudo ufw allow 40000/udp
sudo systemctl daemon-reload && sudo systemctl enable --now udptunnel-broker
journalctl -fu udptunnel-broker
```

**Edge in Folkwang** (use `edge-udk.env` in Berlin):
```sh
sudo cp deploy/udptunnel-edge.service /etc/systemd/system/
sudo cp deploy/edge-folkwang.env /etc/udptunnel/edge.env      # edit if needed
sudo systemctl daemon-reload && sudo systemctl enable --now udptunnel-edge
journalctl -fu udptunnel-edge
```

Manual run, for example while testing:
```sh
udptunnel edge --broker amsel.udk.ai:40000 --site 2 --key-file key --capture 3000-5999 --inject 6000-8999
```

## Modes (`--mode`, set `MODE=` in `edge.env`)

| mode | data path | when to use |
|---|---|---|
| `p2p` (default) | direct if a direct path is up, otherwise relay through the broker | normal operation: lowest latency, broker load ≈ 0 |
| `relay` | always through the broker | P2P not wanted, or firewall policy requires one fixed endpoint |
| `both` | direct **and** relay at the same time; the first copy wins, the duplicate is dropped | maximum robustness: a failing path causes zero loss; costs 2× uplink bandwidth |

How P2P works:
1. Every 500 ms each edge pings the broker. The broker's reply lists the other edges with their
   public `ip:port` as the broker sees them.
2. Both edges send authenticated probes to each other **from the same socket**, so the NAT mapping the broker saw is the one
   the peer aims at. A reply proves the direct path works (`DIRECT path to site N UP`). Data then bypasses the broker.
3. Punching happens in windows aligned to the wall clock, so both sides punch at the same time.
   * The first second uses only low-TTL packets (2..`--punch-ttl`, default 5). They open the local NAT but expire before
     reaching the remote NAT. This avoids a well-known conntrack race that makes Linux-based NATs pick a different port.
   * Next comes 9 s of normal punching, then 35 s of silence so any stale NAT state expires. The cycle repeats every 45 s
     until it succeeds. **So the direct path can take up to ~45 s to come up after start**; until then traffic is relayed.
4. A confirmed path is probed every 500 ms. After 1.5 s without a reply it is declared DOWN and data goes through the relay
   (in `p2p` mode, packets sent during those 1.5 s are lost; `both` mode avoids this).
5. While a direct path is up, losing the broker doesn't affect data. The edge keeps its socket (and so its NAT mapping)
   and only re-resolves the broker in the background.

Expected outcomes by network:
* Most NATs and stateful firewalls (endpoint-independent mapping: Linux/OpenWrt/FRITZ!Box/most enterprise firewalls in their
  default UDP mode) → direct. Verified here with two Linux NAT routers, both LANs `192.168.1.0/24`.
* Symmetric NAT on either side (a new public port per destination, e.g. `--random-fully` or some carrier-grade NAT), or a
  firewall that drops all unsolicited UDP → stays on relay. Verified to fall back cleanly. If campus IT can allow inbound
  UDP to the edge machine (or a static port forward), direct works again.
* Both mappings must keep the same public port toward the broker and the peer. That is the case whenever the edge machine's
  NAT is endpoint-independent.

Clocks need NTP, both for the replay filter and so the punch windows of both sites line up (±1 s is fine).

## Edge options

| option | default | meaning |
|---|---|---|
| `--site N` | – | unique edge id 1..255 (UdK = 1, Folkwang = 2) |
| `--capture LO-HI` | – | local ports whose traffic is sent to the other site |
| `--inject LO-HI` | – | ports accepted from the other site and re-broadcast here |
| `--bcast IP` | 192.168.1.255 | broadcast address used for re-broadcasting |
| `--bind IP` | 0.0.0.0 | `0.0.0.0` captures all datagrams on those ports (including 255.255.255.255). `192.168.1.255` captures only subnet broadcasts and ignores unicast sent to the edge machine. |
| `--inject-bind IP:PORT` | 0.0.0.0:0 | source address/port for re-broadcasts (choose interface on multi-homed hosts) |
| `--mode` | p2p | `p2p`, `relay` or `both`, see above |
| `--punch-ttl N` | 5 | highest TTL of the low-TTL punch packets (must be below the hop count to the remote NAT; 0 = off) |
| `--dup N` | 1 | send every datagram N times on each path from this edge (direct and/or to the broker); for relayed traffic set `--dup` on the broker too |
| `--ping-ms` / `--dead-ms` | 500 / 3000 | keepalive interval / broker-lost timeout |
| `--stats S` | 60 | stats line every S seconds (0 = off) |
| `--verbose` | off | log every datagram |

The log shows `CONNECTED`, `LOST`, `peer site N ONLINE/OFFLINE`, `DIRECT path to site N UP/DOWN`, the first packet seen on each port
(`first packet captured on port …`, `first packet injected on port …`) and periodic stats: path per peer (`1:direct 9.8ms` or `1:relay`), packets sent and received direct/relay, and RTT to the broker.

## Field test without sensors

```sh
# Berlin:  python3 tools/udpprobe.py listen 3001
# Essen:   python3 tools/udpprobe.py send 3001 --rate 20
```
`listen` prints each packet, and also the one-way latency when both clocks are NTP-synced.

## Practical notes

* **Put the edge machine on Ethernet if at all possible.** On Wi-Fi, broadcasts are sent at the lowest basic
  rate and are never retransmitted, so a wireless edge loses and delays packets twice: once receiving the sensor
  broadcast and once re-broadcasting. If the edge must be on Wi-Fi, use `--dup 2` on the edges and the broker.
* Receivers see re-broadcast packets coming **from the edge machine's IP** (and its source port), not from the
  original sensor. Apps that reply by unicast to the sender won't get replies through the tunnel.
  Only broadcast/listen traffic is bridged.
* Keep the clocks roughly right (NTP). The replay filter tolerates clock steps and heals itself within 10 s.
* Max datagram size is 65 477 bytes. Anything above about 1 400 bytes is IP-fragmented on the internet path.
* Several sites work too: give each a unique `--site`. The broker forwards every edge's traffic to all other edges.