Configuration Drift Silently Breaks Multi-Hop Chains in sing-box Reality Fleet

A detailed post-mortem from the operators of an RCQ messenger network has exposed how configuration drift in a sing-box and Reality deployment left four out of seven nodes unreachable, while every monitoring check continued to report OK status.

Network Architecture

The fleet on 10 August comprised 14 endpoints running on seven machines hosted by four providers. Clients fetched a signed configuration (version 144) and used urltest to select routes. Since version 0.80, traffic follows a two-hop path: entry nodes know the client but not the destination, while exit nodes know the destination but not the client. This design requires every entry node to maintain an explicit allowlist of exit nodes because each machine ends its firewall rules with a reject policy.

What Went Wrong

Rules extracted from a live entry node showed only five permitted addresses: two islands and two relays. The remaining five nodes in the fleet were absent from the list. Four of those missing nodes belonged to a single provider, immediately limiting viable chains. Further inspection revealed inconsistent lists across machines; only three of the seven published nodes could reach each other. A free user whose entry landed in the first group therefore had just two possible exits, and the onion routing layer routed traffic through only three nodes instead of seven.

urltest never complained because it simply ignored dead chains and selected from the remaining live options. Canary checks verified that relays answered, external probes confirmed islands were alive, and the relay-lockdown.sh --check script only validated local services such as the masquerade host and mirrors. None of these tools compared the allowlist against the full set of relays published in the signed configuration.

Private Nodes and Additional Findings

Examination of two paid private nodes found no route section at all, meaning anyone holding the tenant key could route arbitrary traffic through the rented machine. The relay-lockdown.sh script also surfaced an outdated SNI value of www.apple.com on two machines, left over from an earlier period before operators realized the masquerade name must reside in the same ASN as the server address.

Remediation Steps

The check script was updated to compare the allowlist against relays listed in the signed configuration and to print missing addresses. A hardcoded fallback list grew from three to seven addresses. The script now runs every thirty minutes via cron on all nine machines and only expands the allowlist; narrowing remains a manual operation. Every new configuration is validated with sing-box check before deployment.

The root cause remains unknown because lists were never dated and drift leaves no log entries. The warning already present in the script header proved accurate: a snapshot locked down from one branch silently rejects every node added later.