High availability

Forming a cluster

The proxy switches to cluster mode as soon as CLUSTER_MEMBERS or CLUSTER_DNS is set. Every node then accepts every request, and routes it consistently to the node in charge of its bucket, based on its hash.

All nodes should reach each other over the LAN. Inside a container, set CLUSTER_ADVERTISE_ADDR if the nodes cannot see each other.

When a node goes down

For a short while, a failed node is not yet marked as such and still receives requests. When they fail, the proxy returns a mocked 429 telling the client to wait one second, with the generated-by-proxy header. Your library therefore retries without anything special; to apply your own backoff, look for that header.

When a node joins or leaves, the hash table changes and 429s can rise for a moment, until the cluster settles. To deploy, replace nodes one at a time, ideally with the replacement already up.

Protocol

The cluster relies on SWIM, through HashiCorp's memberlist. It is an AP system: it needs no quorum and tolerates a network partition. On a partition, you get two independent clusters, which may or may not suit you: configure your network accordingly.

Global limit

A bot's global limit is handled by a single cluster node, with no leader and no election: if that node leaves, the cluster simply picks another. It is a bottleneck that can lengthen tail latency, but the alternatives required external storage or a quorum. Webhooks and requests with no token skip this mechanism.

Bearer tokens

Bot queues live as long as the proxy. Bearer token queues are kept in an LRU cache (see MAX_BEARER_COUNT) and spread by the token's hash rather than the path's, to balance them better across nodes. Their global limit is always handled locally.

Targeting a node

For troubleshooting, add the nirn-routed-to header to a request, with any value: the node that receives it handles it itself instead of routing it.