Skip to main content

Runtime Topology: YU11-aeron-replication

Entrypoints​

EntrypointTransportConsumer
order-matcher:18110HTTP REST/UIunchanged YU10 clients
order-matcher:18130FIX 4.4 over TCPunchanged YU10 FIX initiators
data UDP portsAeron manual unicast MDClocal Archive recording and peer replication follower
ACK UDP portAeron reliable unicastpeer order-matcher replication primary
control UDP portAeron reliable unicastpeer handshake, heartbeat, replay/snapshot control
Archive control/replay UDP portsAeron Archive protocolpeer sidecar/application during catch-up

Components​

  • order-matcher application (2 StatefulSet replicas in HA): retains REST, FIX, input/output rings, primary journal, matching/risk, NATS rollback transport, Lease election, and readiness. Adds SBE codecs, Aeron replicator/follower/ACK agents, exact journal watermark mapping, shadow comparison, transport/policy selection, Archive recovery coordinator, and fast-witness gate.
  • aeron-replication-sidecar (one per order-matcher pod): Java Archiving Media Driver in shared threading mode. The media directory is a memory-backed shared emptyDir; Archive catalog/ segments use the pod's persistent volume.
  • NATS/JetStream: remains the non-replication message bus and File-backed rollback transport. Fast-witness mode also uses the TRADERX_BLP_FAST_WITNESS KV bucket for an atomic promotion claim.
  • Kubernetes Lease: remains the synchronous default promotion authority and asynchronously reconciles fast-witness winners.
  • All YU10 services and databases retain their inherited topology.

Networking​

  • The primary's manual MDC publication sends one claimed frame to two unicast destinations: its local Archive on UDP 40127 and the peer follower on UDP 40123. This preserves one Aeron session and position space for Archive replay-to-live merge; MDC here does not use IP multicast.
  • ACK, control, and Archive replay are reliable unicast UDP between stable StatefulSet ordinal DNS names on the headless order-matcher Service.
  • A namespace-scoped NetworkPolicy permits the named UDP ports only between app=order-matcher pods. No Aeron port uses ingress-nginx, LoadBalancer, NodePort, IP multicast, or a host mapping.
  • Compose uses explicit primary/follower service names on one Docker network.
  • Kind uses a dedicated named multi-node cluster with at least two workers; the ordinary shared single-node cluster is not modified.
  • GKE required anti-affinity keeps each application+sidecar pair on a different c2 node.

Startup / Health Order​

  1. The sidecar opens its Aeron directory, validates/opens the Archive catalog, recording path, schema checksum, and disk watermark, then reports healthy.
  2. Each application completes inherited journal/snapshot recovery and loads the persisted replication checkpoint.
  3. Peers authenticate cluster/pod/ordinal/transport/schema/epoch through the signed control handshake.
  4. The follower catches up through retained journal plus Archive replay; empty-volume recovery installs a complete snapshot bundle first.
  5. Journaled and applied watermarks converge at the observed live high watermark.
  6. Default mode acquires/confirms the Kubernetes Lease; fast mode also requires an atomic witness revision. Only then does the primary admission fence open.

Degraded Behavior​

ConditionBehavior
Follower disconnects under degraded-soloPrimary stops claiming synchronous follower durability, alerts, continues against its journal, and retries authenticated catch-up.
Follower disconnects under strictAdmission closes; new orders receive 503; already-sequenced outcomes remain ambiguous under inherited retry rules.
Aeron offer backpressureBounded retry/backoff; pressure reaches the input ring; timeout transitions through the selected failure policy; no record is dropped.
Schema/transport/epoch mismatchSession rejected; both pods remain non-serving for that pair.
Sequence/checksum gapFollower remains unready and unpromotable; strict primary refuses; shadow mode records a failed comparison.
Sidecar restart/catalog discontinuityApplication refuses replication readiness, reconnects, validates position continuity, and catches up before live state.
Archive disk watermark/fullRecording/catch-up refuses before unsafe exhaustion; diagnostics expose free bytes and affected position.
Empty follower volumeChecksummed snapshot bundle install plus Archive replay; any validation failure remains unready.
Fast peer silence, witness availableOne atomic witness winner promotes and then reconciles the Lease.
Fast peer silence, witness unavailable/ambiguousNo promotion; readiness remains false.
Foreign witness/epoch/confirmed Lease proofLocal primary closes admission and demotes before another order.