TCP Transport — RingLoom

ringloom_tcp is the broker-to-broker transport layer. It is a standalone Zig module that provides connection management, handshakes, message framing, and platform-specific non-blocking I/O backends.

The service runtime does not depend on ringloom_tcp. Services write remote messages into their local broker’s send ring. The broker uses ringloom_tcp to move those messages between broker nodes.

Responsibilities

Responsibility	Description
Platform I/O	Use `io_uring` on Linux and `kqueue` on macOS/BSD-style platforms behind a common API.
Connection lifecycle	Listen for incoming peer connections, initiate outbound connections, reconnect with backoff, and track connection state.
Handshake	Validate peer node ID, target node ID, protocol version, direction, group identity, and session epoch.
Framing	Convert TCP byte streams into bounded length-prefixed frames and handle partial reads/writes.
Socket configuration	Apply options such as non-blocking mode, reuse, no-delay, and configured send/receive buffer sizes.

Layering

The module is organized in three layers:

Layer	Key concepts
Platform backend	`IoUringEngine`, `KqueueEngine`, socket wrappers, completion harvesting.
I/O interface	`ConnectionHandle`, `Completion`, submit/harvest operations.
Transport API	`TcpTransport`, `ConnectionManager`, `FrameReader`, `FrameWriter`, handshake and frame types.

The root module selects the default engine at compile time based on the target platform:

Linux uses io_uring;
macOS and FreeBSD use kqueue;
unsupported platforms fail at compile time for the TCP transport.

Frame format

Broker-to-broker application and admin traffic is carried in a 24-byte frame header followed by payload bytes. The header includes:

Field	Purpose
`frame_length`	Total frame size: header plus payload.
`flags`	Heartbeat/admin/data markers and future extensions.
`source_node_id`	Broker node that sent the frame.
`target_node_id`	Broker node that should receive the frame.
`source_service_id`	Service ID that originated the application message, or broker/admin source.
`target_service_id`	Destination service ID for application frames.
`template_id`	Application or admin message type.
`correlation_id`	Request/response or tracing correlation.

The frame header keeps broker-to-broker routing simple: there is no stream multiplexing and no application-level reliable transport layered on top of TCP.

Handshake

Every peer connection starts with a fixed-size handshake. The receiver validates:

magic value;
protocol version;
source node ID;
expected target node ID;
connection direction;
group-name hash;
session epoch;
reserved bytes.

The session epoch lets brokers distinguish a fresh peer process from stale connections after restart. A group mismatch prevents accidental cross-cluster traffic when multiple RingLoom deployments share hosts or networks.

Sender integration

The broker sender event loop drains records from the broker send ring. For each remote target, it finds the peer connection, enqueues the frame into that peer’s write queue, and submits writes through the transport.

Important operational properties:

writes are budgeted per peer to avoid one peer monopolizing the sender loop;
per-peer write queues smooth short bursts;
disconnected peers cause drops/counters rather than unbounded buffering;
optional per-peer send counters expose pending bytes and connection state to service clients and monitoring tools.

Receiver integration

The broker receiver event loop owns the listener socket and incoming peer connections. It accepts connections, completes handshakes, reads frame headers and payloads, and dispatches complete frames.

Frame categories:

Category	Handling
Application data	Route to the target local service’s message ring.
Admin traffic	Forward to the broker control/cluster path.
Heartbeat	Update peer liveness state.

The receiver follows an always-read model. It keeps reading from sockets and records delivery failures instead of pausing a peer socket when one target service is slow.

Backends

Linux: `io_uring`

The Linux backend batches submissions and completions through io_uring. RingLoom uses it for accept, connect, recv, send, and close operations. Tuning properties include queue depth, completion queue depth, registered buffers, SQPOLL, single-issuer, cooperative task-run, sender CQE batch size, receiver CQE batch size, receive buffer size, and receive buffer count.

macOS / BSD-style: `kqueue`

The kqueue backend uses non-blocking sockets and read/write readiness notifications. It gives the broker the same completion-style interface while mapping to the platform’s event model.

Configuration knobs

Common transport-related broker properties include:

Property	Purpose
`broker.local.host.port`	TCP listener endpoint for this broker.
`broker.member.host.ports`	Peer endpoints in `nodeId@host:port` format.
`broker.max.frame.length`	Upper bound for frame validation.
`broker.peer.write.queue.capacity`	Per-peer outbound queue capacity.
`broker.tcp.send.buffer.size`	OS send buffer size.
`broker.tcp.recv.buffer.size`	OS receive buffer size.
`broker.heartbeat.interval.ms`	Peer heartbeat send cadence.
`broker.heartbeat.timeout.ms`	Peer heartbeat timeout.
`broker.reconnect.initial.delay.ms`	Initial reconnect backoff.
`broker.reconnect.max.delay.ms`	Maximum reconnect backoff.
`broker.io.uring.*`	Linux-specific backend tuning.

Delivery model

RingLoom relies on TCP for in-order byte delivery while a connection is alive. It does not add application-level replay across disconnects. If a peer disconnects, in-flight messages in kernel buffers or broker queues may be lost. Applications that need end-to-end guarantees should use correlation IDs, acknowledgements, timeouts, retries, or domain-level reconciliation.