4  Part III — Make it actually distributed

Five processes on one computer still share one filesystem. Steps 19 to 27 put them on two machines, replicate the message board between them with Syncthing, measure what that costs, and then break a mediator on purpose. By the end, the mediators’ price is a number rather than an opinion.

4.1 Syncthing replicates; it does not deliver

Syncthing keeps a directory replicated between machines. That is all it is asked to do here. It knows nothing about Intent, Status, Event, ROT13 or mediators, and it has no concept of a consumer: no subscriptions, no topics, no delivery guarantees, no notion that anybody is waiting.

The trigger is still ours. Each service runs its own WatchService on a local directory; Syncthing only makes files appear there. That is why none of the three services changed a single line when the system became distributed.

Two consequences follow, and both show up later as observations rather than warnings:

  • replication is asynchronous — the .tmp → .msg rename is atomic on each machine separately, and nothing promises ordering or atomicity between machines;
  • the filesystem is a serious transport with its own strengths: files can be large, replicas survive a reboot, and every message is readable with cat.

4.2 Test one layer at a time

Before any Java touched the replicated folder, the dullest possible test ran first: write hello.txt on one machine, confirm it appears on the other, delete it, confirm the deletion travels back. Both directions worked.

That order matters. When the distributed run later failed, the question “is it replication or is it my program?” had already been answered for the transport, and the search could start one layer up.

4.3 What one keystroke costs

With the sensor on one machine and the mediators, ROT13 and the actuator on the other, one typed line produced four .msg files, one per mailbox:

mailbox content timestamp inside
consoleIn/outbox Hello World 23:02:05.315
rot13/inbox Hello World 23:02:05.315
rot13/outbox Uryyb Jbeyq 23:02:13.471
consoleOut/inbox Uryyb Jbeyq 23:02:13.471

The repeated timestamps are the evidence that two of those four publications are pure copies: the mediators republish the message unchanged, timestamp included. They compute identity(x) = x, as separate distributed processes.

The timing, taken from the four files’ modification times on one machine so that no clock skew enters: the sensor’s file appears at 01:02:05.432, the other three at 01:02:13.460, .504 and .532 — 8.10 s later, and within 72 ms of each other. So the chain’s own work took milliseconds; the eight seconds are Syncthing’s propagation. The same chain, run entirely on one machine, completes in about 24 ms.

4.4 Break a mediator

Stop the first mediator and type two lines. ConsoleIn is alive, ROT13 is alive, ConsoleOut is alive, Syncthing is alive — and the business flow is dead. The two lines pile up in consoleIn/outbox; nothing moves past it. The mediators are part of the availability chain.

Then restart the mediator. Nothing is delivered. FileMessageBoard.watch registers a directory for ENTRY_CREATE and blocks in watcher.take(); it never lists what is already there. The two files are still on disk, in both replicas — so no data was lost. What has no memory is the subscription.

That distinction is worth keeping: the messages survived, the delivery did not. A broker answers this with session state and retained messages; in our own code it would be one line — list the existing .msg files before entering the watch loop — which the material deliberately does not write yet.

4.5 The mediator becomes an ordinary service

Before removing the mediators, one asymmetry gets removed: the services receive their work as messages, while the mediator received its route as shell arguments. So the mediator is given only a name, ConfigurableMediator mediator1, from which it derives one permanent control mailbox, mediators/mediator1/inbox, and the route arrives as a message:

ROUTE
consoleIn/outbox
rot13/inbox

from and to travel in one message on purpose: as two, the mediator would briefly hold a new source with the old destination and forward to the wrong place. A route is one fact, and one fact belongs in one message.

A short-lived orchestrator publishes both routes and exits, so the topology is visible in one place and is not in the data path. Running it after the services, and typing a line before it, shows both halves at once: the early line is stranded, the later one flows.

What improved: the mediator now has an identity, takes configuration through its own inbox, keeps a generic capability, and gets its topology from an orchestrator. What did not improve: the data path. It is still event → mediator → copied event, still a process in the availability chain, still copying every message without adding meaning. And this version accepts exactly one route: waitForRoute(...) returns, then the mediator enters watch(route.from(), …) and never reads its control inbox again. Live rewiring is not promised, because one blocking watch cannot serve a control inbox and a source at the same time — the reason a subscription helper appears later.

4.6 What part III moved

stage where the topology lives in the data path?
step 18 the operator’s command lines the mediators are
step 27 an orchestrator, published as Intent the mediators still are

The question the part ends on is therefore sharp rather than rhetorical: why can’t the service interested in the data simply listen to it itself?