11 KiB
Conversation Log — Part VIII — The Filesystem Has Done Its Job · Step 53, and Part IX — MQTT · Steps 54– (turns 84–)
⬅ Previous: Part VII · Index · Next: (not started yet) ➡
All parts: Setup & Step 0 · Part I · Part II · Part III · Part IV · Part V · Part VI · Part VII · Parts VIII & IX
[84] Student
Filesystem was great: to apprehend the mehtodology, manage huge amount of file and create backup (if versionning is setup), but is clearly too long for short message, detect the changes (need a scan each times), heavy when the message must be shared with multiple subscribers
[84] Tutor — reply
Step 53's own list, matched — and the versioning caveat was spotted independently: the material says replication alone is not a backup strategy, because deletion and corruption replicate too. What the book adds: the filesystem's advantage MQTT cannot match (gigabytes to terabytes, where chunking a huge payload over MQTT means inventing a file-transfer protocol inside a message protocol), hence "use messages for coordination and events, use a file or object store for large data, and let a message carry the location, identity and metadata"; the full list of what the transport taught (including independent languages); and the accumulated machinery — directories, .tmp, atomic moves, .msg, WatchService, rescanning, shared paths, Syncthing, delay — behind "the architecture is getting clean, the transport is increasingly ridiculous". The student's "heavy for multiple subscribers" is the Step 3 queue-versus-topic limitation resurfacing as a transport problem.
[85] Student
I don't expect it to change the logic behind the architecture, i.e. the event driven with the subscription to topic and so on, i instead expect it to change how the subscription and mailling works, by replacing the file system with a lightweighten "software/process" that handle short message
[85] Tutor — reply
Every row of Step 54's mapping table, predicted before reading it: board root → broker/namespace, directory → topic, .msg → publication, contents → payload, WatchService → subscription, dynamic directory → topic filter, control/state/event directories → I, S/…, E/…. The section's punchline is that reaction itself: "Oh. We already know this architecture." MQTT is a transport change, not an architectural claim. Local tooling checked: mosquitto 2.1.2 with mosquitto_pub/mosquitto_sub already installed on the laptop.
[86] Student
how to install it on my server? i'll need it for my project
[86] Tutor — reply
Gave both routes for the home server (Docker Compose, matching how its other services run, and the native package), with the Mosquitto 2.x trap spelled out: since 2.0 it listens on localhost only and refuses anonymous clients unless told otherwise, so a listener line and an explicit allow_anonymous or a password file are required for the laptop, the phone or the vehicle to connect. Noted that port 1883 is unencrypted and belongs on the home LAN only, never exposed to the internet.
Progress record
- Current step: 55 (Part IX); Part VIII closed
- Last observed result:
mosquitto 2.1.2present on the laptop; the server has nothing yet - Unresolved problem: Part VIII chapter placement (own file or folded into the MQTT chapter)
- Next action: student installs Mosquitto on the server, then runs the Step 55 publish/subscribe experiment
[87]–[108] Student & Tutor — Part IX, steps 55 to 78
Condensed, one line per exchange; the full reasoning is in the chapter 08-part-ix-mqtt.qmd.
- [87] "the line was added to host, and i've tested and MQTT works well" —
/etc/hostsentry added for the server; broker reachable. Verified independently with a publish/subscribe round trip. - [88] Step 55 observations: a publication made before subscribing never arrives;
test/#matchestest/aandtest/a/b/cbut nottesting/x, "due to which outbox the sub is listening". ✅ both, with two corrections: a topic is not a container (a routing key matched at publish time, not a place where messages sit), and#works on levels, not string prefixes, with+as the single-level form. Forward link: retained messages answer the late-subscriber case (step 71). - [89] "is it normal that
mvn -q dependency:resolvereturn nothing?" — yes:-qprints only warnings and errors. Verified Paho 1.2.5 and Jackson 2.22.2 in~/.m2. - [90] Client id "to avoid duplicate"; broker address "should come from the configurator, not hardcoded". Second half ✅ and confirmed by the code (
System.getProperty("mqtt.broker", …)), with the bootstrapping asymmetry added: the broker address is the one thing that cannot arrive as Intent. First half corrected: uniqueness is a symptom; the id is session identity — the broker keeps subscriptions, QoS>0 queues and the Last Will association under it. - [91] "i don't really see any message that should be retained" — correct for the system as it stood, and the gap is the point: it published no Status. The course code retains
S/onlineandS/subscriptionsand publishes Events non-retained. - [92] Step 59–60: no more scanning, just wait; careful because callbacks are asynchronous. ✅ with the hazards named — the callback runs on the client's MQTT thread,
waitForCompletion()inside it deadlocks, escaping exceptions break delivery — and the observation that this is whyFileSubscriptionsused a queue. - [93] Ownership per service listed correctly (ConsoleIn publishes
E/text; ROT13 ownsI,S/…,E/crypted; ConsoleOut ownsI,S/…). Nobody gained knowledge. Noted that ConsoleOut owns noE, and thatU/<unit>is what lets two instances coexist. - [94] "what do you mean by unit name?" — the
Ulevel: which concrete instance.MqttConfiguretakes three unit names, and those names are the topology. - [95]
Connection refusedfromMqttRot13— no broker on the laptop (port 1883 closed locally, open on the server). Fixed with-Dmqtt.broker=tcp://homeserver:1883before the class name. - [96] "it works now, got the Tutorial topics in the subscriber". A three-second subscribe with nothing publishing returned four retained Status messages and no Events — retained State, non-retained Events, demonstrated.
- [97] Before
MqttConfigure, each service announced itsItopic and idled. Then: "i should have build a verifier asking if the Intent has been received" — the asking reflex again; corrected:S/subscriptionsalready reports the effect, and an Intent that was rejected would not appear there. The filesystem could show messages, never state. - [98] Why hard-coding
Tutorial/ConsoleIn/U/alice/E/textis worse than hard-coding a mailbox name: it bakes in a unit, where the filesystem's relative paths made instance identity implicit per replica. Consequence added: one build then serves only one deployment. - [99] Step 63: "we want richer intent, doing it manually is undoable" — that is the step's conclusion; its premise is that
TOPIC some/pathhas one field, one meaning, no nesting and nothing to validate against. - [100] The two Jackson mappers: "one serialises, one deserialises" — the names, not the reason. Reading uses a YAML factory with unknown properties tolerated; writing uses strict indented JSON. Tolerant in, strict out.
- [101] "rot13 just has to send a request and the broker manage it" ✅ — and the consequence: the broker owns the subscription list, no third party can query it, so
S/subscriptionsis the component's own declaration, published so the architecture knows what the transport hides. - [102] "i'm a bit lost with all the files in 06" — file map given (four programs, two helpers) plus the runtime-rewiring recipe with a second ConsoleIn under another unit.
- [103] Rewiring confirmed on a running system; the invalid filter produced a local log line. Verified from the retained Status that
S/subscriptionshad moved to the new source and that the rejection left it untouched. Gap named: rejections are invisible to the architecture. - [104] Last Will:
Ctrl+Cleavesonline=truefor late monitors; the will lets the broker publishfalse. ✅ with three additions — the will is retained (replacing the staletrue), it is registered at connect time, and a clean DISCONNECT does not trigger it. - [105]
Dis the description of what a service accepts and publishes; unnecessary so far because three self-explanatory services and one human did the wiring. Sharpened:Dmakes the Part VII compatibility rule machine-checkable. - [106] Compatibility test
type(out) == type(in)— necessary but not sufficient: compatibility is semantic (Celsius versus Fahrenheit type-check identically). What MQTT changes: deciding is now observable (live traffic,D), and inserting a transformer is additive and invisible to the producer. - [107] "there is still coupling done by the broker" — corrected: the remaining coupling is agreement on meaning between components (syntax, semantics, topic contracts, Intent, ownership, State and Event meaning). The broker is a shared infrastructure dependency, a different kind of thing.
- [108] Step 78 checklist answered for the running system. Corrections: no hard-coded producer identity (it arrives as Intent); "foreign contract" means one owned by someone else, and here it is one Event contract per service plus its own Intent schema; and question 12 reads the other way round — with no mediators at all, it passes vacuously.
Part IX — chapter written
08-part-ix-mqtt.qmd covers Part VIII and Part IX up to step 78, in all three languages: the filesystem's verdict, the mapping table, Paho and the client id, structured Intent and the two mappers, DUISE with the runtime rewiring and its rejection gap, retained Status with the Last Will, and the coupling that remains with the six checklist answers.
Progress record
- Current step: Part IX read through step 78; steps 79 and 81 left to read, step 80 is the student's own project
- Last observed result: MQTT chain running against the server's broker; rewiring and rejection both observed; retained Status verified twice
- Unresolved problem: rejections invisible to the architecture (a design decision for the report)
- Next action: the student starts the graded project (step 80); steps 79 and 81 remain as reading
⬅ Previous: Part VII · Index · Next: (not started yet) ➡
All parts: Setup & Step 0 · Part I · Part II · Part III · Part IV · Part V · Part VI · Part VII · Parts VIII & IX