Files
ProcessControl/conv/08-part-VIII-IX.md
T

11 KiB
Raw Blame History

Conversation Log — Part VIII — The Filesystem Has Done Its Job · Step 53, and Part IX — MQTT · Steps 54– (turns 84–)

⬅ Previous: Part VII · Index · Next: (not started yet) ➡

All parts: Setup & Step 0 · Part I · Part II · Part III · Part IV · Part V · Part VI · Part VII · Parts VIII & IX


[84] Student

Filesystem was great: to apprehend the mehtodology, manage huge amount of file and create backup (if versionning is setup), but is clearly too long for short message, detect the changes (need a scan each times), heavy when the message must be shared with multiple subscribers

[84] Tutor — reply

Step 53's own list, matched — and the versioning caveat was spotted independently: the material says replication alone is not a backup strategy, because deletion and corruption replicate too. What the book adds: the filesystem's advantage MQTT cannot match (gigabytes to terabytes, where chunking a huge payload over MQTT means inventing a file-transfer protocol inside a message protocol), hence "use messages for coordination and events, use a file or object store for large data, and let a message carry the location, identity and metadata"; the full list of what the transport taught (including independent languages); and the accumulated machinery — directories, .tmp, atomic moves, .msg, WatchService, rescanning, shared paths, Syncthing, delay — behind "the architecture is getting clean, the transport is increasingly ridiculous". The student's "heavy for multiple subscribers" is the Step 3 queue-versus-topic limitation resurfacing as a transport problem.

[85] Student

I don't expect it to change the logic behind the architecture, i.e. the event driven with the subscription to topic and so on, i instead expect it to change how the subscription and mailling works, by replacing the file system with a lightweighten "software/process" that handle short message

[85] Tutor — reply

Every row of Step 54's mapping table, predicted before reading it: board root → broker/namespace, directory → topic, .msg → publication, contents → payload, WatchService → subscription, dynamic directory → topic filter, control/state/event directories → I, S/…, E/…. The section's punchline is that reaction itself: "Oh. We already know this architecture." MQTT is a transport change, not an architectural claim. Local tooling checked: mosquitto 2.1.2 with mosquitto_pub/mosquitto_sub already installed on the laptop.

[86] Student

how to install it on my server? i'll need it for my project

[86] Tutor — reply

Gave both routes for the home server (Docker Compose, matching how its other services run, and the native package), with the Mosquitto 2.x trap spelled out: since 2.0 it listens on localhost only and refuses anonymous clients unless told otherwise, so a listener line and an explicit allow_anonymous or a password file are required for the laptop, the phone or the vehicle to connect. Noted that port 1883 is unencrypted and belongs on the home LAN only, never exposed to the internet.

Progress record

  • Current step: 55 (Part IX); Part VIII closed
  • Last observed result: mosquitto 2.1.2 present on the laptop; the server has nothing yet
  • Unresolved problem: Part VIII chapter placement (own file or folded into the MQTT chapter)
  • Next action: student installs Mosquitto on the server, then runs the Step 55 publish/subscribe experiment

[87]–[108] Student & Tutor — Part IX, steps 55 to 78

Condensed, one line per exchange; the full reasoning is in the chapter 08-part-ix-mqtt.qmd.

  • [87] "the line was added to host, and i've tested and MQTT works well" — /etc/hosts entry added for the server; broker reachable. Verified independently with a publish/subscribe round trip.
  • [88] Step 55 observations: a publication made before subscribing never arrives; test/# matches test/a and test/a/b/c but not testing/x, "due to which outbox the sub is listening". ✅ both, with two corrections: a topic is not a container (a routing key matched at publish time, not a place where messages sit), and # works on levels, not string prefixes, with + as the single-level form. Forward link: retained messages answer the late-subscriber case (step 71).
  • [89] "is it normal that mvn -q dependency:resolve return nothing?" — yes: -q prints only warnings and errors. Verified Paho 1.2.5 and Jackson 2.22.2 in ~/.m2.
  • [90] Client id "to avoid duplicate"; broker address "should come from the configurator, not hardcoded". Second half ✅ and confirmed by the code (System.getProperty("mqtt.broker", …)), with the bootstrapping asymmetry added: the broker address is the one thing that cannot arrive as Intent. First half corrected: uniqueness is a symptom; the id is session identity — the broker keeps subscriptions, QoS>0 queues and the Last Will association under it.
  • [91] "i don't really see any message that should be retained" — correct for the system as it stood, and the gap is the point: it published no Status. The course code retains S/online and S/subscriptions and publishes Events non-retained.
  • [92] Step 59–60: no more scanning, just wait; careful because callbacks are asynchronous. ✅ with the hazards named — the callback runs on the client's MQTT thread, waitForCompletion() inside it deadlocks, escaping exceptions break delivery — and the observation that this is why FileSubscriptions used a queue.
  • [93] Ownership per service listed correctly (ConsoleIn publishes E/text; ROT13 owns I, S/…, E/crypted; ConsoleOut owns I, S/…). Nobody gained knowledge. Noted that ConsoleOut owns no E, and that U/<unit> is what lets two instances coexist.
  • [94] "what do you mean by unit name?" — the U level: which concrete instance. MqttConfigure takes three unit names, and those names are the topology.
  • [95] Connection refused from MqttRot13 — no broker on the laptop (port 1883 closed locally, open on the server). Fixed with -Dmqtt.broker=tcp://homeserver:1883 before the class name.
  • [96] "it works now, got the Tutorial topics in the subscriber". A three-second subscribe with nothing publishing returned four retained Status messages and no Events — retained State, non-retained Events, demonstrated.
  • [97] Before MqttConfigure, each service announced its I topic and idled. Then: "i should have build a verifier asking if the Intent has been received" — the asking reflex again; corrected: S/subscriptions already reports the effect, and an Intent that was rejected would not appear there. The filesystem could show messages, never state.
  • [98] Why hard-coding Tutorial/ConsoleIn/U/alice/E/text is worse than hard-coding a mailbox name: it bakes in a unit, where the filesystem's relative paths made instance identity implicit per replica. Consequence added: one build then serves only one deployment.
  • [99] Step 63: "we want richer intent, doing it manually is undoable" — that is the step's conclusion; its premise is that TOPIC some/path has one field, one meaning, no nesting and nothing to validate against.
  • [100] The two Jackson mappers: "one serialises, one deserialises" — the names, not the reason. Reading uses a YAML factory with unknown properties tolerated; writing uses strict indented JSON. Tolerant in, strict out.
  • [101] "rot13 just has to send a request and the broker manage it" ✅ — and the consequence: the broker owns the subscription list, no third party can query it, so S/subscriptions is the component's own declaration, published so the architecture knows what the transport hides.
  • [102] "i'm a bit lost with all the files in 06" — file map given (four programs, two helpers) plus the runtime-rewiring recipe with a second ConsoleIn under another unit.
  • [103] Rewiring confirmed on a running system; the invalid filter produced a local log line. Verified from the retained Status that S/subscriptions had moved to the new source and that the rejection left it untouched. Gap named: rejections are invisible to the architecture.
  • [104] Last Will: Ctrl+C leaves online=true for late monitors; the will lets the broker publish false. ✅ with three additions — the will is retained (replacing the stale true), it is registered at connect time, and a clean DISCONNECT does not trigger it.
  • [105] D is the description of what a service accepts and publishes; unnecessary so far because three self-explanatory services and one human did the wiring. Sharpened: D makes the Part VII compatibility rule machine-checkable.
  • [106] Compatibility test type(out) == type(in) — necessary but not sufficient: compatibility is semantic (Celsius versus Fahrenheit type-check identically). What MQTT changes: deciding is now observable (live traffic, D), and inserting a transformer is additive and invisible to the producer.
  • [107] "there is still coupling done by the broker" — corrected: the remaining coupling is agreement on meaning between components (syntax, semantics, topic contracts, Intent, ownership, State and Event meaning). The broker is a shared infrastructure dependency, a different kind of thing.
  • [108] Step 78 checklist answered for the running system. Corrections: no hard-coded producer identity (it arrives as Intent); "foreign contract" means one owned by someone else, and here it is one Event contract per service plus its own Intent schema; and question 12 reads the other way round — with no mediators at all, it passes vacuously.

Part IX — chapter written

08-part-ix-mqtt.qmd covers Part VIII and Part IX up to step 78, in all three languages: the filesystem's verdict, the mapping table, Paho and the client id, structured Intent and the two mappers, DUISE with the runtime rewiring and its rejection gap, retained Status with the Last Will, and the coupling that remains with the six checklist answers.

Progress record

  • Current step: Part IX read through step 78; steps 79 and 81 left to read, step 80 is the student's own project
  • Last observed result: MQTT chain running against the server's broker; rewiring and rejection both observed; retained Status verified twice
  • Unresolved problem: rejections invisible to the architecture (a design decision for the report)
  • Next action: the student starts the graded project (step 80); steps 79 and 81 remain as reading

⬅ Previous: Part VII · Index · Next: (not started yet) ➡

All parts: Setup & Step 0 · Part I · Part II · Part III · Part IV · Part V · Part VI · Part VII · Parts VIII & IX