Jonathan Mellette Contact

Applied AI · Operations, not a product

Infra­structure The machines the AI runs on

A 128 GB laptop serving local models through a fail-closed proxy, an agent host built deliberately bare, an edge LLM on an NVIDIA Jetson, a backup that cannot fail silently, and outages traced to a 32-bit clock in Apple’s own kernel source.

Status
In use · operations Model host, proxy and backup verified on 12 Oct 2026; the agent hosts are described as last recorded, on 7 Oct.
Licence
Private operations scripts, not released. The 27.8B model’s weights are Apache-2.0; the Jetson’s gemma3n model is under Google’s Gemma Terms of Use.
Stack
Ollama · a local 27.8B model in 8-bit and 4-bit · NVIDIA JetPack 7.2 · Open WebUI · systemd and launchd · a stdlib Python proxy
Links
None: private scripts, LAN-only interfaces.

Measured · figures marked recorded are older · how

The constraint

Run a self-hosted agent on a local 27.8B model without giving it the run of a real machine, and know when the machines underneath fail.

At a glance

awake time after boot at which the 32-bit TCP clock in macOS 26.0⁠–⁠26.3 stops advancing (232 ms)
49d 17h
2m 47s
on disk, then resident at a 262,144-token context (+65%)
31 → 51 GB
silent backup nights since the mid‑July hardening
0 of 89
from install start to a serving web UI with the edge model pulled, on a Jetson recorded 23 Jul 2026
~5 min
How these were measured
  • 49 d 17 h 2 m 47 s: 232 milliseconds converted in Python on 12 Oct 2026 (49.71 days). The release range comes from reading calculate_tcp_clock() in Apple’s published kernel source, which so far covers macOS 26.0 through 26.5: 26.0 to 26.3 carry the plain compare, and 26.4 and 26.5 the fixed one.
  • 31 → 51 GB: ollama list and ollama ps on the model host, 12 Oct 2026. The 8-bit build is 31 GB on disk and 51 GB loaded, fully on the GPU, at a 262,144-token context (51 ÷ 31 = 1.65). The 4-bit build goes from 18 to 29 GB.
  • 0 of 89: a short script read the backup’s own log on 12 Oct 2026 and checked every night from 16 Jul to 12 Oct for a run report or a failure record at the scheduled hour. Every night has one.
  • ~5 min: timestamps in the Jetson install logs from 23 Jul 2026. The install started at 09:02:50, and the web UI answered HTTP 200 at 09:07:53.
Isolation by machine boundary The agent host holds no repositories, no synced files and no credentials beyond its own keys. It reaches the model API over the LAN only through an allowlist proxy, which answered it with 200. Any other LAN host is refused with a 403 and a log line. The proxy passes allowed requests over loopback, on the same machine, to a model server that listens on loopback only. Agent host no repos no synced files only its own keys Others any LAN host not listed LAN · 200 403 Allowlist proxy 78 lines of stdlib Python the model API’s only LAN port anyone else: 403 + a log line same machine Model server bound to loopback only 27.8B model, 128 GB laptop
Fig. 1 Isolation by machine boundary, as built in September 2026.
Clock wrap Six steps. At boot the kernel’s TCP clock counts milliseconds in 32 bits. After 2 to the 32 milliseconds of awake time, which is 49 days, 17 hours, 2 minutes and 47 seconds, the new reading wraps to near zero. The plain less-than comparison then stays false, so the clock stops advancing. TIME_WAIT timers are checked against that clock, so they stop expiring. The 16,384-port pool drains. Outbound connections fail with EADDRNOTAVAIL while inbound traffic still works. Boot tcp_now counts ms in 32 bits 2³² ms later: 49 d 17 h 2 m 47 s the new reading wraps near zero The plain < compare stays false so tcp_now stops advancing TIME_WAIT stops expiring expiry is checked on tcp_now The 16,384-port pool drains no closed socket frees its port Outbound connections fail EADDRNOTAVAIL; inbound works
Fig. 2 The failure chain in macOS 26.0⁠–⁠26.3, from Apple’s published source. The dashed segment is 49.7 days in which nothing visible happens; a reboot only restarts the countdown.

The short version

  1. Traced an outage to one comparison in Apple’s kernel source. A stop-and-count experiment and a control host showed uptime, not load, was the variable. The fix was an OS point release.
  2. Isolated an AI agent by machine boundary, and verified it with a negative control. The model server listens only on loopback, and the one denial in the proxy’s log is the control.
  3. Built a backup that cannot fail silently. Integrity-gated snapshots and a record on every abort: 0 silent nights out of 89, and 0 degraded runs.

Case 1 of 5

A macOS kernel bug, traced to Apple’s published source

Two agent hosts kept losing every outbound connection. The cause was one comparison in Apple’s kernel.

Two always-on Mac mini agent hosts failed in the same odd way three times: once in July, then both within six days in October. Every outbound connection failed instantly with EADDRNOTAVAIL while inbound traffic still worked, and 96–98% of the 16,384 ephemeral ports sat in TIME_WAIT. recorded Jul–Oct 2026

The decisive experiment was to stop the agent and count again. Not one socket was reaped, though they should have cleared in about 30 seconds: the table had stopped draining. A control then isolated the variable: under the same workload, a host 6 days past boot held 3 sockets and one at 68 days held 15,882. Uptime, not load. recorded Oct 2026

Then the primary source. In macOS 26.0 through 26.3, the kernel’s calculate_tcp_clock() advances the 32-bit millisecond clock only if the stored value is less than the new reading. After 232 ms the reading wraps, the comparison stays false and the clock freezes, so TIME_WAIT timers checked against it never expire. Apple’s 26.4 source uses a modular compare. The harness’s own polling bug was only an accelerant: an older version without it filled the table about ten times more slowly and still died.

Sockets in TIME_WAIT under the same workload, against the 16,384-port pool recorded 7 Oct 2026
Port pool (the ceiling)16,384
Host at 68 days of uptime15,882
Host at 6 days of uptime3
 
- if (tmp < current_tcp_now) {
+ if (TSTMP_LT(tmp, current_tcp_now)) {
Fig. 3 calculate_tcp_clock() in Apple’s published XNU source: macOS 26.0⁠–⁠26.3 against 26.4 and 26.5. Re-read 12 Oct 2026.
How we know

Apple’s published kernel source, fetched 12 Oct 2026: bsd/netinet/tcp_subr.c at xnu-12377.81.4, line 3853 (macOS 26.3) and xnu-12377.101.15, line 3906 (macOS 26.4). The kernel each release ships was read from Apple’s distribution-macOS repository: 26.0, 26.1, 26.2 and 26.3 carry the plain compare, and 26.4 and 26.5 the modular one. TIME_WAIT expiry compares against tcp_now in tcp_timer.c. The socket counts, the stop-and-count result and the fill rates were recorded at the time with netstat on each host, and the version comparison from each host’s harness version and source; none can be re-measured now. The port range is the macOS default, read with sysctl on the model host.

Case 2 of 5

Isolating an AI agent by machine boundary, not configuration

The agent ran on a machine with almost nothing to take, and reached the model only through a proxy that fails closed.

The model needs the memory, so it runs on a 128 GB M5 Max laptop. The agent harness needs almost none, so it ran on a bare 16 GB Mac mini recorded 28 Sep 2026: no repositories, no synced files, no credentials beyond its own keys, and no screen-control driver, since an agent that can drive the desktop could get around every other limit.

The model server listens only on loopback, behind a 78-line allowlist proxy. I chose that over an OS firewall rule on purpose: a firewall config that an OS update silently overwrites fails open, while a mistake in the list fails closed and leaves a log line. It was tested both ways (Fig. 4).

Sizing was measured before anything was installed. The 4-bit build needs about 18 GB, so it could never fit the 16 GB box, and the 8-bit build is 31 GB on disk but 51 GB resident at full context, 65% more than the download.

Model proxy · both controls
200  agent host        listed
403  another LAN host  not listed
     GET /api/tags · DENY logged

$ grep -c DENY proxy.log
1
Fig. 4 Re-rendered without addresses, log name shortened. The controls ran once recorded 28 Sep 2026; the count was re-read on 12 Oct.
How we know

lsof on the model host, 12 Oct 2026: the model server listens on loopback only, so the proxy’s port is the only LAN path to its API. The proxy is 78 lines (wc -l); its allowlist, its 403 path and the methods it relays are in its source. grep -c DENY on its log returns 1, and the process has run without a restart since 28 Sep. Sizes, quantization, context and licence come from ollama list, ollama ps and ollama show on 12 Oct. The 200 result, the driver check and the agent host’s 16 GB were recorded on 28 Sep.

Case 3 of 5

Edge LLMs on Jetson: choosing the deterministic path for a batch

For eight boards, a repeatable host flash beat NVIDIA’s newer, easier installer.

Eight Jetson Orin Nano Super boards needed flashing. NVIDIA had recently shipped a USB installer that needs no host PC, though its Orin Nano quick-start page, dated days earlier, still showed only the old microSD flow. The research showed that was documentation lag, and found the trap that mattered for a batch: reports that the installer often applies the non-Super device tree, silently dropping the 25 W modes the boards were bought for.

The research proposed a one-board pilot. The runbook went straight to host-flashing with the explicit Super configuration instead: a flash host was already set up, and that path comes out identical on every board. Each board’s power modes are checked before it is trusted.

The logs show why the flash method matters. A board flashed with the plain configuration took a 536-package update that even installed the Super power profiles, rebooted, and still offered only 15 W and 7 W. Today 2 of 8 boards run the Super configuration, 2 owe a reflash, and 4 are still boxed.

  1. Scripted installs start: Ollama, and Open WebUI in a uv environment with no Docker
  2. Web UI set up as a systemd service
  3. 7.5 GB edge model pulled
  4. Web UI answers HTTP 200
Fig. 5 One Super board’s install logs: about five minutes to a serving web UI. recorded 23 Jul 2026
How we know

The update and reboot logs of the plain-flashed board: 536 packages upgraded, the Super power profiles installed and a firmware capsule staged, then only 15 W and 7 W offered and the bootloader version unchanged. The install, pull and readiness logs of a Super board, 23 Jul 2026: Ollama 0.32.1, Open WebUI 0.10.2 with 274 Python packages, and the gemma3n:e4b model. The five scripts total 210 lines (wc -l). Board counts come from the hardware inventory dated 10 Oct 2026, not from a command. The research and the runbook are private notes.

Case 4 of 5

A backup that cannot fail silently

Every night since mid-July has left a success report or an explicit failure record: 89 of 89.

A nightly launchd daemon backs up a workstation’s working tree, live SQLite databases included, to a NAS with no login session needed. Each database is snapshotted with sqlite3 .backup, checked with PRAGMA integrity_check, and copied only if it passes, into a per-run folder, so a bad night never overwrites good history. The rsync --delete mirror refuses an empty source, and a degraded run exits non-zero.

It earned that shape through two silent failures in its first eight days: a per-user job that never fired after an update reboot, and a mount check that never matched because macOS reports /var mounts as /private/var. The hardening went further: canonical paths, a 5-second liveness probe, bounded retries, and a failure record on every abort.

In all: 88 run reports since 8 July, none marked degraded. Six real failure nights in late August, from a DHCP reservation that was believed set and never was, were found by reading the log. The streak is now 24 clean nights.

The 89 nights from 16 Jul to 12 Oct 2026, from the backup’s own log
Success report at the scheduled hour72
Explicit failure record17
No record at all0
How we know

grep and a short Python script over the backup’s log on 12 Oct 2026: 88 run reports on 80 dates since 8 Jul (76 at the scheduled hour, 12 manual); 77 with an OK status and 0 degraded, the 11 earliest predating the status line; 17 failure records, all an unmounted NAS (six genuine, eleven during an expected off-network window). The daemon is 186 lines, and the empty-source guard, the liveness probe, the retries and the exit code are in its source.

Case 5 of 5

Diagnosing a display blackout from the kernel’s own log, with a falsification rule

The log showed a bandwidth renegotiation, not a dropped link, and the write-up names what would prove it wrong.

Two displays on a Thunderbolt 5 dock kept blacking out for a second at a time. The macOS unified log showed that the Thunderbolt data link never dropped. Inside the tunnel, DisplayPort bandwidth was being reallocated: a request for zero bandwidth, forced link training, then reallocation, one display after the other.

Software levers were ruled out on Apple Silicon: no link-rate override, a compression toggle that exists only on Windows, no dock firmware. Dropping the panel to 60 Hz, which had cured an earlier flicker, only partly helped. A replacement dock cured the blackouts, held to a clean-week bar before the case closed. The cause stays a working theory, a defective or under-provisioned unit, with a falsification rule: if the symptom returns, the explanation is wrong and the diagnosis starts fresh. The panel runs at 100 Hz today (system_profiler, 12 Oct 2026).

The earlier 100 Hz flicker, in Gbps: a 5120 × 2160 panel against a DisplayPort 1.4 link (ceiling inferred from behaviour)
60 Hz (the fix on the first dock)15.9
Usable on DisplayPort 1.425.9
100 Hz26.5
How we know

The write-up’s log signatures from the macOS unified log, its list of ruled-out levers and its decision log (private notes). The bandwidth arithmetic was re-run on 12 Oct 2026 at 24 bits per pixel: 5120 × 2160 × 24 × 100 Hz = 26.54 Gbps, × 60 Hz = 15.93 Gbps, and 4 lanes × 8.1 Gbps × 0.8 = 25.92 Gbps.