blog
hearth llm case study, the morning digest
2026-10-15
verdict
the morning digest is the one thing hearth’s model writes that i read every day, so i tested the models on that job by itself.
- granite4.2:3b, the pick from the first and second case studies, named 9 of 21 planted problems in its headlines. qwen3.8:27b named 20 of 21.
- qwen3.8:27b doesn’t fit on the gpu, so ollama runs 84% of it on the cpu. a cold headline takes 77 to 86 s, which is fine for a message that goes out once a day at 8 am.
- no model told me everything was fine on a day with a problem.
- the only problem every model missed at least once was a nas share going offline. hearth printed it as a plain line saying the mount had no data, with no ⚠ marker, so that one was partly on hearth.
why a separate test
the first two posts scored tool choice and facts on free-text questions. in the digest, code collects the disk, error, import, backup and gpu facts and prints them in full, and the model only writes a two-line headline on top, saying what needs attention, if anything. it calls no tools, and nobody is waiting on it at 8 am. so i scored two things: whether the headline names the real problem, and whether it ever calls a bad day fine.
method
- i rendered the digest facts from the same frozen fixture as the other tests, then made eight versions by editing that text. each planted problem uses the exact wording hearth’s code prints for it.
- the eight cases: the facts as recorded (the data disk projected full within a week), a quiet day, a backup two days old, a backup that shrank 62%, the nightly backup job not running, a nas share offline, a burst of jellyfin database errors, and two problems at once (the data disk and the stale backup).
- four models and three interleaved runs, with every model unloaded before each run, so each run’s first headline is a cold load like the real 8 am one. that’s 96 headlines.
- a headline passes when it names every planted problem, never says all clear on a problem day, has no leaked thinking text, and fits in three lines. the quiet day has nothing to name, so it isn’t in the score.
- a keyword check scored them first, and then i read all 96.
the keyword check passed a wrong answer
the first headline i read said two nas shares would be full within a week. the warning was on the data disk. the keyword list had accepted “full within” without the disk’s name, so i tightened it and read every headline after that. that changed five more verdicts. four headlines named the problem in words the list didn’t have, like “skipped its run” or “not reporting at all”, and one had passed only because “missing” showed up in an unrelated warning.
results
| planted problem | granite4.23b | granite4.28b | qwen3.54b | qwen3.827b |
|---|---|---|---|---|
| data disk projected full | granite4.2:3b, data disk projected full: named in 1 of 3 runs | granite4.2:8b, data disk projected full: named in 3 of 3 runs | qwen3.5:4b, data disk projected full: named in 2 of 3 runs, 1 empty | qwen3.8:27b, data disk projected full: named in 3 of 3 runs |
| backup two days old | granite4.2:3b, backup two days old: named in 2 of 3 runs | granite4.2:8b, backup two days old: named in 3 of 3 runs | qwen3.5:4b, backup two days old: named in 1 of 3 runs, 2 empty | qwen3.8:27b, backup two days old: named in 3 of 3 runs |
| backup shrank 62% | granite4.2:3b, backup shrank 62%: named in 2 of 3 runs | granite4.2:8b, backup shrank 62%: named in 1 of 3 runs | qwen3.5:4b, backup shrank 62%: named in 3 of 3 runs | qwen3.8:27b, backup shrank 62%: named in 3 of 3 runs |
| backup job didn't run | granite4.2:3b, backup job didn't run: named in 2 of 3 runs | granite4.2:8b, backup job didn't run: named in 3 of 3 runs | qwen3.5:4b, backup job didn't run: named in 3 of 3 runs | qwen3.8:27b, backup job didn't run: named in 3 of 3 runs |
| nas share offline | granite4.2:3b, nas share offline: named in 0 of 3 runs | granite4.2:8b, nas share offline: named in 0 of 3 runs | qwen3.5:4b, nas share offline: named in 0 of 3 runs, 3 empty | qwen3.8:27b, nas share offline: named in 2 of 3 runs |
| jellyfin database errors | granite4.2:3b, jellyfin database errors: named in 2 of 3 runs | granite4.2:8b, jellyfin database errors: named in 3 of 3 runs | qwen3.5:4b, jellyfin database errors: named in 2 of 3 runs, 1 empty | qwen3.8:27b, jellyfin database errors: named in 3 of 3 runs |
| data disk + stale backup | granite4.2:3b, data disk + stale backup: named in 0 of 3 runs | granite4.2:8b, data disk + stale backup: named in 2 of 3 runs | qwen3.5:4b, data disk + stale backup: named in 3 of 3 runs | qwen3.8:27b, data disk + stale backup: named in 3 of 3 runs |
| named | 9/21 | 15/21 | 14/21 | 20/21 |
| cold load | 14–20 s | 28–49 s | 33–44 s | 77–86 s |
| on the gpu | 100% | 78% | 100% | 16% |
- qwen3.8:27b named 20 of 21. its one miss was a run where the offline share didn’t come up at all.
- granite4.2:8b named 15 and never caught the offline share. it also missed the shrunken backup in two of three runs.
- qwen3.5:4b named 14, and all seven of its misses were empty headlines. every headline it did return named the problem. i didn’t look into why it sometimes returned nothing.
- granite4.2:3b named 9. it never caught the offline share, never got both halves of the two-problem case, and in two of three runs on the recorded facts it pointed at the nas shares instead of the data disk.
what qwen3.8 gets wrong
qwen3.8:27b writes the most useful headlines. it usually leads with the worst problem and quotes the numbers that changed, like the backup dropping from 19.4 to 7.4 GB. six of its 24 headlines also claimed something the facts don’t support. three said a share would fill within days, or by the weekend, when the shares had 11 to 16 days left at their current growth. one blamed a config change the facts never mention, one warned that a share would go read-only, and one called a database error a quick config fix. i didn’t count this for the other models.
the quiet day
on the day with nothing planted, all four models flagged the same nas share in all three runs. that share sits at 98.5% on purpose, and hearth only alerts when it drops below 20 GB free, but the facts didn’t say so, and the models had no way to know. hearth now prints “full by design” and the floor next to those shares.
what changed in hearth
- the digest headline now runs on qwen3.8:27b with a 300 s timeout for that one call. its slowest headline in the test took 108 s, too close to the old 120 s limit. dms and alerts stay on granite4.2:3b, where a few seconds matter.
- a mount that stops reporting now gets a ⚠ marker, like a disk that’s projected to fill.
- the full-by-design shares say so in the facts, along with their free-space floor.
the first scheduled digest on the new model went out at 8:05 the same morning. its headline took 70 s.
the first live digest
the first time i ran the new digest on live data, its headline said the docker proxy was returning errors and the container view was dark. hearth reads docker through a read-only proxy, and docker had restarted the day before, after the proxy started. with live-restore on, containers keep running through a docker restart, so the proxy was still holding the old, deleted socket and had been failing for 22 hours. hearth’s own source alert had been open the whole time. the proxy now mounts the socket’s directory, so a docker restart can’t strand it again.
the decision
hearth now runs two models. granite4.2:3b is still the right model for dms and alerts, where it picks the right tool almost every time and answers in seconds. for the digest headline, qwen3.8:27b named 20 of 21 problems, and it costs about 80 s of cpu once a day.
everything here is from the 2026-10-15 runs: 96 headlines, all read by hand.