hearth llm case study

verdict

hearth keeps granite4.2:3b. i tested it against three other small local models, and it’s the best fit for a 6 GB card it has to share:

1. 97.8% right tool and 97.8% right facts across 45 answers
2. 6.5 s median answer once the model is loaded
3. 100% of the model on the gpu, 2,489 MiB at peak

the only perfect scorer, granite4.2:8b, doesn’t fit on the card. ollama ran a quarter of it on the cpu, and it took twice as long to answer.

a dedicated router model, tev1, picked tools less accurately than granite, so it’s out too.

the setup

hearth is a read-only assistant for my homelab. it’s a discord dm bot, an mcp server, alerts that live in code, and a morning digest. when i ask it something in plain text, a small model running locally in ollama picks one of about ten read-only tools (disk usage, containers, logs, backups, gpu, the torrent and sonarr queues, endpoint probes) and writes the answer.

it runs on a gtx 1660 ti with 6 GB of vram. that card also runs the machine learning for my photo library (immich), and the other gpu is kept for media transcoding. so the model has to pick the right tool and still leave room on the card.

method

1. i recorded one pass of every tool’s real output as a frozen fixture, so every model saw identical data.
2. 15 questions: 14 about the homelab and one off-topic one (the weather) that should call no tool.
3. 4 models, 15 questions, 3 runs each: 180 scored answers. every model was unloaded before each run, so question 1 is always a cold load. runs were interleaved across models.
4. each answer gets two scores. right tool means it called a tool that can answer the question (and none for the weather). right facts means the answer contains the key value, like a disk %, a date or a MiB figure. latency is wall-clock time per question.
5. a speed probe sent each model the same short prompt through ollama’s api for load time, warm tokens per second and how much of the model landed on the gpu. vram came from prometheus.

the scorer was wrong first

the first baseline missed 4 of 15 facts, and all 4 were the scorer’s fault. the model wrote dates with non-breaking hyphens (U+2011) and thousands with a narrow no-break space (U+202F), said “no errors” in more words than the scorer expected, and once rightly said nothing was restart-looping where the scorer wanted a container count. the fix was to fold typography before matching and to let each fact list alternatives.

later, every miss across all 180 answers was read by hand, and 8 facts gained alternatives that say the same thing in other words. every miss that’s left is a real model error.

results

three of the four models are within one answer of perfect. lfm2.5:8b misses about 1 in 9.

accuracy · % of 45 answers per model

model	right tool	right facts
granite4.2:8b	100.0%	100.0%
qwen3.5:4b	100.0%	97.8%
granite4.2:3b (production)	97.8%	97.8%
lfm2.5:8b	88.9%	88.9%

on the two “what’s using the space” questions, lfm2.5:8b twice reported overall disk use instead of what was using it, and once called no tool and said none of its tools could tell. its tool scores were 12/15, 15/15 and 13/15 across the three runs, so judging it on one run would have been misleading.

once loaded, three models answer in a median of 4.5 to 6.5 s. granite4.2:8b takes 12.9 s.

seconds per answer · 3 runs

model	warm median	warm range	cold runs
granite4.2:3b (production)	6.5 s	2.2–27.3 s	15.7 s, 22.5 s, 14.9 s
qwen3.5:4b	5.3 s	3.6–63.7 s	45.6 s, 12.4 s, 12.9 s
granite4.2:8b	12.9 s	4.7–82.1 s	45.3 s, 37.3 s, 35.8 s
lfm2.5:8b	4.5 s	2.0–53.5 s	21.3 s, 11.0 s, 12.1 s

“cold” covered two different starts here. for most models, run 1 read the model file from disk. in runs 2 and 3 the model was unloaded from the gpu, but the file was most likely still in the os page cache. qwen3.5:4b took 45.6 s on its first cold question and 12.4 and 12.9 s on the next two.

both 8b models stop at the same point on this card and spill the rest to the cpu. the 3b uses under half the card.

peak vram · GTX 1660 Ti (6 GB)

model	peak vram	on gpu
granite4.2:3b (production)	2,489 MiB	100%
qwen3.5:4b	3,737 MiB	100%
granite4.2:8b	4,681 MiB	76%
lfm2.5:8b	4,683 MiB	89%

ollama put 76% of granite4.2:8b and 89% of lfm2.5:8b on the gpu. granite4.2:3b fits entirely and peaks at 2,489 MiB. immich’s ml held 0 MiB of vram during every run, so these are best-case numbers.

lfm2.5:8b generates fastest at 132.8 tok/s. granite4.2:8b, with a quarter of it on the cpu, manages 22.3.

warm tokens per second · load time

model	warm speed	load time
granite4.2:3b (production)	90.1 tok/s	2.4 s
qwen3.5:4b	66.6 tok/s	3.0 s
granite4.2:8b	22.3 tok/s	3.6 s
lfm2.5:8b	132.8 tok/s	4.8 s

classifier as a router

i also tried a model built only for the routing step. tev1 is a decision model, a qwen3.5 fine-tune from together ai that they label experimental. it doesn’t chat: it picks one option from a list and returns probabilities. for each of the 15 questions i asked it “which homelab tool answers this question?”, with hearth’s 10 tool descriptions and “none” as the options. as before, 3 interleaved runs per size, unloaded before each run.

tev1:4b picks the right tool 93.3% of the time and tev1:0.8b 80%. granite4.2:3b gets 97.8% while also writing the answer.

router · % right tool per model

model	right tool	time measured
tev1:4b	93.3%	routing decision 1.87 s (warm median)
tev1:0.8b	80.0%	routing decision 0.36 s (warm median)
granite4.2:3b (production)	97.8%	full answer 6.5 s (warm median)

1. tev1:4b’s 3 misses were all one question: the data-disk “what’s using the space” one, sent to overall disk usage in all 3 runs. lfm2.5:8b once made the same mistake.
2. tev1:0.8b also missed the nas version of that question and sent “errors in the jellyfin logs” to container health.
3. both sizes gave identical answers in all 3 runs, unlike lfm2.5:8b.
4. a warm routing decision takes 1.87 s on the 4b and 0.36 s on the 0.8b, about 5 s and 2.8 s cold. granite’s 6.5 s is the whole answer (route, tool call and reply), so these aren’t the same measure.
5. ollama puts tev1:4b at 4.7 GB on the gpu, against 2.5 GB for granite4.2:3b. a router still needs a chat model to write the answer, and 4.7 + 2.5 GB doesn’t fit the 6 GB card. the 0.8b plus granite, about 3.4 GB, would fit, at 80% routing.

so a router isn’t worth it for hearth, and i skipped the second test i had planned (tev1 judging which log lines matter).

the decision

production stays on granite4.2:3b.

1. granite4.2:8b is the most accurate, and i can’t afford it: it doesn’t fit on the card and takes twice as long to answer.
2. lfm2.5:8b is the fastest, but it picks the wrong tool or no tool on about 1 in 9 answers.
3. qwen3.5:4b is accurate (100% right tool) and it fits. its one miss was an invented backup time, the worst kind of error for an ops bot. it also thinks before it answers: on the speed probe it wrote about 1,000 output tokens (876, 1,228 and 1,099) where the 3b wrote 135 to 172 tokens. and it peaks at 3,737 MiB, 1,248 MiB more than the 3b.
4. granite4.2:3b ties qwen on facts and uses the least vram of the four. its one miss was on the free-space question, where it wrote the disk_usage call out as xml text instead of making it, so no tool ran.

reproducing it

the comparison is reproducible: a frozen fixture, every answer stored, and one script that re-scores them without a re-run and writes the numbers behind these charts. everything here is from the 2026-10-11 runs.