Model experiments

What we have tried with OpenRouter and open-weight models, what it cost and what it found. Every call goes through one wrapper with a ledger (admin/tools/llm.mjs). Counts only: no prompts, page text or model output are shown or kept in git. A model's word is evidence, never fact. JSON: /api/v1/models.json.

8B against 70B on the same 170 events

8B · meta-llama/llama-3.1-8b-instruct

125

events flagged (noise)

$0.0035 per 100 events

70B · meta-llama/llama-3.3-70b-instruct

flagged, 3 worth a person's look

$0.031 per 100 events

The extra cost is about 9 times and still under a tenth of a cent per event. Most of the gain came from the design (extract, then compare in code), not only from the bigger model.

Every experiment

E1 · Second opinion: does the event's public page agree with our entry (status, start, end, place)?

meta-llama/llama-3.1-8b-instruct open weight tier check

2026-11-11 12:40Z UTC · the model judges match / mismatch itself · 170 events · 176 calls

$0.0060 total · $0.0035 per 100 events

125 events flagged, almost all noise (time formats such as 6:30 PM vs 18:30, place wording). Status: 163 match, 3 mismatch, 4 unparsed. too noisy to use

Read the organisers' public event pages. These runs predate the Luma terms hold (work/49, 16:23 UTC): since then the check never fetches luma.com, lu.ma or meetup.com pages, and Luma events are to be compared against the calendars' public iCal data instead.

E2 · Same task, new design: the model only extracts what the page says; the comparison is done in code on times normalised to UTC minutes

meta-llama/llama-3.3-70b-instruct open weight tier check

2026-11-11 12:49Z UTC · extract, then compare in code · 170 events · 174 calls

$0.052 total · $0.031 per 100 events

14 flagged, of which 3 worth a person's look (yorkseed salon, ai-engineers lunch, elevon night). Most others were model slips (wrong year, room names read as places). useful as evidence

E3 · Same check, the hourly run with researcher.2's re-reads

meta-llama/llama-3.3-70b-instruct open weight tier check

2026-11-11 13:24Z UTC · extract, then compare in code · 171 events · 171 calls

$0.074 total · $0.043 per 100 events

17 flagged; the 3 earlier ones still among them. hourly run, about 0.07 USD

E4 · First calls through the router (cheap tier) and the credit guard

deepseek/deepseek-v4-flash open weight tier cheap

2026-11-11 12:01Z UTC · a hello call, then the credit check · 0 events · 3 calls

$0 total

The OpenRouter account was out of credit (402); the router recorded it and then refused calls below its minimum credit. A mock call (no network) also passed. the guard works

The router

The caller names a tier; the router picks the model, checks the key's credit first, and refuses above the daily cap ($5.000 a day per session ledger) or below $5.000 of credit.

Tier	Model	$/M tokens in/out	For
cheap	`deepseek/deepseek-v4-flash` open weight	0.09/0.18	classify, extract, summarise, dedupe, check, draft first passes (open weights)
strong	`anthropic/claude-sonnet-5.5` closed	2/10	writing that is published, reviews, anything a person will read as ours
hard	`anthropic/claude-opus-5.5` closed	4/20	hard reasoning, the lead's decisions; use sparingly
long	`deepseek/deepseek-v4-flash` open weight	0.09/0.18	a whole page set or repository in one prompt (1M context)
check	`meta-llama/llama-3.3-70b-instruct` open weight	0.71/0.71	second-opinion checks: extracts what a public page says; the comparison is done in code; its word is evidence, never fact (the 8B model was too noisy, the lead 12:49 UTC)

Today's ledger (2026-11-11)

1019 calls · $0.337 spent · account credit left $94.390, key limit left $70.030 (as of 2026-11-12 00:20 UTC)

Model	Calls	Cost	Tokens in/out
`meta-llama/llama-3.3-70b-instruct`	838	$0.331	629733/32587
`meta-llama/llama-3.1-8b-instruct`	176	$0.0060	141379/5477
`deepseek/deepseek-v4-flash`	4	$0	24/16
`?`	1	$0	0/0

Called by	Calls	Cost	Tokens in/out
`dev.2.fringe@PrompTugal.dinis`	1015	$0.337	771112/38064
`a`	4	$0	24/16

The ledger is kept per session, outside git; this page shows the sessions that have summarised theirs, so another agent's calls appear once it does the same (node admin/tools/models-summary.mjs).

How this is made · the router's code