{"version":"https://jsonfeed.org/version/1.1","title":"discostew test on Cloudflare","description":"Local AI experiments and notes from testing what is possible now.","home_page_url":"https://discostew.dev/","feed_url":"https://discostew.dev/feed.json","items":[{"id":"https://discostew.dev/posts/good-news-jesus-is-risen/","url":"https://discostew.dev/posts/good-news-jesus-is-risen/","title":"Good News: Jesus Is Risen","summary":"Just trying to share the Word: the tomb is empty, hope is alive, and Christ has conquered death.","content_text":"!Vintage newspaper-style Good News Gazette image proclaiming that Jesus is risen, with Jesus standing outside the empty tomb. Just trying to share the Word: Jesus is risen, the tomb is empty, and hope is alive. He is not here; he has risen, just as he said. — Matthew 28:6 That is the good news I want people to know.","date_published":"2026-07-06T00:00:00.000Z","date_modified":"2026-07-06T00:00:00.000Z","authors":[{"name":"discostew"}],"tags":["faith","Jesus","resurrection"],"attachments":[{"url":"https://discostew.dev/posts/good-news-jesus-is-risen.md","mime_type":"text/markdown","title":"Good News: Jesus Is Risen markdown source"}],"_markdown_url":"https://discostew.dev/posts/good-news-jesus-is-risen.md"},{"id":"https://discostew.dev/posts/local-model-fireworks-approval-results/","url":"https://discostew.dev/posts/local-model-fireworks-approval-results/","title":"The fireworks test was really a runtime test","summary":"A receipt-heavy proof run for three local runtime lanes: GLM 5.2, MiniMax M3, and DeepSeek V4 Flash.","content_text":"I published a small proof run from a local model evaluation lane. The prompt was simple: make a fullscreen browser fireworks animation. The actual test was not simple. This run used three different local runtime lanes: 1. GLM 5.2 through a GLM-capable llama.cpp server 2. MiniMax M3 through a MiniMax-specific llama.cpp PR build 3. DeepSeek V4 Flash through DS4, a separate DeepSeek-specific runtime That is the part I care about. The benchmark was not only “which model made nicer fireworks?” It was whether I could keep incompatible local model runtimes isolated, start them one at a time, validate their browser artifacts, stop them cleanly, and preserve the receipts. Result: GLM 5.2: published MiniMax M3 Q5: excluded after hard-gate failure DeepSeek V4 Flash: published MiniMax actually made a lively scene, but it failed the hard gate because visible non-canvas overlays helped the output. A model can make something that looks good and still fail the contract, so that artifact was excluded from the public catalog. Blog post and animations: https://discostew.dev/posts/local-model-fireworks-approval-results/ GLM 5.2 animation: https://discostew.dev/benchmarks/fireworks-approval-2026-06-21/glm-5-2/ DeepSeek V4 Flash animation: https://discostew.dev/benchmarks/fireworks-approval-2026-06-21/deepseek-v4-flash/ Runtime sources: GLM lane used ggml-org llama.cpp at b9716: https://github.com/ggml-org/llama.cpp/tree/b9716 MiniMax M3 lane used the preliminary MiniMax-M3 llama.cpp PR: https://github.com/ggml-org/llama.cpp/pull/24523 Fetch command for that PR branch: git fetch origin pull/24523/head:minimax-m3 DeepSeek V4 Flash used DS4: https://github.com/antirez/ds4 Model downloads: GLM 5.2 GGUF, UD-Q3_K_M: https://huggingface.co/unsloth/GLM-5.2-GGUF/tree/main/UD-Q3_K_M MiniMax M3 GGUF, UD-Q5_K_M: https://huggingface.co/unsloth/MiniMax-M3-GGUF/tree/main/UD-Q5_K_M DeepSeek V4 Flash GGUF for DS4: https://huggingface.co/antirez/deepseek-v4-gguf The durable lesson: local AI work is not just about getting a model to run. It is about keeping the runtime, artifact, validation, and failure mode inspectable.","date_published":"2026-06-21T00:00:00.000Z","date_modified":"2026-06-21T00:00:00.000Z","authors":[{"name":"discostew"}],"tags":["evals","local models","AI reliability","benchmark"],"attachments":[{"url":"https://discostew.dev/posts/local-model-fireworks-approval-results.md","mime_type":"text/markdown","title":"The fireworks test was really a runtime test markdown source"}],"_markdown_url":"https://discostew.dev/posts/local-model-fireworks-approval-results.md"},{"id":"https://discostew.dev/posts/proof-david-wilderness-stealth-prototype/","url":"https://discostew.dev/posts/proof-david-wilderness-stealth-prototype/","title":"Proof: David Wilderness Stealth Prototype","summary":"Evidence from a real Godot 4 render and validation run for a hex-grid stealth tactics prototype about David reaching the Cave of Adullam.","content_text":"!Real Godot capture of the David Wilderness Stealth prototype showing the turn UI, suspicion and alarm meters, David on a hex map, patrol enemies, terrain, and the Cave of Adullam objective. This is the actual rendered prototype, not a mockup image. I captured it by running the Godot 4 project and instantiating the main scene through the engine renderer. David Wilderness Stealth is a small mobile-friendly, turn-based stealth tactics prototype: David starts on the left side of an axial hex map and wins by reaching the Cave of Adullam on the right. The point of the slice is not combat or Bible trivia. The point is whether restraint, timing, patrol vision, terrain, and sound can carry a readable stealth puzzle. What is proven | Evidence | What it proves | | --- | --- | | Real Godot render capture | scenes/Main.tscn instantiates, lays out the UI, draws the hex board, draws units, and renders the game state through Godot rather than a hand-made diagram. | | godot --headless --editor --quit | Godot 4.6.3 can load the project, register the scripts/classes, and complete the editor import/syntax pass without parser or compile errors. | | python3 tools/validate_project.py | The repo has the expected project files, core classes, terrain types, enemy definitions, controls docs, and a passable route from David's start tile to the cave objective. | Validation output: OK: Godot prototype structure and level data look sane. What the slice does Draws an 8×8 axial hex map. Represents terrain as open ground, brush, rock, high ground, and cave/objective tiles. Lets David move one adjacent passable hex per turn. Provides Wait and Throw Stone actions. Runs two enemy patrol units: Saul's Scout and Saul's Soldier. Tracks enemy facing, patrol state, suspicion, alarm, last known David position, and investigation targets. Draws vision overlays and noise/debug overlays so the stealth rules are inspectable while tuning. Wins when David reaches the Cave of Adullam. Loses when the alarm fills. Includes restart flow. Why the mechanics matter The prototype is trying to prove a specific game feel: biblical wilderness survival as a stealth puzzle, not a combat skin. That means the meaningful actions are deliberately modest: move carefully, wait quietly, use terrain, misdirect patrols with sound, read vision cones, escape without turning David into an action hero. The interesting design question is whether those systems create tension before any art pass, narrative layer, or progression system exists. Architecture snapshot The slice is split into small Godot/GDScript modules rather than one scene script: scripts/core/hex_grid.gd — axial hex math, distance, neighbors, pixel conversion, line sampling. scripts/core/level_data.gd — width, height, terrain rows, David start, cave exit, enemy patrol routes. scripts/core/tile.gd — terrain behavior. scripts/core/unit.gd — David/enemy unit model and AI state. scripts/core/game_state.gd — board state, suspicion, alarm, debug markers. scripts/player_controller.gd — selected action and click dispatch. scripts/systems/enemy_ai.gd — patrol, suspicious, alerted, and returning behavior. scripts/systems/vision_system.gd — facing-cone vision with terrain interaction. scripts/systems/noise_system.gd — movement, wait, and thrown-stone noise. scripts/systems/turn_manager.gd — player action, enemy phase, suspicion/alarm resolution, win/loss. scripts/ui/hex_board.gd — board rendering and input hit testing. scripts/main.gd — scene wiring, labels, and buttons. What is not proven yet This is still a prototype, not a finished game. The current proof covers the engine-load path, render path, level structure, and core rules. It does not yet prove final art, balance, onboarding, mobile UI polish, or whether the patrol timing is fun after repeated play. The next useful work is to playtest the loop, tune suspicion/alarm pacing, and make enemy vision previews clearer before hiding any debug aids.","date_published":"2026-06-17T00:00:00.000Z","date_modified":"2026-06-17T00:00:00.000Z","authors":[{"name":"discostew"}],"tags":["game prototype","Godot","stealth tactics","agent proof"],"attachments":[{"url":"https://discostew.dev/posts/proof-david-wilderness-stealth-prototype.md","mime_type":"text/markdown","title":"Proof: David Wilderness Stealth Prototype markdown source"}],"_markdown_url":"https://discostew.dev/posts/proof-david-wilderness-stealth-prototype.md"},{"id":"https://discostew.dev/posts/chromadb-was-extra-state-not-evidence/","url":"https://discostew.dev/posts/chromadb-was-extra-state-not-evidence/","title":"ChromaDB was extra state, not extra evidence","summary":"Why I removed ChromaDB from Baseball RAG after realizing it did not give me anything the model and database could not already do better.","content_text":"Chroma gave me another place to look. It did not give me another source of truth. I had ChromaDB in Baseball RAG for a while because it seemed like the obvious thing to do. It is a RAG project. RAG projects usually have a vector database. Baseball has biographies, player descriptions, old names, weird history, and fuzzy questions. So a vector store sounded useful. But after working through the actual system, I do not think Chroma owned a real job. That was the problem. The system I actually needed Baseball RAG is not supposed to be a chatbot that sounds good about baseball. I wanted natural language in and grounded evidence out. That means different parts of the system need clear jobs: DuckDB answers structured stat questions. Lahman is the primary factual authority. Retrosheet can add optional secondary evidence for some biography claims. Local stat-definition Markdown handles supported glossary questions. The local LLM writes prose, classifies intent, and explains grounded results. The eval gate catches drift when those boundaries break. That is already a lot. So when I looked at Chroma honestly, the question became simple: What does this component own? And I did not like the answer. Chroma did not own facts If I needed a stat, DuckDB was better. It had the tables. It had the rows. It had SQL. It had the dataset manifest. It had checksums and source metadata. If the user asked who had the most RBIs in 1962, I did not want a nearest-neighbor search over baseball text. I wanted the database to answer it. A vector result can feel like evidence because it returns something that looks relevant. But similarity is not authority. If Chroma retrieved a paragraph saying a player hit 714 home runs, I still needed DuckDB or another structured source to verify it. If the model generated that same sentence on its own, I still needed the same verification step. Chroma did not remove the need for the database. It added another path that could be stale, missing, or rebuilt differently. Chroma did not own prose The other argument for Chroma was biographies. That made sense at first. Biographies are fuzzier than leaderboard queries. They need context. They need readable language. They do not always map cleanly to one SQL query. But that is exactly the part the model can already do. If the goal is readable prose, the LLM is the prose engine. Chroma was not writing better biographies by itself. It was retrieving text that the model still had to turn into an answer. And if that retrieved text was generated or stale or only loosely related, then I had not improved the system. I had just given the model a different pile of words to lean on. That is not enough. For biographies, the better design was: 1. Resolve the player identity through DuckDB/Lahman. 2. Let the model write readable prose. 3. Extract supported stat claims. 4. Verify those claims against structured sources. 5. Show warnings when something is unsupported or conflicting. That gives the model a useful job without pretending the model is the source of truth. Chroma did not make that boundary clearer. It made it fuzzier. Chroma did not own verification This was the part that bothered me most. The system already needed claim verification. If a biography says a player hit a certain number of home runs, played for a team in a specific season, or led a league in a category, I need that checked against structured data when structured data exists. Chroma could not replace that. A retrieved paragraph can support a claim in a loose human sense, but it is not the same as a checked row from a known source. It does not tell me enough about authority, freshness, or whether the number came from generated text that got embedded earlier. So the system still needed: source authority structured rows SQL or equivalent evidence warnings verification states eval coverage Once I accepted that, Chroma looked less like retrieval and more like extra state. The debugging problem The worst architecture is the one that makes a future bug harder to explain. With Chroma in the middle, a wrong answer could come from too many places: Did DuckDB return the wrong row? Did the model invent something? Did Chroma retrieve stale text? Was the vector index missing? Was the local index rebuilt differently? Did generated text get embedded and treated like evidence later? I do not want users debugging that. I do not want future me debugging that either. For a portfolio project, that matters. The point is not just that the system answers baseball questions. The point is that the system shows taste about where facts come from. Adding another component is easy. Removing the component with unclear authority is the harder call. The better boundary The simpler architecture made more sense: Question | v Router |-- stat query -----------> DuckDB |-- database question ----> typed spec -> parameterized SQL -> DuckDB |-- player biography -----> DuckDB identity -> LLM prose -> stat-claim checks |-- stat explanation -----> local definitions first, then LLM if needed | v Answer with sources, warnings, metadata, and review state That is less magical. Good. The database owns structured facts. The model owns language. The verifier owns supported factual claims. Unsupported or ambiguous questions fail closed. No hidden vector memory in the middle. The lesson Chroma was not bad technology. It was bad ownership. It did not own facts. DuckDB owned facts. It did not own prose. The model owned prose. It did not own verification. The claim checker owned verification. So it became extra state with unclear authority. That is exactly the kind of component I want to remove from an AI system. The useful lesson for me was not \"never use vector databases.\" The lesson was: do not add retrieval just because the project is called RAG. If retrieval does not make the answer more inspectable, more grounded, or easier to verify, it may not be helping. It may just be giving the system one more place to hide the truth.","date_published":"2026-06-13T00:00:00.000Z","date_modified":"2026-06-13T00:00:00.000Z","authors":[{"name":"discostew"}],"tags":["AI reliability","RAG","ChromaDB","DuckDB","local AI"],"attachments":[{"url":"https://discostew.dev/posts/chromadb-was-extra-state-not-evidence.md","mime_type":"text/markdown","title":"ChromaDB was extra state, not extra evidence markdown source"}],"_markdown_url":"https://discostew.dev/posts/chromadb-was-extra-state-not-evidence.md"},{"id":"https://discostew.dev/posts/ai-reliability-accessibility/","url":"https://discostew.dev/posts/ai-reliability-accessibility/","title":"Making AI Reliability Accessible","summary":"A short note from discostew on treating AI reliability as an accessibility practice, not just an infrastructure concern.","content_text":"Reliability work often gets framed as something reserved for platform teams, benchmarks, or incident reviews. discostew's angle is more practical: a system that explains itself, fails clearly, and keeps its promises is also a more accessible system. That matters for AI because the rough edges are not always visible. A model can sound confident while drifting away from the task, burying uncertainty, or asking a user to do detective work. The accessibility problem is not only whether the interface can be read by a screen reader. It is whether the whole experience gives people enough structure to understand what happened and recover. Good reliability habits help: Make state visible before asking the user to trust an answer. Prefer plain language around uncertainty, blocked actions, and next steps. Keep source files and operating instructions in git so automated helpers can make small, reviewable edits. Treat validation as part of the user experience, especially when generated content may be reused by people or machines later. That is why this blog is intentionally boring in the best way. Posts are Markdown or MDX files. The generated RSS feed, sitemap, robots file, JSON-LD metadata, and /llms.txt index all point back to the same simple body of writing. There is not much machinery to hide the work from future discostew, future Hermes, or future readers.","date_published":"2026-06-09T00:00:00.000Z","date_modified":"2026-06-09T00:00:00.000Z","authors":[{"name":"discostew"}],"tags":["AI reliability","accessibility","software practice"],"attachments":[{"url":"https://discostew.dev/posts/ai-reliability-accessibility.md","mime_type":"text/markdown","title":"Making AI Reliability Accessible markdown source"}],"_markdown_url":"https://discostew.dev/posts/ai-reliability-accessibility.md"},{"id":"https://discostew.dev/posts/fidelity-over-backward-compatibility/","url":"https://discostew.dev/posts/fidelity-over-backward-compatibility/","title":"Fidelity over backward compatibility","summary":"Why I accepted breaking API changes in Propontis when the old schema hid important evidence boundaries.","content_text":"Backward compatibility is valuable until it makes the system lie about what happened. TL;DR In Propontis, I accepted breaking schema changes because the old interface blurred evidence types that needed to stay separate. A source conversation, executable prior context, and an executed PyRIT conversation are not the same thing. Treating them as interchangeable made the API easier to keep stable and harder to trust. The better choice was fidelity. The problem Propontis deals with attack runs, evidence, model behavior, and reports. That kind of system needs a clean audit trail. The old shape made some things too convenient. It let different kinds of conversation evidence collapse into one mental bucket: the full source conversation prior context that can be executed or replayed the actual conversation produced by a PyRIT run Those are related, but they are not interchangeable. If a report mixes them together, a reviewer has to guess what they are looking at. Is this the input? The replay context? The executed attack trace? A transformed version of something else? That guessing is where trust goes to die. The uncomfortable decision The easy move would have been to preserve the old schema and keep adding fields around it. That would have avoided churn. It also would have preserved the confusion. So I accepted a breaking change. The schema needed names that matched the evidence. The output needed to make boundaries explicit. Validation placeholders had to become real facts. Expected text-mode failures had to stop dumping tracebacks as if every known failure were a surprise. A vague config_status field needed to become something closer to the truth, like model_readiness. None of that sounds glamorous. It is the kind of work that makes demos slightly less smooth and operations much less weird. Why fidelity mattered more A red-team or evaluation system does not earn trust by keeping old field names alive. It earns trust by making review possible. If the system says an attack used a certain context, that context should be inspectable. If a report includes a conversation, the report should be clear about whether it is source material or executed evidence. If a model was not ready, the output should say that plainly instead of hiding behind a generic config status. Backward compatibility can protect users from unnecessary churn. But it can also protect bad abstractions from being corrected. In this case, the old abstraction was the problem. The accessibility angle Auditability is accessibility for reviewers. A reviewer should not need to reconstruct internal architecture from vague output names. They should be able to read the artifact and understand: what was provided to the system what was executed what failed what was expected what evidence supports the report When a schema hides those distinctions, it shifts work onto the reviewer. That is especially bad in security and evaluation work, where the whole point is to make failure easier to inspect. Breaking the schema reduced that burden. The rule I would use again I would not break an API just because I found a prettier name. But I would break one when the old contract makes users less informed. A stable lie is still a lie. A stable ambiguity is still ambiguity. If the interface hides truth, compatibility is not the highest virtue anymore. Takeaway Compatibility matters. Fidelity matters more when the system is supposed to prove what happened. The public contract should preserve the evidence boundary, not the convenience of the first version.","date_published":"2026-06-09T00:00:00.000Z","date_modified":"2026-06-09T00:00:00.000Z","authors":[{"name":"discostew"}],"tags":["Propontis","API design","auditability","red teaming","AI reliability"],"attachments":[{"url":"https://discostew.dev/posts/fidelity-over-backward-compatibility.md","mime_type":"text/markdown","title":"Fidelity over backward compatibility markdown source"}],"_markdown_url":"https://discostew.dev/posts/fidelity-over-backward-compatibility.md"},{"id":"https://discostew.dev/posts/hashing-the-contract-not-the-file/","url":"https://discostew.dev/posts/hashing-the-contract-not-the-file/","title":"I fixed an eval by hashing the contract, not the file","summary":"Why an evaluation baseline should track the public behavior users depend on instead of private implementation bytes.","content_text":"The eval was technically watching the data. It was watching the wrong thing. TL;DR One of my baseball RAG checks warned when the dataset version changed. The idea was right. The contract was wrong. I was hashing raw source files. That made the eval sensitive to private implementation details, not the thing the API promised to users. The better baseline was the compact provenance payload returned with answers. Hash the contract. Not the accident of how the contract is built. The original check The project uses structured baseball data and returns answers with provenance. I wanted CI to notice when the factual base changed, because a changed data source can change answers. The first version reached for the obvious thing: hash the underlying files. That seems reasonable. If a CSV changes, the hash changes. If the hash changes, CI can warn that the baseline needs review. But the raw file was not the public promise of the system. It was an input. The user never sees the raw file bytes. The API consumer does not care whether a newline changed, a source export was normalized differently, or a loader got refactored while producing the same public evidence. The thing users rely on is the provenance the system returns. The problem with private truth Tests can accidentally lock you to the wrong surface. A raw CSV hash is easy to compute, but it answers a narrow question: did these bytes change? The question I needed was different: did the factual contract exposed by the system change? Those are not the same. If I refactor loaders, reorder intermediate data, or normalize source files without changing the answer evidence, a raw-byte baseline can scream even though the system's contract is stable. Worse, a private hash can make the test feel rigorous while avoiding the real contract. It tells you something changed somewhere. It does not tell you whether the user-visible evidence changed in a meaningful way. That kind of test creates noise. Noise teaches people to ignore the test. The better baseline The fix was to build the baseline from the compact provenance payload. That payload is closer to the public contract: answer source authority dataset identity query path evidence rows or summaries warnings verification status That is what downstream code and human readers can reason about. If this payload changes, I want CI to force a review. Maybe the data changed. Maybe a source was upgraded. Maybe the provenance contract drifted. Any of those are worth seeing. If an internal file changes but the payload stays the same, the eval should stay quiet. Why this matters for AI systems AI systems already have enough moving parts. A test suite should reduce ambiguity, not add another layer of mystery. For generated answers, the public contract is not only the final paragraph. It is also the evidence around the paragraph: citations, provenance, verification state, warnings, and failure modes. That is the part users need when the answer matters. A model can make a wrong answer sound smooth. A good contract makes the system show its work anyway. The accessibility angle Accessible systems reduce unnecessary user burden. When an eval watches the wrong thing, failures become hard to interpret. The maintainer has to ask, \"Did anything important change, or did a private file just move around?\" That burden eventually leaks to users because noisy tests lead to weaker review habits. A contract-level baseline is easier to explain: This changed because the answer evidence changed. This changed because a source identity changed. This changed because a warning or verification state changed. That is a healthier failure. Takeaway A baseline is a promise about what deserves attention. If you hash private files, you promise to care about private files. If you hash the public provenance contract, you promise to care about the behavior users actually depend on. For this project, that was the right promise.","date_published":"2026-06-09T00:00:00.000Z","date_modified":"2026-06-09T00:00:00.000Z","authors":[{"name":"discostew"}],"tags":["evals","AI reliability","testing","provenance"],"attachments":[{"url":"https://discostew.dev/posts/hashing-the-contract-not-the-file.md","mime_type":"text/markdown","title":"I fixed an eval by hashing the contract, not the file markdown source"}],"_markdown_url":"https://discostew.dev/posts/hashing-the-contract-not-the-file.md"},{"id":"https://discostew.dev/posts/let-the-llm-write-make-the-database-check-facts/","url":"https://discostew.dev/posts/let-the-llm-write-make-the-database-check-facts/","title":"Let the LLM write the bio. Make the database check the facts.","summary":"A design note on generated biographies, stat-claim extraction, and using structured data to keep prose honest.","content_text":"I do not need the model to stop writing. I need the system to stop trusting every sentence it writes. TL;DR For player biographies, the LLM is allowed to write prose. That is the part it is good at. But if the prose includes a factual stat claim, the system extracts the claim and checks it against structured data before returning the answer. Lahman and DuckDB handle the primary verification path. Retrosheet can add secondary evidence when it applies. The model writes. The database checks. Why biographies are tricky A biography is not like a leaderboard query. If the user asks for the RBI leader in 1962, the system can run SQL and return a structured answer. The path is clear. If the user asks, \"Who was Hank Aaron?\", the answer needs prose. It needs context, transitions, and some judgment about what matters. That is where a model helps. The problem is that prose invites unsupported facts. A model might say a player had a certain number of home runs, played a specific season for a team, or led a league in a category. Some of those claims may be right. Some may be wrong. All of them sound the same when they are wrapped in confident language. That is not good enough. The design The biography path became a two-step process: 1. Generate readable biography prose. 2. Extract stat claims and verify them against structured sources. The system does not treat every sentence the same. General background may remain prose. Extractable stat claims get checked. A claim can land in states like: verified_by_all verified_primary_only contradicted_by_all conflict Those states are more useful than a single true/false label. A claim verified by the primary source is different from a claim verified by every available source. A conflict is different from a contradiction. A missing secondary source is different from a source disagreeing. The point is not to pretend the system knows everything. The point is to make the system honest about what it knows. Why not make the model self-check? Because that would put the same kind of component on both sides of the judgment. A model can help extract claims. It can help phrase a caveat. It can explain why a verification state matters. But the final check should come from structured evidence when structured evidence exists. If the claim is about a career total, a season stat, or a team relationship, that belongs in the data path. The model should not grade its own homework. What this changes for the reader A normal generated bio asks the reader to trust the paragraph. A checked bio can show more texture: This claim was verified against the primary source. This claim had secondary support. This claim could not be verified. This claim conflicted with another source. That turns the answer from a polished blob into something a reader can reason about. The output may be less sleek. Good. Sleek is overrated when facts are at stake. The accessibility angle People should not need baseball-reference-level expertise to know whether an AI answer is safe to reuse. Verification states lower the burden. They tell the user which parts of the generated prose are grounded and which parts need caution. That matters for anyone using the system with limited time, limited domain knowledge, or assistive workflows where manually cross-checking every claim is expensive. The system should carry more of that work. Takeaway The goal is not to ban the LLM from writing. The goal is to give it a narrow, useful job and then wrap that job with checks. Let the model make the biography readable. Make the database keep the biography honest.","date_published":"2026-06-09T00:00:00.000Z","date_modified":"2026-06-09T00:00:00.000Z","authors":[{"name":"discostew"}],"tags":["LLMs","verification","baseball RAG","AI reliability"],"attachments":[{"url":"https://discostew.dev/posts/let-the-llm-write-make-the-database-check-facts.md","mime_type":"text/markdown","title":"Let the LLM write the bio. Make the database check the facts. markdown source"}],"_markdown_url":"https://discostew.dev/posts/let-the-llm-write-make-the-database-check-facts.md"},{"id":"https://discostew.dev/posts/mcp-servers-and-context-tax/","url":"https://discostew.dev/posts/mcp-servers-and-context-tax/","title":"MCP servers have a context tax","summary":"Why I do not want persistent MCP servers in my default Hermes profile unless the recurring value is obvious.","content_text":"A tool can be useful and still not belong in every conversation. TL;DR I almost installed the Cloudflare MCP servers into my default Hermes setup. Then I stopped. The problem was not Cloudflare. The problem was persistence. MCP servers add tools, schemas, and descriptions to the agent's tool surface. If those tools load for every session, they consume attention and context even when I am not working on Cloudflare. That is a tax. Sometimes it is worth paying. Usually it should be paid on purpose. What the servers would have given me The Cloudflare setup exposed a few different kinds of capability. The low-risk one was docs search. That is genuinely useful. A Cloudflare-native docs tool can be better than generic web search when I am building Workers, Pages, D1, R2, KV, or related infrastructure. The authenticated servers were different. They could inspect or change real account resources. During discovery, one server exposed tools for things like listing Workers, reading Worker code, managing KV namespaces, working with R2 buckets, and querying D1 databases. That is powerful. It is also not something I want casually hanging off every agent session. The hidden cost MCP tools do not only matter when they are called. The agent has to know they exist. Tool names, schemas, and descriptions become part of the decision space. The model has to choose among them. The session has to carry enough context for them to be available. If I am writing a blog post, debugging a local Python test, or thinking through architecture, I do not need five Cloudflare servers sitting in the room. Even if the cost is small per tool, it is the wrong shape of cost. It is global. It applies before I know whether the task needs the tool. That is the part I dislike. The account-risk part There was also an account boundary issue. OAuth flows use whatever browser/account is active at the moment unless I am careful. For a personal setup with multiple accounts, that is a real footgun. A docs-only server is one thing. An authenticated server tied to the wrong Cloudflare account is another. The right default is not \"install everything and sort it out later.\" The right default is to keep the default profile lean and add account-bound tools only in a scoped environment. The profile boundary The better pattern is simple: default Hermes profile -> common tools only cloudflare profile -> Cloudflare MCP, OAuth, deployment work one-off task -> add temporarily, remove afterward That keeps the default agent sharp. It also makes the security boundary easier to reason about. If I open a Cloudflare-specific profile, I know why those tools are present. I know which account should be active. I know the session is allowed to think in Cloudflare terms. That is very different from every conversation carrying every integration forever. Stewardship of attention This is partly a local AI principle for me. I care about owning my tools, running things close to the metal, and keeping workflows inspectable. But ownership does not mean hoarding every integration in the default path. Good stewardship sometimes means adding capability. Sometimes it means refusing ambient capability because it makes the system harder to reason about. An agent should have the tools it needs for the work in front of it. Not every tool it might ever need. Takeaway MCP is useful. Persistent MCP is a design decision. For my default Hermes profile, the bar should be high: recurring value, low account risk, and a context cost that is clearly worth it. Otherwise, use the docs, use the CLI, use direct API calls, or spin up a dedicated profile for the job. A lean agent is easier to trust.","date_published":"2026-06-09T00:00:00.000Z","date_modified":"2026-06-09T00:00:00.000Z","authors":[{"name":"discostew"}],"tags":["Hermes","MCP","local AI","agent workflows","stewardship"],"attachments":[{"url":"https://discostew.dev/posts/mcp-servers-and-context-tax.md","mime_type":"text/markdown","title":"MCP servers have a context tax markdown source"}],"_markdown_url":"https://discostew.dev/posts/mcp-servers-and-context-tax.md"},{"id":"https://discostew.dev/posts/model-should-explain-evidence-not-be-evidence/","url":"https://discostew.dev/posts/model-should-explain-evidence-not-be-evidence/","title":"The model should explain the evidence, not become the evidence","summary":"A short decision story about source authority, provenance, and keeping generated prose out of the truth path.","content_text":"The model is useful when it explains evidence. It becomes dangerous when it quietly replaces it. TL;DR In my baseball RAG project, I had to make a boring but important decision: the model was not allowed to become the source of truth. Lahman stayed the primary authority for structured baseball facts. DuckDB stayed the place those facts could be queried. Retrosheet could support a claim when it had relevant evidence, but it did not become a second universal truth source. The model could write and explain. It could not decide what was true. That sounds obvious until you build the system and see how easy it is to blur the line. The tempting version The tempting version of a RAG system is simple: retrieve some text ask the model return the answer That works for a demo. It is not enough for a system that needs to be audited. The hard part is not getting a plausible paragraph. Models are good at plausible paragraphs. The hard part is knowing why the paragraph should be trusted. When the user asks, \"Who had the most RBIs in 1962?\", the answer should not come from a model's memory, a generated biography, or a loose semantic match. It should come from a query over the source data. The system should be able to show the SQL, the returned rows, and the source manifest behind those rows. If the system cannot do that, the user has to trust the tone of the answer. That is a bad interface. The boundary I wanted The boundary became: facts -> database and source manifests claims -> checked against facts prose -> written by the model answer -> returned with provenance That split gave each part of the system a job. The database handles structured truth. The provenance layer says where that truth came from. The model turns supported facts into readable language. The API response exposes enough evidence that a user, evaluator, or future version of me can inspect what happened. That is less impressive than a magic chatbot. It is also more honest. Why Retrosheet stayed secondary Retrosheet is useful. It has event-level baseball data that Lahman does not try to represent in the same way. But useful is not the same as primary. I did not want every source to compete for the same authority. For the core stat path, Lahman had already become the stable source. Retrosheet could provide secondary evidence for specific biography stat claims, especially when a claim needed extra confidence or a conflict needed to be surfaced. That meant the system could say something more precise than \"verified\" or \"not verified.\" It could distinguish between a claim supported by the primary source, a claim supported by multiple sources, a conflict, and a contradiction. Those states matter because they keep uncertainty visible. The accessibility angle This is also accessibility work. A system that hides its evidence asks the user to do extra labor. The user has to wonder whether the answer came from a table, a search result, a stale index, or the model's own memory. That is cognitive load the system should carry. A more accessible AI system should answer these questions without making the user dig: What source did this use? Was the claim checked? Did sources disagree? Is this prose or evidence? Can I trace the answer back to something stable? That is why provenance is not garnish. It is part of the user experience. Takeaway The model can be a writer. It can be an explainer. It can even be a useful parser when the output is checked. But it should not silently become the evidence. If an AI system needs to be reliable, the truth path should be boring enough to inspect. The model can make that truth easier to read. It should not be the place truth lives.","date_published":"2026-06-09T00:00:00.000Z","date_modified":"2026-06-09T00:00:00.000Z","authors":[{"name":"discostew"}],"tags":["AI reliability","provenance","RAG","source authority"],"attachments":[{"url":"https://discostew.dev/posts/model-should-explain-evidence-not-be-evidence.md","mime_type":"text/markdown","title":"The model should explain the evidence, not become the evidence markdown source"}],"_markdown_url":"https://discostew.dev/posts/model-should-explain-evidence-not-be-evidence.md"},{"id":"https://discostew.dev/posts/removing-chromadb-made-rag-reliable/","url":"https://discostew.dev/posts/removing-chromadb-made-rag-reliable/","title":"Removing ChromaDB made my RAG system more reliable","summary":"Why I removed a vector database from my baseball RAG project and kept DuckDB as the source of truth for structured facts.","content_text":"The best reliability fix I made in my baseball RAG project was deleting a database. TL;DR I removed ChromaDB from my baseball RAG system because it was doing the wrong job. It duplicated generated facts, depended on rebuildable local vector state, and made biography answers rely on an index that was not the source of truth. DuckDB stayed. Lahman stayed as the primary factual authority. Retrosheet stayed as optional secondary evidence for biography stat claims. The LLM still writes prose, but it does not get to decide what is true. The system The project answers baseball questions like: \"who had the most RBIs in 1962\" \"who played for the Braves in 1936\" \"what is OPS\" \"who was Babe Ruth\" I did not want a chatbot that simply sounded confident. I wanted a system that could show its work. That meant answers needed evidence: SQL, rows, source metadata, checksums, warnings, and verification results. The model could help with language, but the facts needed to come from somewhere inspectable. Where ChromaDB became a problem ChromaDB seemed useful at first for player biographies. Biographies are fuzzy, so a vector store felt like a natural fit. But that created a bad boundary. The structured facts already lived in DuckDB. Player identity resolved through Lahman-backed data. Stats came from tables. Provenance came from the manifest. ChromaDB added another place where facts could appear, drift, or go stale. That meant biography behavior could depend on local vector index state instead of the factual path I could audit. If the index was missing, stale, or rebuilt differently, the answer could change while still looking polished. That was the part I did not like. The decision I removed ChromaDB from the core runtime path. The architecture became simpler: Question | v Router |-- stat query -----> DuckDB |-- database question -> typed query spec -> SQL -> DuckDB |-- player bio -----> DuckDB identity -> LLM bio -> claim checks |-- explanation ----> local definitions first, then LLM explanation | v Structured answer with sources, warnings, and metadata The important part is the boundary. DuckDB answers structured baseball questions. Lahman is the primary factual authority. Retrosheet can add secondary consensus evidence for biography stat claims. The LLM can write a readable biography, but extractable stat claims get checked before the answer is returned. That is less magical. Good. Why this mattered A vector index is useful when search is the main problem. Here, authority was the main problem. If the user asks who had the most RBIs in 1962, I do not want the answer coming from a semantic memory of baseball text. I want it coming from a table, with the SQL and source rows visible. If the user asks who Babe Ruth was, I am fine with the model writing prose. But if that prose includes a stat, the system needs to check it. The model can explain the evidence. It cannot replace the evidence. The accessibility angle This is also an accessibility issue. A user should not need to reverse engineer the system to understand why an answer appeared. Hidden vector state, stale generated facts, and silent fallbacks all create extra work for the user. A more accessible AI system should make recovery easier: What handled this question? What source did it use? Did it run SQL? Were factual claims checked? Is anything unsupported? That is why provenance matters. It is not decoration. It is part of the interface. Takeaway The fix was not to make retrieval better. The fix was to remove retrieval from the place where it weakened the system. A RAG system does not become reliable because it retrieves more text. It becomes reliable when every part of the system knows its job.","date_published":"2026-06-09T00:00:00.000Z","date_modified":"2026-06-09T00:00:00.000Z","authors":[{"name":"discostew"}],"tags":["AI reliability","RAG","provenance","DuckDB","accessibility"],"attachments":[{"url":"https://discostew.dev/posts/removing-chromadb-made-rag-reliable.md","mime_type":"text/markdown","title":"Removing ChromaDB made my RAG system more reliable markdown source"}],"_markdown_url":"https://discostew.dev/posts/removing-chromadb-made-rag-reliable.md"}]}