# Shane Logsdon — Full Article Corpus
> Full text of the published articles on https://shane.logsdon.io, for LLM ingestion.
---
# Evaluate Your DevRel Program Like an Agentic Workflow
_Published 2026-09-02_
_1,109 words, about 6 min read._
*AI recommendation is a distribution channel that emits no telemetry. Evaluate your DevRel program the way you'd evaluate an agentic workflow: a presence gate (are you cited?), then a quality gate (is the citation accurate?), baselined and audited on a cadence.*
My team maintains an internal guide for evaluating agentic workflows: the automated pipelines where an LLM analyzes developer feedback, generates READMEs, or produces artifacts downstream systems depend on. We wrote it to answer a boring operational question, which is how you know an automated workflow is actually working when no human watches every run.
Somewhere in the third or fourth revision I noticed the framework was answering a second question I'd been circling for months: how do you evaluate a DevRel program when your primary distribution channel is an AI that doesn't report back?
That's the actual situation now. When ChatGPT recommends your competitor, no analytics event fires. When a coding agent reads your docs and quietly picks a different tool, nothing shows up anywhere. The channel that increasingly decides developer adoption emits no telemetry to you by default. Which is exactly the problem the workflow evaluation framework was built for: output you can't watch continuously, produced by a system you don't fully control, where failures are silent until they compound.
The mapping turned out to be almost embarrassingly direct.
---
## The two gates
The workflow guide splits evaluation into two patterns, and the split carries over cleanly.
**Schema-critical evaluation** is binary. In a workflow, either the JSON parses or it doesn't; downstream steps break on failure, so validation is a hard gate. The DevRel equivalent is citation presence: when a developer asks an AI "what should I use for [your category]?", either your product appears in the answer or it doesn't. There's no partial credit. This is testable today with nothing but the four major assistants and a spreadsheet.
**Quality-first evaluation** is semantic. In a workflow, the artifact can vary in structure as long as the substance is right, so the gate is a second LLM call evaluating output against a rubric. The DevRel equivalent is citation quality: when your product *is* cited, is the guidance accurate? Is the framing favorable? Is the AI recommending your deprecated auth flow, or hallucinating a pricing tier you killed in 2024? An inaccurate citation is worse than absence, and it's invisible unless you evaluate for it, which, conveniently, is also a job you can give an LLM with a rubric.
Two gates, run in that order. Presence first, because quality of a citation that doesn't exist is undefined. Then accuracy, because presence with wrong guidance is a support ticket generator wearing a distribution win's clothes.
---
## The rest of the framework carries over too
| Workflow evaluation | DevRel program evaluation |
|---------------------|---------------------------|
| Schema validation as hard gate | Citation presence: cited or not, per query |
| Second-LLM quality rubric | Citation accuracy and favorability review |
| Human review loop, heaviest early | Monthly citation test cadence, heaviest at program start |
| Self-audit before humans see output | You run the citation test before a developer hits the gap |
| Retry then escalate | Fix the content gap, re-test, escalate to infrastructure work (llms.txt, structured content, docs restructuring; see post 2) if citation doesn't move |
| Pre-automation baseline (3+ manual runs) | Pre-AEO citation baseline before any optimization |
| 60-day written evaluation | Quarterly citation audit and retrospective |
| Graduation by deliberate sign-off | Deciding when monitoring moves from manual to automated |
Three of these deserve expansion, because they're where DevRel programs actually fail.
**The self-audit.** In our workflows, the system checks its own output before a human ever sees it. The DevRel version is running the citation test on yourself before a developer encounters the gap in the wild. When I ran mine, I got 0% citation on service queries and 100% on branded ones. Every assistant knew who I was; none recommended me for what I do. That's a self-audit result. It stung, and it was worth ten dashboards, because it was the output of the actual channel rather than a proxy for it.
**The baseline.** The workflow guide is strict about this: no credible time-saved claim without at least three documented manual runs before automating. The DevRel version: no credible AEO claim without a documented citation baseline before you optimize anything. Run the 20-query test: roughly twenty category and use-case queries ("what should I use for [your category]?"), a mix of service, category, and branded phrasings, across ChatGPT, Perplexity, Claude, and Gemini. Write down the numbers, then start work. Otherwise you'll ship llms.txt and structured content, citations will improve for reasons that may include model updates and competitor mistakes, and you'll have no way to claim any of it.
**Graduation.** This is the one I'd push hardest, because it cuts against the tooling instinct. In our framework, a workflow moves from active human review to automated-gate-only through deliberate sign-off, never on a timer or a metric threshold alone. The DevRel equivalent: don't hand citation monitoring to an automated service the moment one exists. The manual test phase is where you develop intuition for how the channel behaves, which queries matter, how answers drift after model releases. Automate detection once you know what normal looks like. Keep correction human indefinitely, because deciding *what to do* about a citation gap is a judgment call about your product's positioning, and that's the roughly 25% of DevRel that stays irreducibly human judgment (post 3).
---
## Why frameworks transfer
The reason this mapping works isn't a coincidence of vocabulary. Both problems have the same shape: a system that acts on your behalf, at a volume you can't inspect manually, where sentiment about it tells you nothing and only behavior counts. [Jono Bacon's line](https://blog.stateshift.com/beyond-github-stars/) about community metrics, that sentiment is noise and behavior is signal, is the same principle both evaluation designs land on independently. You don't ask whether the workflow feels reliable. You gate its output. Same with the DevRel program: don't ask whether it feels visible to AI, test the citations.
The instinct DevRel needs to borrow from platform engineering is treating the program itself as a production system: baselined, gated, audited on a cadence, with escalation paths and written retrospectives. Not because the spreadsheet is the point, but because the channel that now mediates your first developer touchpoint will never send you an error report. You have to go ask it.
*See also: [How to Know If Your Agentic Workflow Is Actually Working](https://shane.logsdon.io/articles/technical-deep-dives/evaluating-agentic-workflows/), the public write-up of the evaluation framework this post adapts.*
---
*Part 7 of 9 in the Developer Relations in the Age of AI series.*
*← Previous: What Do We Measure? · Next: Community When AI Answers the Questions → · ← All articles*
---
# What Do We Measure When AI Answers the Questions?
_Published 2026-08-26_
_1,030 words, about 6 min read._
*Docs views, GitHub stars, and event attendance now measure a shrinking channel. The DevRel dashboard that works in 2026 tracks four things instead: AI citation rate, agent-mediated activation, behavior over sentiment, and trust quality.*
A metric that goes up when you're less necessary is a broken metric.
That's the trap most DevRel dashboards are sitting in right now. If AI handles the first 75% of developer education, then tutorial views and docs engagement are increasingly measuring AI's effectiveness at replacing your content, not your program's impact. The decline is ambiguous in the worst way, too. Falling docs traffic could mean your content is failing, or it could mean your AEO is succeeding and developers went from AI citation straight to integration. It's the same chart with opposite conclusions, and nothing on the chart tells you which.
The old metrics were already weak before AI. [Jono Bacon's line](https://blog.stateshift.com/beyond-github-stars/) about the most-reported one is blunt: "GitHub stars are the participation trophies of the developer world." Stars are bookmarks with delusions of grandeur. You can't accidentally download a package the way you can accidentally star a repo. His prescription, that sentiment is noise and behavior is signal, was right then. AI raised the price of ignoring it, because AI broke the sentiment metrics' channel and created a new adoption path the old dashboard can't see, both at the same time.
Walk the board. Docs views measure a shrinking channel. Community Q&A volume drops as AI answers questions faster, which looks like community decline and may actually be community function shifting from Q&A to identity and trust. Event attendance measures a consumption mode that AI summarization is eroding. Social impressions were always distribution rather than impact. Every one of these can decline while your actual adoption accelerates through channels the dashboard doesn't instrument.
---
## The four categories that replace them
### 1. AI citation metrics
The leading question is no longer "how many developers saw this content?" It's "when a developer asks an AI what to use for this problem, do we appear?"
Run a monthly citation test: 20 category and use-case queries across ChatGPT, Perplexity, Claude, and Gemini, with citation frequency tracked over time. Track accuracy too. A wrong AI recommendation about your product is worse than absence, and you won't know it's happening unless you look. Between tests, AI referrer traffic is the always-on proxy, though a lossy one, since plenty of AI answers never produce a click at all. GA4 already segments sessions from chat.openai.com, perplexity.ai, claude.ai, and gemini.google.com.
### 2. Agent-mediated activation
[Neon tracked 80% of databases provisioned by AI agents](https://www.databricks.com/blog/databricks-neon), which is a category of activation traditional DevRel metrics don't capture at all. The versions of it worth tracking: agent skill installs (how many developers' coding agents now carry your context), the percentage of integrations that are agent-initiated (readable from coding-agent user-agent strings like Cursor or Claude Code, or API-key-first sessions with no docs referrer), and skill usage rather than install counts. Install count is stars again, one abstraction layer up. That's the trap to watch across this whole category. Every new channel grows its own vanity metric, and the test is always whether the number represents behavior or sentiment.
### 3. Behavior over sentiment
This is Bacon's framework, weighted heavier: package downloads over stars, integration completion rate over integration starts, and production deployment rate over quickstart runs. The gap between "ran the quickstart" and "reached production" is where DevRel programs lose developers silently, and it's also where the human 25% that AI can't do (post 3) either exists or doesn't. Weight PR contributions and detailed bug reports over community post counts. Someone filing a reproducible issue has skin in the game. Someone asking a question AI could have answered is low signal now.
### 4. Trust quality
The hardest to measure, the most important, and the least developed. Start with developer NPS scoped specifically to AI accuracy: "how accurate is the AI-generated guidance you've received about our product?" The AI mediation layer is now part of your developer experience whether you built it or not. Then time to first successful API call, decomposed by path: working alone, using general AI tools, or using your agent skills. The delta between those paths is the ROI measurement for your entire agent-experience (AX) investment. Finally, community self-sufficiency rate: what percentage of questions get answered by other community members (tag answerer role in your forum or Discord and track accepted answers from non-staff)? Trending up means the community is retaining expertise, and that's the metric that tells you whether you have a community or an audience.
---
## You'll have to build the instruments
Here's the honest catch: almost none of the citation and agent-activation metrics exist in standard DevRel tooling. Citation rate is a manual test. Answer accuracy is manual review. Agent-initiated integration tracking requires your own instrumentation, and skill install telemetry barely exists.
This is the same position SEO was in before analytics tooling matured, and the lesson transfers. The teams that built their own tracking first had the data advantage for years. A spreadsheet and a monthly hour of citation testing beats a polished dashboard of metrics that stopped meaning anything.
A workable cadence: weekly for AI referrer sessions, package downloads, and skill installs; monthly for the 20-query citation test, integration completion rate, and developer NPS; quarterly for production deployment rate, agent-initiated percentage, community self-sufficiency, and an AI answer accuracy audit.
Not everything changed. Time to first API call, integration completion, and production deployment are still valid and still behavior, so keep them. The mistake isn't measuring the old survivors. It's measuring only them while the first touchpoint of your developer experience moved somewhere your dashboard has never looked.
Your leadership will still ask about stars. Put them in the deck. But next to them, put the citation rate, because when the board asks "are developers finding us?", one of those numbers answers the question as it's actually being asked in 2026. The other is macaroni art.
---
*Part 6 of 9 in the Developer Relations in the Age of AI series.*
*← Previous: DevRel Is Distribution Work · Next: Evaluate Your DevRel Program Like an Agentic Workflow → · ← All articles*
---
# DevRel Is Distribution Work. Neon Just Proved What It's Worth.
_Published 2026-08-19_
_1,062 words, about 6 min read._
*Neon reached 80% AI-agent-provisioned databases and Databricks paid about $1B for it: proof that DevRel is distribution work. The old Code → Content → Community flywheel gained an AI-citation node, and agent skills are the new SDK.*
When Neon hit general availability, about 30% of the databases on the platform were being created by AI agents rather than humans. By the time Databricks acquired them, the number was 80%. Databricks paid roughly a billion dollars, and [their stated rationale](https://www.databricks.com/blog/databricks-neon) leaned directly on that statistic as evidence of "explosively growing agentic workloads."
Sit with the mechanism for a second. Neon invested early in agent-native distribution: AI rules files, Claude Code plugins, Cursor integrations, and a public agent skills library. That work made Neon the default answer when a coding agent needed a Postgres database. Being the agent's default became the majority of their growth, and the growth became the acquisition premium.
That's a DevRel story. It's arguably the most financially consequential DevRel story ever told, and it happened without a conference talk, a blog series, or a community program anywhere in the causal chain.
---
## The flywheel didn't break. It got a new node.
The flywheel I've run my own advocacy work on for years is Code → Content → Community. Working code becomes content. Content attracts community. Community feeds back signal about what to build next. When advocacy feels hollow, I diagnose which link is broken.
What AI changed isn't the flywheel's logic. It's the path between nodes. There's a new node between Content and Community now: AI citation. When an assistant recommends your product in response to a developer's question, that citation is word-of-mouth at scale: it earns the same trust a community referral used to, and that trust is what drives adoption, without the developer ever finding your content through search or social. So the main road is now Content → AI Citation → Adoption, where Adoption is the outcome Community was always a proxy for. Content that isn't structured for citation doesn't fire that new node, which is why teams see their content producing less community than it used to, with nothing in their metrics explaining why.
The channel is measurably real. AI platforms sent about [1.13 billion referrals to the top 1,000 websites in June 2025, up 357% year over year](https://techcrunch.com/2025/07/25/ai-referrals-to-top-websites-were-up-357-year-over-year-in-june-reaching-1-13b/), and AI-referred visitors [convert at roughly 4.4 times the rate](https://www.semrush.com/blog/ai-search-seo-traffic-study/) of standard organic search. Higher intent and faster growth, and most of it invisible in the dashboards DevRel teams report from.
Neon's number says something stronger than "there's a new referral channel," though. For most tools today the new node augments the old flywheel; at the leading edge it starts to replace the human path outright. At 80% agent-provisioned, Neon's flywheel isn't amplified. It's bypassed. The agent doesn't read your blog or join your Discord, and it has never attended a talk. It reads your integration surface and provisions the database. That's adoption without a single human touchpoint in the loop, the far end of the same dial, arriving early.
---
## Agent skills are the new SDK
The deepest version of this is the agent skill: a small, installable context package that teaches a coding agent how your tool works, what patterns to follow, and what mistakes to avoid. One install, and every agent-assisted PR in that codebase carries opinionated knowledge of your product. [One essay on the topic](https://www.battery.com/blog/agent-skills-are-the-new-sdk-and-you-should-be-building-one/) called it "a 10x solutions engineer for every single customer account, one that works on every PR, never goes on vacation, and never forgets the naming convention."
Product-led growth optimized the first five minutes, and Stripe's seven-line integration and Twilio's copy-paste quickstart won that era. Skills solve the second distribution problem: getting your tool instrumented correctly across an entire organization, forever. For usage-priced infrastructure that's revenue mechanics rather than marketing. As a first approximation, an account at 20% instrumentation coverage is leaving most of its potential billing on the table, and every PR the agent instruments correctly is incremental ARR without a new logo or a sales motion.
Skills are also a viral discovery channel that didn't exist before. A developer joins a team, opens their coding agent, and the agent surfaces the tools the organization already uses. Not through Slack or a wiki. Through context. Your installed base recruits for you.
---
## DevRel has always been distribution. Now it's legible.
DevRel has fought a losing battle for years over proving its pipeline contribution. [Mary Thengvall](https://www.marythengvall.com/devrelbook) built the "DevRel Qualified Leads" framework largely because the function's real influence on adoption was structurally invisible to attribution systems. Awareness compounded through talks and community into adoption nobody could trace.
Agent-mediated distribution flips that. Citation rates can be tested. Skill installs can be counted. Agent-provisioned signups can be instrumented, and Neon literally reported the percentage. The distribution work DevRel always did on faith is becoming distribution work you can put on a dashboard. That's an enormous political gift to every DevRel leader who's ever defended a headcount, and it comes with an obligation: if the impact is now measurable, you have to actually build the things that get measured.
That means the DevRel roadmap grows some unfamiliar line items: an agent skills library maintained with the same seriousness as an SDK, LLM context files treated as first-class deliverables, and citation monitoring as a standing program. I've made LLM context files an explicit OKR deliverable in my own advocacy work this year, not because it's fashionable but because that's the artifact the actual distribution channel consumes.
One warning about the window. From tracking the same first-mover dynamic in [local-business AEO](https://shane.logsdon.io/articles/strategic-insights/what-aeo-actually-means-for-a-local-business/), I estimate a 12 to 18 month gap there. For developer tools the window is shorter, because the elite tier of Stripe, Clerk, and Neon has already built the base layer, and one of them already got acquired for it. The advantage compounds with time spent in the agent's defaults. Supabase responded within months by [bringing in BKND](https://supabase.com/blog/bknd-joins-supabase) to build a lite backend offering for agentic workloads. The market is telling you what it thinks embedding in agent workflows is worth.
---
*Part 5 of 9 in the Developer Relations in the Age of AI series.*
*← Previous: Witnessed Practice Is the New Tutorial · Next: What Do We Measure? → · ← All articles*
---
# Witnessed Practice Is the New Tutorial
_Published 2026-08-12_
_1,010 words, about 6 min read._
*Witnessed practice (showing a competent developer work through real uncertainty, dead ends and recoveries included) is the DevRel format AI can't replicate. Demand for it is strong, and corporate DevRel produces almost none of it.*
Technical educators keep reporting the same thing: engagement is softer, course sales are softer, and people are less willing to pay for educational material than they were a year ago. Which is strange, because more code is being produced than ever. If developers are building more, why is the market for learning how to build shrinking?
[Sunil Pai's answer](https://sunilpai.dev/posts/developer-relations/), in an essay ("developer relations after the cheat code machine") I've reread more than anything else written about DevRel this year, is that people were never really buying courses to learn APIs. They were buying a way of working: how to structure things, debug, choose between options, ship, and notice when something is wrong even when it technically works. AI crashed into the layer below that. Every time you pull the lever on what he calls the cheat code machine, plausible code comes out. The API-knowledge layer collapsed in value, and the learning demand moved up to the layer AI can't produce: judgment in motion.
He gave the format that serves this demand a name. Witnessed practice means watching someone competent actually work: what they delegate, what they check by hand, where they trust the model and where they absolutely don't, and how they recover when the agent has confidently wandered off in the wrong direction.
I think this is the DevRel content format most likely to survive AI saturation, and I think corporate DevRel is producing approximately none of it.
---
## Tutorial vs. witnessed practice
| Tutorial | Witnessed practice |
|----------|-------------------|
| Polished path to a known destination | Messy navigation through genuine uncertainty |
| Explains what works | Shows what gets checked, skipped, inspected, questioned |
| Optimizes for task completion | Optimizes for pattern absorption |
| Writer already knows the answer | Writer is figuring it out |
| AI can generate it at volume | AI can generate text *about* it, but can't demonstrate it |
The distinction that matters is the last row. AI-generated content is optimized for plausibility and completeness. Witnessed practice is optimized for authenticity of process, and authentic process requires genuine uncertainty and genuine recovery, which is precisely what a text generator can't have about your product.
---
## The apprenticeship root
The format works because it restores something two separate shifts eroded. Pai tells a story about starting out in Hyderabad, sitting next to a senior engineer and copying everything. Not the code so much as how she debugged, how she moved around a codebase, and how she asked for help. Nobody makes a course out of "watch how this person narrows down a problem." You observe it, try it, and absorb it.
Remote work reduced that ambient apprenticeship. AI tools added a second layer of separation: many earlier-career engineers may be touching less of the raw material of the work, because the machine now generates so much of it. That leaves less osmosis at exactly the moment osmosis matters more. "Watch me work" content partially restores what's missing. [swyx's "learn in public"](https://www.swyx.io/learn-in-public) ethos was an early version of the same insight: showing your working process compounds trust in a way polished output doesn't.
---
## The supply gap
The demand side is visible. Look at what's still healthy while course sales sag. ThePrimeagen's streams work because ten years of performance-critical engineering is visible in the process rather than asserted in the bio. [Theo's](https://t3.gg/) judgment-first content ("why I chose X over Y," "I was wrong about Z") shows decision-making instead of conclusions. [Simon Willison's](https://simonwillison.net/) posts reconstruct entire debugging sessions, down to what he tried, what failed, and the prompt that finally worked. [Julia Evans](https://jvns.ca/) explicitly shows the confusion-to-clarity arc instead of the clarity alone.
Notice what that list has in common: they're all independents. I went looking for a company DevRel team that has systematized witnessed practice and found none. The closest things are individual advocates doing it personally, or "build in public" programs optimized for marketing rather than judgment transmission. The demand signal is strong, the supply from corporate DevRel is near zero so far, and the format is sitting there unclaimed.
---
## What it looks like in practice
The mechanics that separate witnessed practice from a sloppy stream:
**Narrate the decision layer, not the action layer.** "I'm going to try X" is narration. "I'm reaching for X rather than Y because the schema isn't stable yet" is judgment made visible. The judgment narration is the entire product.
**Include the dead ends.** A workflow showing only successful moves is a tutorial with a camera running. The backtracks and the "actually, no" moments are what make it witnessed.
**Make the recovery visible.** The moment the agent went wrong and you caught it, or didn't catch it immediately, is the highest-value moment in the format. It's more informative than any best-practices section you will ever write.
My own sample-project workflow is a small version of this. I write the first language implementation by hand, use an LLM to translate to the other languages, then validate with code spot-checks, compile-and-run tests, browser testing of the user flow, and a cross-language consistency pass. LLM passes for comments and README come last. Written down as a numbered list, it's a process doc. Narrated while it happens, with why the first implementation is by hand, what the spot-checks are actually looking for, and which language the translation mangles and how I notice, it's witnessed practice. It's the same workflow. The judgment narration is the difference.
"Here is how a thoughtful person uses this SDK in a real workflow, including the dead ends" is closer to apprenticeship than marketing. Which, as Pai points out, is maybe what good DevRel was always closer to anyway.
---
*Part 4 of 9 in the Developer Relations in the Age of AI series.*
*← Previous: The 25% That AI Can't Do · Next: DevRel Is Distribution Work → · ← All articles*
---
# The 25% That AI Can't Do Is the Whole Job Now
_Published 2026-08-05_
_895 words, about 5 min read._
*AI already handles the first 75% of developer education: syntax, boilerplate, quickstarts. What's left is DevRel's whole job now, the 25% of judgment AI can't do: architecture reasoning, tradeoffs, spec quality, and your product's sharp edges.*
When I estimate AI-assisted work, I've learned to do something that looks backwards: I over-invest time in the last quarter of the task. AI tools get you to roughly 75% of a working solution fast. Scaffolding, boilerplate, and the standard integration patterns generate in minutes. The remaining 25% is edge cases, domain judgment, and understanding why something breaks and what the right fix actually is. That part takes longer than the first 75% did, and it's where the actual value lives.
Addy Osmani called a version of this [the 70% problem](https://addyo.substack.com/p/the-70-problem-hard-truths-about), after watching non-engineers get almost all the way to working software and then hit a wall the tools couldn't carry them past. The exact percentages don't matter. What matters is the failure mode: treating the 75% as the finish line, then being surprised that the last stretch is the hard part.
I've used this as a planning heuristic for a couple of years. It took me longer to notice what it means for DevRel. If AI handles the first 75% of developer education, then most of what DevRel teams produce is competing with a machine that does it instantly, for free, inside the developer's editor.
---
## What the 75% covers
Be honest about the inventory: syntax questions, boilerplate, "how do I call this API?", getting-started guides, the standard auth flow, and the common gotchas that appear in every project. That's the bulk of most DevRel content calendars, and AI already handles it well. Not perfectly, but well enough that developers reach for the assistant before they reach for your tutorial.
That content isn't worthless now. It's table stakes. It needs to exist, structured so AI can consume and cite it, which is the AEO work from the last post. But it stopped being differentiation. Nobody builds preference for your product because your quickstart explained pagination clearly. The machine explains pagination clearly.
[Sunil Pai's framing](https://sunilpai.dev/posts/developer-relations/) is that the object of learning moved up a layer. People were never really paying to learn APIs. They were paying to learn how to work, and AI crashed into exactly the layer below that. What's left above is what he calls judgment and taste, and he's careful to insist taste isn't ornamental. It's operational. When an agent generates ten plausible solutions, the scarce skill isn't producing a solution. It's telling which one is brittle, which one hides complexity instead of removing it, and which one will be miserable to maintain.
---
## The 25%, translated to DevRel
Here's what the judgment layer looks like as a content strategy. These are the questions AI answers badly and developers still pay attention for.
**Architecture guidance.** Not "how do I structure my integration" but why to structure it a certain way. The reasoning behind API design decisions, the tradeoffs, the opinionated patterns an LLM can't reliably infer from documentation, because the documentation records the decision and not the deliberation.
**Answering "why?"** When I build sample projects, the scaffolding generates in minutes. The time goes into payment auth flows, error handling edge cases, and the places where the domain bites. A developer hitting those places doesn't need a tutorial. They need the reasoning of someone who's been bitten.
**Spec quality.** This one is newer, and I think underrated. The quality of AI-generated output is determined by the quality of the starting spec. Helping developers write better integration specs before they start building is leverage in a way it never was when humans wrote every line. DevRel used to optimize time to first API call. Time to good spec might matter more now, and a guide on how to write the integration spec before you prompt is a category of DevRel content that had no reason to exist two years ago.
**The edges.** Domain-specific judgment, integration nuance, the stuff that falls outside standard patterns. The machine is trained on the standard patterns. Your product's sharp edges are underrepresented in the training data almost by definition.
---
## The uncomfortable audit
Take your last quarter of DevRel output and sort it into two piles. One pile explains what the docs already say. The other transmits judgment: why-answers, tradeoff reasoning, edge-case navigation, spec guidance. For most teams the first pile is embarrassingly tall, and every month it becomes more redundant with what the developer's assistant already told them.
This isn't an argument for producing less. It's an argument about where differentiation moved. The first pile becomes structured, extractable, machine-legible reference that you build once and maintain well. The scarce, expensive advocate-hours go to the second pile, because that's the content AI can generate text about but can't actually produce. Producing it requires having made the judgment calls yourself.
There's a format question hiding here. If judgment is the product, what does judgment-transmission look like as content? The honest answer isn't a listicle of best practices, which is the 75% wearing a costume. It involves showing the judgment operating in real time, dead ends included. That format has a name, and it's the next post.
---
*Part 3 of 9 in the Developer Relations in the Age of AI series.*
*← Previous: AEO Is DevRel Infrastructure · Next: Witnessed Practice Is the New Tutorial → · ← All articles*
---
# AEO Is DevRel Infrastructure (And Nobody Owns It)
_Published 2026-07-29, updated 2026-08-09_
_1,859 words, about 10 min read._
*Answer engine optimization (FAQ schema, llms.txt, structured content, agent-facing signals) is DevRel infrastructure, not a content chore, and at most developer-tool companies nobody owns it yet. DevRel is the function best positioned to claim it.*
> **Update — 9 August 2026.** I re-ran this audit against the same six-layer stack. The headline finding below is no longer true: Stripe, Clerk, and Neon have each closed three more layers since May. The original text is preserved throughout for the record, with inline update notes marking what changed.
>
> | Layer | May 2026 | August 2026 |
> |---|---|---|
> | 1. Crawler access | no explicit policy | all three declare `Content-Signal` |
> | 2. llms.txt | all three | all three, now with per-section indexes |
> | 3. Skill manifests | absent | all three ship one |
> | 4. Content formatting | absent | `.md` twin on every page |
> | 5. Token surfacing | absent | **still absent everywhere** |
> | 6. Copy-for-AI | absent | all three |
> | **Total** | **2 of 6** | **~5 of 6** |
>
> One caveat worth naming, since it cuts against me: this piece published on 29 July but reported a May audit, so it was already two months behind its subject on the day it shipped. That lag is itself part of the argument — this layer is moving faster than the writing about it.
In May I audited three developer-tool documentation sites (Stripe, Clerk, and Neon), looking for the structural signals that determine whether AI assistants can find, extract, and cite their content. I picked those three because they're the sites everyone in DevRel points at when they mean documentation done right.
All three have an llms.txt. None of them have an explicit AI crawler policy in robots.txt. The differences between the three turned out to be more instructive than the similarities.
> **Update, August 2026:** the second sentence no longer holds. All three now carry `Content-Signal: ai-train=yes, search=yes, ai-input=yes` in robots.txt. Clerk goes furthest, blocking CCBot, Bytespider, YouBot, and Cohere-ai by name under a comment that distinguishes extract-only crawlers from the ones that cite and refer. Worth noting how it closed: Cloudflare shipping `Content-Signal` as a managed default did more in one quarter than the advocacy did.
Stripe's llms.txt reads like a product catalog. It's comprehensive and organized by product line, with use cases relegated to a secondary Solutions section, so a developer asking an AI "how do I handle subscriptions?" mostly needs the model to infer the mapping from catalog to use case. Clerk's is indexed by use case and framed educationally, with framework-specific content that lets an AI answer "how do I add auth to my Next.js app?" with Clerk specifically. Neon's is the most sophisticated of the three. It explicitly names AI builders as an audience and covers MCP integrations, pgvector, and agent tooling. It's written for the machine that reads it.
> **Update, August 2026:** the Neon assessment holds and has widened — its llms.txt now leads with a "Common Queries" section ahead of the product tree, and tells agents outright to append `.md` to any doc URL or send `Accept: text/markdown`. The Stripe characterisation is now incomplete rather than wrong: `stripe.com/llms.txt` is still a product catalog, but there is a second, larger file at `docs.stripe.com/llms.txt` that opens with a section headed "Instructions for Large Language Model Agents."
Here's the context that reframes the whole audit. Across nearly 300,000 domains SE Ranking analyzed, only about one in ten has *any* llms.txt. Set a stricter bar and the number drops further: among the top 10,000 sites, [barely 6% have a *valid, well-formed* one](https://caseyrb.com/blog/state-of-llms-txt-adoption/) as of mid-2026 (a different study, counting a different thing). Almost nobody has started. But the companies you actually benchmark against have: all three sites I audited shipped one. If you've been waiting to see whether this matters, they've already answered.
---
## If you've read my local business AEO work, I won't repeat it
I've [written elsewhere about answer engine optimization (AEO) for local businesses](https://shane.logsdon.io/articles/strategic-insights/what-aeo-actually-means-for-a-local-business/): the five structural signals (FAQ schema, Quick Answer blocks, llms.txt, open crawler access, extractable content) that determine whether an AI recommends you. Those signals apply to developer tools unchanged, so I won't re-teach them here.
What changes for developer tools is everything else. The query is "what should I use for serverless Postgres?" instead of "best plumber near me." The stakes are a competitor getting embedded in agent defaults. The part with no local-business equivalent at all is the second surface, where the consumer of your content isn't a human asking an AI but a coding agent working autonomously. That second surface is where the infrastructure framing earns its name.
The underlying migration lesson is older than the current AEO vocabulary: a site only keeps its authority when its structure survives change. I documented that pattern in [Migrate Your Site Without Killing Your Search Engine Presence](/archive/2013/migrate-your-site-without-killing-your-search-engine-presence/).
---
## The stack, and how much of it is missing
The most complete framework I've found so far for the agent-facing layer is [Addy Osmani's six-layer stack](https://addyosmani.com/blog/agentic-engine-optimization/) for what he calls Agentic Engine Optimization (the agent-facing sibling of the answer-facing AEO above, confusingly abbreviated the same way):
1. robots.txt access control
2. llms.txt discovery
3. skill.md capability signaling (a machine-readable manifest telling an agent what your tool can do and how to call it)
4. Content formatting: markdown, heading hierarchies, parameter tables
5. Token surfacing: publishing page token counts as metadata
6. "Copy for AI" buttons: clean markdown for context inclusion
Here's what my audit found: layers 3 through 6 were absent from all three sites. Stripe, Clerk, and Neon, the top of the market, are two layers into a six-layer stack. The field is that early. Your gap to the leaders is measured in months, not years.
> **Update, August 2026:** this is the paragraph the re-audit overturned. Layers 3, 4, and 6 are now present on all three sites. Stripe ships a `.well-known/skills/index.json` declaring seven agent skills, plus first-party plugins for Claude Code, Codex, and Cursor. Clerk ships a `SKILL.md` with a full CLI runbook. Neon ships a hierarchical `skill.md` that loads a parent skill, alongside a 6.2 MB `llms-full.txt`. Every one of the three now serves a `.md` twin for every docs page, and all three have a copy or ask-AI affordance in the docs chrome.
>
> Layer 5 is the exception, and it has not moved at all: not one of the five sites I re-audited publishes a token count, and Stripe and Neon publish no dates whatsoever — no `article:modified_time`, no sitemap `lastmod` across thousands of URLs. A model cannot tell whether any of it is current.
>
> "Two layers into six" became roughly five of six in about three months. That is a shorter window than "months, not years" implied, and if you were using this piece as a gap estimate, revise it down.
---
## Why this lands on DevRel
I went looking for DevRel teams writing about this work and found something strange: nobody is. Stripe, Clerk, and Neon built their llms.txt files silently, with no posts about who owns the work or why. Trade press has started discussing "AI DevRel manager" roles, but I couldn't find a single published example of a DevRel team claiming AEO as theirs.
> **Update, August 2026:** the silence broke, partly. Clerk has since published *Clerk for the AI era*, *Introducing Clerk CLI*, and a Series C announcement centred on agent identity; Stripe ships a public skills install page documenting how to add its skills to three different agent harnesses. These companies are talking about agent-facing work openly now. What I still could not find is a DevRel team framing it as their mandate — the narrower claim stands.
The organizational reality explains the silence. Look at who owns each signal in a typical dev-tool company:
| Signal | Typical owner | What actually happens |
|--------|---------------|----------------------|
| FAQ schema / JSON-LD | SEO team, if one exists | Not in DevRel purview; doesn't get done |
| llms.txt | Nobody | New artifact, no owner |
| robots.txt AI crawler access | IT / Infra | Treated as a security setting, not distribution |
| Agent skills library | Nobody | No standard owner; someone has to invent the mandate |
| Citation monitoring | Nobody | No tooling in the standard DevRel stack |
| Structured API reference | Docs team | LLM-readability isn't on their criteria list |
The work falls between teams, so it mostly doesn't happen. DevRel is better positioned to own it than any other function, because every input to the work is something DevRel already knows. What developers actually ask becomes the FAQ schema. What context an agent needs to use the tool correctly becomes the skills library. How products actually spread through developer communities becomes the trust signals AI assistants weight.
> **Update, August 2026:** the table still describes the gaps correctly, but the re-audit changed my view of the mechanism, and this is the part I'd write differently now. The layers that closed did not close because DevRel claimed them. Layer 1 closed because Cloudflare made `Content-Signal` a managed default. Layers 2, 4, and 6 closed because docs platforms shipped them as platform features — no docs team decided this. Layer 3 closed because all three companies ship a CLI or an MCP server, so the skill manifest attached itself to a product team that already had a mandate.
>
> The layers still open are exactly the ones with no vendor default and no adjacent owner: token metadata, JSON-LD at dev-tool companies (Stripe's docs pages emit none at all, while a one-person site out-schemas all three), and citation monitoring. "Nobody owns it" was right, but the useful version is sharper. The question is not who *should* own the work. It is whether the work can attach to an owner who already exists, or has to wait for a vendor to default it. DevRel's real opening is the second category, and it is smaller and more urgent than this piece originally implied.
The barrier isn't capability. It's framing. Most DevRel teams still think of themselves as producing content, meaning posts, talks, and sample projects. AEO is infrastructure work that happens to produce content artifacts. Until a DevRel team makes that mental shift, the work has no home.
---
## The baseline test
I ran a citation baseline on my own site earlier this year and got 0% on service queries and 100% on branded queries. Every AI assistant knew who I was when asked by name. None of them recommended me when asked what I do. That gap exists because the site was written for humans reading top to bottom.
The same test applies to your product, and it costs nothing. Ask ChatGPT, Perplexity, Claude, and Gemini: "what should I use for [your product's core use case]?" Not your product name, the use case. Count how many answers include you. That number is your baseline, and my bet is that nobody at your company has ever measured it.
Run the test. If you're in the answers, find out which content is earning the citations and protect it. If you're not, you have a rare thing in DevRel: a gap that's measurable, fixable with known techniques, and invisible so far to your competitors, nine in ten of whom haven't started.
---
*Part 2 of 9 in the Developer Relations in the Age of AI series.*
*← Previous: The Split Audience · Next: The 25% That AI Can't Do → · ← All articles*
---
# The Split Audience: The First Reader of Your Docs Isn't a Developer
_Published 2026-07-22_
_1,123 words, about 6 min read._
*AI coding agents read your documentation before any developer does, and no analytics metric you report captures it. Writing docs for the agent (clear structure, direct answers, an llms.txt index) is what keeps you in the answer, and it serves the human reader too.*
There's a sentence at the end of [a Sunil Pai essay](https://sunilpai.dev/posts/developer-relations/) from April that I keep coming back to. After several thousand words about what AI is doing to developer education, he drops this almost as an aside: "there's a whole other thread here about what happens when agents themselves become the primary consumers of your docs, APIs, and error messages, and when devrel has to be legible not just to humans but to the machines working alongside them."
He said he'd pull on that thread in a different post. I've been pulling on it for months, because it describes the thing my analytics couldn't show me.
Here's the shape of the problem. When a developer uses Cursor or Claude Code to build an integration with your API, the agent reads your documentation first. It fetches the README, the API reference, maybe an OpenAPI spec. It compresses what a human would experience as a twenty-minute reading session across a dozen pages into one or two HTTP requests. Then it makes decisions on the developer's behalf: which auth pattern to use, which endpoint to call, whether your product is the right choice for the task at all.
Your documentation got read, evaluated, and acted on, and no metric you report captured any of it.
---
## Two audiences, one set of artifacts
Software has walked this road before. Each wave re-centered design on whoever actually consumes the thing. UX redesigned interfaces around users rather than the engineers who built them, and DX did the same for APIs and the developers consuming them. AX (Agent Experience), designing your product's surfaces so an autonomous agent can consume them, is the third turn of that wheel, the term [Netlify's Mathias Biilmann](https://biilmann.blog/articles/introducing-ax/) put a flag in. It asks the question again, now that the consumer is a machine: is this system designed for the thing actually using it?
I've [written about AX from the engineering side](https://shane.logsdon.io/articles/technical-deep-dives/the-ax-shift/), where specs and context files become the communication layer between you and your coding agent. This post is about the other side: what AX means when the thing that changed is your distribution channel, not your development workflow.
Because the audience for developer documentation has split in two, and the halves want opposite things.
A human developer reads selectively. They skim, build a mental model across pages, tolerate narrative, and recover from ambiguity by inferring intent or asking a colleague. An agent parses for structure, and it does so under a token budget. One documented example: Cisco's firewall REST API quick start guide weighs [193,217 tokens](https://arxiv.org/abs/2604.02544), past the context window of many coding agents, which means parts of documentation at that scale are effectively invisible to them. The agent can't infer what you meant from what you almost said, nor can it ask a colleague. It fills every ambiguity with the statistically most probable answer, which is how a plausible-but-wrong integration gets confidently assembled from docs that were clear enough for humans.
Joey de Villa frames the stakes bluntly: when the docs aren't optimized for machine ingestion, the AI hallucinates the implementation, and the developer blames your product. The failure is yours either way. The agent never files a support ticket.
---
## The two surfaces
The split shows up on two distinct surfaces, and most DevRel programs I've seen are building for neither.
The first surface is a human asking an AI what to use. "What's the best way to handle auth in a Next.js app?" The assistant synthesizes a recommendation from content it can extract and cite. If your docs are prose written for top-to-bottom human reading, you may not appear in the answer at all, regardless of product quality or community size.
The second surface is an agent consuming your docs autonomously. The developer already chose you, or their agent did, and now the agent is reading your reference material to build the integration. Structure quality here determines whether the integration comes out correct, and whether the agent quietly recommends switching to a competitor whose docs it can actually parse.
The first is a discovery problem and the second is a correctness problem. Both are documentation problems, and both land on DevRel's desk whether or not anyone assigned them there.
---
## What breaks first: your instruments
The subtle casualty is measurement. When agents compress multi-page navigation into single requests, bounce rate, session depth, and page views stop meaning anything. A decline in docs traffic is now ambiguous. It could mean your content is failing, or it could mean your AEO (answer engine optimization, being the source an AI cites) is succeeding, because developers got the answer through an AI citation and went straight to integration without ever loading your site.
If you're reporting docs traffic to leadership as a health metric, you're reading an instrument that no longer measures what it used to. What to measure instead is a big enough question that it gets its own post at the end of this series.
---
## What to do about it
The good news is that writing for the agent doesn't mean writing worse for the human. Agents need clear heading hierarchies, direct answers near the top of pages, parameter tables instead of parameters buried in prose, markdown versions of your docs, and an llms.txt index (a plain-text map of your docs for crawlers) at your site root. Tired humans at 2am appreciate every one of those too. There is no bifurcation crisis here. Structure serves both audiences. Only vagueness had to choose.
The test I'd run this week, before any tooling investment: open a coding agent, point it at your quickstart, and ask it to build the hello-world integration. Watch what it fetches and where it guesses. Every guess is a place where your documentation was legible to a human and illegible to the machine that now reads it first.
Then ask the more uncomfortable question: when a developer asks ChatGPT what to use for your product's core use case, are you in the answer? That one, the discovery surface and the infrastructure behind it, is the next post.
---
*Part 1 of 9 in the Developer Relations in the Age of AI series.*
*Next: AEO Is DevRel Infrastructure → · ← All articles*
*Related reading: [The AX Shift: You're Still Designing for Yourself](https://shane.logsdon.io/articles/technical-deep-dives/the-ax-shift/): the engineering-side treatment of the same consumer change.*
---
# Building on the Margins
_Published 2026-07-13_
_1,881 words, about 10 min read._
I have a full-time job. A family. And I'm building [LeadSurface](https://www.leadsurface.com), a real product with real customers who expect it to get a little better every week.
Something has to give. For a long time I figured it would be the product. You can't build software well in the forty-minute windows between a meeting and dinner. Context dies the moment you're interrupted. Come back to the code three days later and you burn twenty minutes remembering what you were doing and why.
It's not just the code that evaporates. The ideas go too. A fix for a nagging bug, a feature a customer would love, the right wording for an onboarding email, they show up at red lights, in the shower, halfway through a meeting you can't leave. Without a place to capture them, they're gone by the time you're back at the keyboard. On the margins, most of the work is catching the thought before it slips, so a spark at 2pm reaches the forty-minute window at 9.
What I've learned is that interruption kills the discipline, not the code. When I'm tired, or I've only got half an hour, or I'm holding four other things in my head, I skip the steps that matter later. I don't write down what I'm building before I build it. I tell myself I'll test it later. I make a call at 10pm and can't reconstruct why by the weekend. None of that is a skill problem. It's an attention problem.
So I stopped trying to fix my attention and started externalizing the discipline. That's my dev workflow. Not a stack of clever AI tools, but a delivery team that never forgets context, so my attention goes to the parts only I can do.
The same split works for a local business owner: keep the judgment about the business and hand the repeatable web-presence work to a system. That is the model behind my work with [local businesses](/local-businesses/).
## What I keep, what I hand off
This is the distinction that makes the whole thing work. There's work on LeadSurface that's genuinely mine and can't be delegated, and it comes from two careers I've already had. As a product manager I know the market, I know the customer, and I can tell a real problem from a loud one. As a developer I know when an architecture is sane, when a test actually proves something, and where a dollar of infra is worth spending. That judgment took years on both sides of the table, and no model has it. It's why the product is worth building, and why I can tell when the pipeline is quietly wrong.
Then there's the second category: the disciplined execution of turning a decision into shipped, tested, reviewed code. That work is real and necessary, but it doesn't need me specifically. It needs someone who won't cut corners and won't lose the thread between sessions. That's exactly what I'm bad at on the margins, and exactly what a skill pipeline is good at.
So I split it. I stay the strategist and the expert. The pipeline is the delivery team.
But the handoff is never total, and that's the part people miss. The pipeline runs the stretches between decisions. It doesn't make the decisions. There are fixed points where it stops and waits for me, and those points are the actual job. I call the whole approach Loop & Gate: an autonomous loop with a fixed set of human gates on it:
- Should this exist at all? Kill or build, before a line of code gets written.
- Who's it for, and what would make it a win? The context everything downstream inherits.
- Of the directions on the table, which fits the strategy?
- Is the plan actually right, and what did it miss?
- Is the architecture sane, or clever in a way that'll hurt in six months?
- Is the change going off the rails, or making a call the plan left open?
- Does the test actually prove it, or did the agent run it once and call it done?
- Is the cost right, or is it reaching for the biggest model on every call?
- Is the risk that's left acceptable to ship?
- Is now the moment to ship, and who should see it first?
Underneath all of those is the one that never automates: how much of this process does this particular change even deserve? A typo fix and a new data pipeline don't get the same treatment, and telling which is which is judgment, not process.
Some gates are business calls, some are engineering calls, and that split is why both careers earn their keep. When the pipeline says a change is tested, the product manager in me asks whether we tested what the customer actually cares about, and the developer in me asks whether the test proves anything at all. There's a real difference between the agent running a feature once in chat and announcing it works, and a deterministic browser test that fails the same way every time the feature regresses. One's a demo. The other's evidence. Cost is the same story, the agent will reach for the biggest model on every call if you let it, and it takes someone who's read the bill to route cheap work to cheap models and save the expensive ones for where they earn it.
I'll say the quiet part though: I don't have every lens either. My weak side is go-to-market. I've done some of it, but I'm no expert, so I work those gates like someone with no engineering background works the technical ones. I make the agent lay out the options in plain terms, I decide against the structure instead of a gut I haven't built yet, and I get a little better every time I ship through it. That's what I'd tell anyone holding one hat, or none. The gates don't ask you to already know both sides. They tell you which side each decision needs, so you know when you're on home ground and when to slow down and let the structure carry you. The one thing that never works is using the agent to skip the gate instead of to work it. That just gets you to the wrong answer faster.
## The pipeline, in one breath
I compose the workflow out of skills, small, focused procedures I can invoke by name inside Claude Code. A feature moves through roughly six stages:
1. **Business context.** Before anything else, I capture who this is for, what the opportunity is, and how I'd know it worked. This is the one step that's pure me, the market and customer knowledge going in on the record so everything downstream inherits it.
2. **Brainstorm.** Explore the problem and pick a direction, with that business context as the frame.
3. **Plan.** Turn the direction into an actual implementation plan, file paths, steps, how each piece gets tested.
4. **Build.** Isolated branch, one small change at a time, a test before each. When something breaks, a systematic debugging pass instead of guessing.
5. **Validate.** Every "it's done" claim has to be backed by evidence, not vibes. Then a focused code review, a security pass, and live testing in a browser.
6. **Ship.** A launch checklist, then merge and clean up.
Written out it looks heavy. In practice it isn't, because I don't run all six every time. More on that at the end. The point of the structure isn't ceremony. It's that each stage holds the context so I don't have to. I can walk away after the plan and come back cold two days later, and the plan's still there, still exact. The workflow remembers. I don't have to.
## What it looks like on a real feature
Say a LeadSurface customer tells me the lead export is missing a field they need. Old me would open the code, add the field, ship it between two other things, and half the time introduce a bug I wouldn't notice for a week.
Now the request goes in as business context first: who's asking, why it matters, what "fixed" means to them. That framing is mine. Then I brainstorm the actual shape of the change and write a plan. And here's the part that matters for a fragmented life: I can stop there. The plan is a durable artifact. When I next get a window, I don't reconstruct anything. I pick up the plan, spin an isolated branch, and build it one tested slice at a time. Before I call it done, the verification step makes me actually prove it works, with a browser test that'll catch the regression next time, not a one-off I eyeballed once. Then review, then ship.
The customer feedback loop tightens because the mechanical cost of acting on feedback drops. I'm close to the customer, which is the whole advantage of a solo operator, and the pipeline means being close turns into shipped changes fast instead of a backlog of good intentions.
## Why it's three toolkits, not one
No single plugin covers idea-to-ship, so I compose three, and where the seams fall is deliberate.
The core loop of brainstorm, plan, build, test, and review comes from one plugin. It's the spine. But it leaves gaps: no dedicated security pass, no live browser driving, no launch prep. So a second toolkit fills exactly those three holes and nothing more. A third layer, my own custom skills, provides the parts no general toolkit could: the business-context step up front, and domain-specific build skills for the stacks I actually work in.
I didn't design that split top-down. I reached for whatever filled the next hole, and this is where it settled. The lesson isn't the specific tools, those will churn. It's that the workflow is composed from what's missing, not adopted wholesale from one vendor.
## Where it's still rough
I want to be honest about the limits, because the tidy version of this is a lie.
I don't run all six stages every time. It's a menu, not a mandate. A one-line copy fix doesn't get a spec and a security pass. The judgment about how much process a change deserves is itself unautomated, it's still me, reading the stakes.
And the pipeline is only as good as the context I feed it. The business-context step is load-bearing. When I rush it, everything downstream confidently builds the wrong thing, fast. The tooling doesn't save me from being wrong about the customer. It just makes me wrong more efficiently, which is another reason the strategy and market work stays mine, and stays first.
That's the trade I've settled into. The pipeline carries the discipline and the memory across the gaps in a full life. I carry the judgment, from both the product side and the engineering side. LeadSurface gets built in the margins, and the margins are enough.
---
*[LeadSurface](https://www.leadsurface.com) is competitor-switch intelligence for revenue teams. It reads developer and SaaS communities and surfaces high-intent switching signals while the conversation is still live.*
*This piece sits alongside my [series on agentic product development workflows](https://shane.logsdon.io/articles/agentic-workflows/), which goes deeper on the specifics: specs, evaluation, and the workflow itself.*
---
# The Framework Emergence Loop: How Durable AI Workflows Are Actually Found
_Published 2026-07-09_
_1,339 words, about 7 min read._
## The framework emergence loop: How durable AI workflows actually get found
I rebuilt it for the fifth time and thought, "Something is wrong with me."
It was an AI agent framework, a system for managing context, encoding agent behavior, and keeping useful state across sessions. Version one solved the problem it was built for. Version two fixed what version one broke. Version three introduced the structural idea that made everything click. Version four folded in six months of real-world use. Version five existed because version four had developed a failure I couldn't patch my way out of. I had to rebuild around the lesson.
Looking at version five, I figured maybe this was finally the one.
It wasn't. There's a sixth now.
Somewhere in the not-giving-up, my thinking about the rebuilding changed. It stopped feeling like proof the last version failed and started looking like the actual mechanism for getting better. The loop isn't the symptom. It's the process.
---
## Premature convergence in AI frameworks
This has a name in software: premature convergence, the urge to lock a design down before it's met real conditions. The classic case is API design. A team ships a clean v1 that solves the known problems, then finds in production that the known problems were never the real ones. Then they spend six months either bolting on compatibility shims or admitting v2 has to exist. Auth, state management, build tooling, the library world has done this over and over. Every version looks done until production reveals the failure modes that force the next one.
AI agent frameworks hit this harder and faster than most software. Session context collapse, agent drift, persistence gaps, rigidity under weird inputs, none of that shows up in design. It only appears when the framework is doing real work across real sessions. You can't see it all in advance. You have to run into it.
---
## Six iterations and what each taught
The six-version chain started with a project called `gsd-planner`. The core insight was multi-stage structure: Research, then Plan, then Implement. Good enough to validate. Version two, `gsd-planner-2`, learned that cross-session state has to be explicit, not implied. When the agent lost its thread between sessions, nothing could recover it, so the session restarted and everything established was gone. Version three, `gsd-planner-3`, introduced the `knowledge-graph.json` pattern as a deliberate approach to persistent memory: a structured JSON file the agent reads at session start and updates as it goes. That was the breakthrough. Not elegant, not complete, but the structure was right, a stable artifact that outlives context windows.
Version four, `product-work`, made the PRD the source of truth: the spec the agent works against instead of reconstructing from chat history. Version five, `product-work-2`, added skills as installable units and a plugin marketplace, a way to ship agent knowledge as discrete, reusable pieces instead of one monolithic config. Version six, `claude-code-config`, added role-aware agents and drift detection: noticing when an agent's behavior slipped from its defined role and pulling it back before the slip compounded.
Each version taught one lesson the previous one couldn't have seen coming. That's not a design failure. That's how you find the lesson. The artifacts that emerged, persistent context files, authoritative specs, installable skills, are the same ones Posts 2 and 3 present as first principles. The difference is this chain had to break first to understand why they were necessary.
---
## The myth of getting it right the first time
Here's the misconception about this kind of iteration: that a team with enough experience could design the right framework upfront and skip the cycle. That's the wrong takeaway from the right observation, that each version's failure looks obvious in hindsight. You need persistent state. Role drift needs detection. The spec should be authoritative. These are obvious after you've built a system without them and watched it break exactly where the absence said it would. They are not obvious before.
The failure mode isn't rebuilding. It's refusing to rebuild once the lesson is clear, instead polishing version four past the point where a new architecture would do better, because the sunk cost makes starting over feel like admitting defeat. That instinct is expensive. The framework that survives is built by people who can see when they've hit the structural ceiling of the current version and choose to build the next one.
---
## The knowledge-graph pattern
The most transferable idea from this chain is the `knowledge-graph.json` pattern: a structured file the agent reads at session start and updates as it works. It's the closest practical thing to persistent memory across context windows. Not a database or vector store, a human-readable JSON file with a defined schema, committed alongside the code, loaded by the agent as its first move each session.
What makes it durable is that it outlives the failure it was built to address. When a session ends mid-work, the knowledge graph keeps what got established: decisions, context, completed tasks. The next session starts there instead of from nothing. The sessions compound instead of restarting.
The pattern got refined across iterations. The schema changed, the update protocol changed, what's worth persisting changed, and the core idea survived all of it. That's usually the test of whether something is a real structural idea or the most convenient fix for last week's problem.
The production form isn't a JSON file anymore. It's an Obsidian vault: connected notes are the nodes, agents update them across sessions to capture decisions and concepts, and the graph persists not just across context windows but across tooling changes. The JSON file was the prototype that proved the structure. The vault is what it became when it had to survive real use.
The move from prototype to production has a long history in software work. My earlier piece on [Application Deployment](/archive/2014/application-deployment/) makes the same case from a different generation of tools.
Look at what the `knowledge-graph.json` pattern actually encodes and you see the loop doing something more specific than generic software refinement. The schema isn't for a human to read later. It's for an agent to load at session start and act on immediately. Every field exists because the agent's next decision depends on it, not because the info is nice to have written down. That's AX thinking in its most concrete form: the artifact is built for the agent's navigation, not the developer's convenience. What the loop teaches, version by version, is what an agent needs to move through your system without guessing. Harder than it sounds, and only answerable by watching the previous version fail.
---
## What this means in practice
For teams building AI-assisted workflows: your iteration budget is not a failure budget. When an internal framework for AI-assisted code review breaks on edge cases you didn't see coming, that's the system telling you what the next version needs. The discipline is writing down what broke and why, specifically, not vaguely, so the next build starts from that knowledge instead of rediscovering it.
For developers building these systems, instrument the failure modes. When the agent loses its thread, when drift happens, when context collapses, capture that as signal, not noise to filter out. The failure modes are the spec for the next version. A team that can say "our current system breaks this specific way under these specific conditions" is already most of the way to the next version.
---
The frameworks that work in production AI workflows weren't designed to work. They were found through a series of deliberate failures, each one surfacing the lesson that made the next version possible. The loop isn't a symptom of something going wrong. The loop is how something good gets built.
The answer has rarely been better prompts. It's been better decisions, made earlier, written down somewhere the agent can find them. If you're starting from zero, that's where the work begins.
---
*Part 6 of 6 in the [Agentic Product Development Workflows](/articles/agentic-workflows/) series.*
*← Previous: [How to Know If Your Agentic Workflow Is Actually Working](/articles/technical-deep-dives/evaluating-agentic-workflows/)*
---
# How to Know If Your Agentic Workflow Is Actually Working
_Published 2026-06-18_
_1,719 words, about 9 min read._
You ship the workflow. It runs. The output lands in the right place, formatted correctly, and there are no errors in the log. You move on. Two weeks later, someone mentions that the summaries have been missing caveats, that the code review has been passing style issues, or that the documentation looks right but is citing an API that changed a month ago. The workflow never broke. It has been wrong the whole time, and you did not catch it because there was nothing to catch—only clean runs and accumulating drift.
The wrong workflow is harder to find than the broken one. A broken workflow is loud: the JSON parse fails, the output file is empty, or the downstream step throws an exception. There is a notification, a log entry, or something else to chase. The wrong workflow runs cleanly and produces output that is approximately what you intended but not quite, and the gap accumulates. For example, a summarization workflow might consistently omit caveats because the prompt did not specify that they were required. A code review workflow might pass style issues because the style guide was written for human reviewers rather than model evaluators. Or a documentation generator might produce accurate-looking content with outdated API references because no session context told it what had changed.
These do not break; they degrade. This degradation is invisible until someone asks why the output quality has been slipping, at which point the answer is that it has been like this for a while.
The distinction matters because the evaluation strategy for a broken workflow (detect failure, escalate, and retry) is completely different from the strategy for a wrong workflow (define correctness, measure against it, and maintain the signal). In the wrong workflow, the agent did not fail. Instead, the artifacts it was navigating failed it.
---
## Broken vs. wrong: two different failure modes
There is one principle worth establishing: you can only evaluate output against something authoritative. This sounds obvious, but its implications are often skipped. A workflow without an explicit definition of what "correct" looks like cannot be evaluated. It can only be spot-checked, which is not evaluation. Spot-checking catches catastrophic failures, but it does not catch drift.
What "authoritative" means in practice depends on the workflow type. For schema-critical workflows, where downstream steps depend on a specific output format, authoritative means a JSON schema or a structural contract.
At Global Payments, the developer advocacy team runs a monthly Power Automate pipeline that reads inbound developer emails, categorizes them by theme and urgency, and delivers structured insights to the product teams responsible for APIs and SDKs. This process takes a month of developer feedback—such as questions, integration friction, feature requests, and complaints—and turns it into something product teams can act on.
The flow started as one prompt handling the analysis and producing three artifact outputs. The results were inconsistent. Category labels drifted between runs, insights varied in structure depending on the volume and mix of that month's emails, and output could not be reliably parsed by downstream consumers. Splitting it into four single-responsibility prompts (one owning the categorization pass and one owning each artifact type) made the output stable enough to be useful. That is a schema-critical workflow where the gate is structural. Either the output matches the schema or it does not. Structural validation at the parse step catches failures before they propagate.
For quality-first workflows, where semantic accuracy matters more than rigid structure, authoritative means a rubric. This includes ensuring required sections are present, tone is within a specified range, and claims are checked against source material with acceptable variance defined explicitly in the prompt rather than inferred by the evaluator. A README generator that produces accurate-looking content in varied section orders is a quality-first workflow. The gate is semantic. A second LLM call (using a different model from the one that produced the output) compares the output against the rubric and returns a quality signal. Defining those criteria is itself an artifact design decision. Without a precise spec of what "correct" means, both gates reduce to spot-checking.
---
## Evaluation is downstream of speccing
There is a misconception in how teams typically think about evaluation: that it is a QA step added after the workflow is built, such as a quality gate at the end or human review before delivery. This framing treats evaluation as downstream of production, implying it catches problems after they occur.
A more useful frame is that evaluation is downstream of speccing. You can only build an automated gate if you have something to evaluate against. The schema that enables structural validation exists because a design document named the required fields and their types. The rubric that enables semantic evaluation exists because a prompt author defined what "correct" output looks like before building the prompt. The 60-day retrospective that measures time saved against a manual baseline works because someone documented the manual baseline before automating. Evaluation without a spec is spot-checking, which is what the AX thesis has been arguing against since Post 1. You can only measure against something authoritative, and that authoritative thing only exists if you designed for the agent rather than yourself.
This explains why evaluation collapses for workflows built without a spec. There is nothing authoritative to compare against, only output that feels approximately right. This leads to spot-checking that misses drift until the drift is significant enough to be noticed. Drift does not mean the model is degrading; it means the artifacts the model navigates are drifting from the intent they were meant to encode.
---
## The self-audit layer
The self-audit layer is where this becomes concrete. It has two components, both of which run before a human sees the output.
The first is structural validation. This checks if the output matches the expected schema or structure. For schema-critical workflows, this is a hard gate that determines if the output is valid or invalid. For quality-first workflows, it checks for the presence of required sections. This runs without an LLM, is deterministic, and is the cheapest signal available.
The second is plan-versus-output comparison. This checks if the output matches the intent of the prompt. A second LLM call, using a different model from the one that produced the output, compares the original prompt intent against the actual result on two dimensions: whether the agent followed the intended approach (rather than improvising in ways that deviate from the specified process) and whether the artifact meets the quality bar for its type. Both components run before escalation. Escalation happens when one or both fail after the retry limit.
The retry limit for most workflows is three to five attempts. After you exhaust retries, escalate. Do not fail silently. For automated flows, this means notifying the right people through the right channel. For interactive flows where a human is already in the loop, the error surfaces in real time.
---
## How to define "correct" before you ship
Building useful evaluation is harder than it looks. It requires defining what "good" looks like before the workflow exists, not after it has been running for two weeks. The prompt that produces high-quality output on the first 10 runs without an explicit quality definition will produce inconsistent output on runs 11 through 40 because there is no stable reference. The evaluator, whether it is a schema validator or a second LLM call, needs to know what it is comparing against. That definition must be written before the workflow ships.
In practice, this often surfaces gaps in the original design. "The output should accurately summarize the customer feedback" is not a rubric. However, "The output must include: the top three feature requests by frequency, any safety or compliance concerns flagged by customers, and the overall sentiment distribution; the summary must not introduce claims not present in the source material" is a rubric. The difference is evaluability. The first version can only be spot-checked, while the second version can be evaluated at scale and automatically on every run.
---
## The 60-day evaluation
After 60 days of production use, run a minimal retrospective covering these areas: what the workflow does, how much time it saves against the documented manual baseline, the error rate at the automated gate, the escalation rate, what held up that was not expected to, what failed in ways the design did not anticipate, and a recommendation. You can then choose to continue as-is, adjust the prompt, adjust the gate, retire the workflow, or expand scope.
The manual baseline matters more than teams usually invest in it. To measure time saved credibly, you need the baseline before automating. Take three manual runs, measure the wall-clock time from trigger to delivered artifact, and average them. Without that, the time-saved metric is an estimate that feels more authoritative than it actually is. While that is fine for an internal conversation, it is unreliable as a signal for deciding whether to expand the workflow.
Graduation from active human review to automated-gate-only requires a deliberate sign-off rather than a timer. Time-based and error-rate signals can inform the decision, but they do not make it. The stakes differ enough across workflows that a policy threshold would either be too conservative for simple ones or too permissive for high-consequence ones. The sign-off is the mechanism.
---
## Where to start
The practical starting point for a workflow you are building now is to write the definition of correct output before you write the prompt. Do not write a vague description; instead, write a specific rubric that a second evaluator could apply without your explanation. Build the structural validator first. Build the semantic evaluator after you have 10 outputs to test it against. Document your manual baseline before automating. The evaluation infrastructure takes longer than the prompt. That is expected, and it is not optional if the goal is something you can trust rather than something you can demonstrate.
---
Running without errors is not a success criterion. The true criterion is output that matches a defined standard consistently across the conditions your workflow actually encounters. The gap between those two things is where the wrong workflows live—running cleanly and degrading quietly until someone asks why.
---
*Part 5 of 6 in the [Agentic Product Development Workflows](/articles/agentic-workflows/) series.*
*Next: [The Framework Emergence Loop: How Durable AI Workflows Are Actually Found](/articles/technical-deep-dives/framework-emergence-loop/) →*
---
# What AEO Actually Means for a Local Business
_Published 2026-05-12_
_752 words, about 4 min read._
> **Quick answer:** Answer engine optimization (AEO) is the practice of structuring your web site so AI assistants (ChatGPT, Perplexity, Google's AI Overviews) can extract and cite your business when someone asks for a local recommendation. Unlike traditional SEO, it's not about ranking on a results page. It's about being the answer.
I ran a search last month for "best financial advisor near me" and then asked ChatGPT the same question.
Google gave me ten results. ChatGPT gave me three names. None of the three were advisors I know — people who are genuinely excellent at what they do, with strong reviews and a decade of client relationships.
They hadn't thought about how AI assistants surface recommendations. Most local business owners haven't.
---
## What's actually happening
When someone types a query into ChatGPT, Perplexity, or Google's AI Overview, they're not getting a list of links. They're getting a synthesized answer. The AI is reading available content, extracting what's useful, and presenting a recommendation. This often happens without the user ever clicking through to a web site.
For local service businesses, this matters more than most SEO advice will tell you. AI-assisted local search is already answering "best [service] in [city]" queries at scale. Gartner estimates a 25% shift in organic traffic toward AI chatbots by 2026. Google AI Overviews appear in roughly 68% of local searches.
Most local business web sites were not built for this. They were built for humans who read top to bottom. AI assistants don't read that way. They parse.
---
## What AEO actually requires
AEO isn't a rebrand of SEO. There are specific structural signals that determine whether an AI assistant can cite your business accurately. Most local sites have none of them.
**FAQ schema markup.** FAQPage JSON-LD is the highest-citation-rate structured data type for AI assistants. It tells the AI exactly what questions your business answers and in what words. A site without it is forcing the AI to guess.
**Quick Answer blocks.** A 40–60 word direct answer at the top of each page or section. This is the format AI assistants pull from when generating responses. If your content isn't structured this way, you're invisible to the extraction layer.
**llms.txt.** A file at the root of your site that tells AI crawlers what the site is about, who it's for, and which content matters. Without it, crawlers are inferring — and inferring wrong more often than you'd think.
**Open crawler access.** Your robots.txt needs to explicitly allow GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and the other AI crawlers. Many sites inadvertently block them with overly broad rules.
**Structured, extractable content.** Prose written for human reading is hard for AI to parse. Content structured around clear questions and direct answers is easy to work with. This doesn't mean writing worse content — it means writing it in a way that works for both audiences.
---
## Why it matters now
The window for first-mover advantage here is real and limited. Most local web agencies haven't adapted their service offerings to AEO. DIY builders produce static artifacts that fail every one of these checks. Most individual practitioners haven't heard of llms.txt.
That's a 12–18 month opening. The businesses that get structured for AI citation now will have a compounding advantage as AI-assisted search grows — citations accumulate, patterns reinforce, and later entrants pay a higher price to break in. The ones that wait until it's table stakes will be playing catch-up against that history.
The technical bar isn't high. For most local sites, full AEO implementation takes a few hours of structured work. What it requires is knowing which signals matter and building them correctly, not uncritically following a plugin setup and hoping.
The technical bar is low enough to handle as a focused web-presence system rather than a plugin checklist. I describe that implementation work on the [services page](/services/).
---
## What this looks like in practice
I added AEO signals to my own site as part of a baseline audit before opening this service to clients. The work came to four things: llms.txt at the root, FAQPage schema on the services page, Quick Answer blocks on key content, and Article schema on every post. About four hours total.
Before those changes, an AI assistant asked what I do gave back nothing useful. After, it gets it right.
That's the test. Not a ranking report. Not an analytics dashboard. Ask ChatGPT or Perplexity: "who would you recommend for [your service] in [your city]?" If your name isn't in the answer, you have an AEO gap.
---
# The Specification Boundary: Why AI-Assisted Builds Stall at 75%
_Published 2026-05-18_
_1,305 words, about 7 min read._
The demo worked. I had run it three times, walked through the flow twice, and it did exactly what I'd described. The scaffold was clean, with components wired up, API calls returning data, and the basic interaction pattern intact. By any reasonable interpretation of "working," it was working.
Then I tried to ship it.
The authentication edge case that only appeared with expired tokens turned out to be one thing. The schema field that was optional in the prototype turned out to be required by the consuming service. The error state that the UI handled gracefully on the happy path handled catastrophically in the one case that actually matters in production. The performance characteristic that was invisible at demo scale became obvious at real scale. None of these were complicated problems. They were the 25% of the problem that the first 75% hadn't touched yet.
The uncomfortable part is that the 75% felt like 95%. The demo looked complete, the code compiled, and the tests I'd written passed. The AI had been fast and capable and had produced something that genuinely worked. The 75% mark doesn't announce itself. It looks, from the inside, a lot like being nearly done.
---
## Why the 75% mark feels like the finish line
Much has been made of this shape. Addy Osmani called it the 70% problem: AI gets you most of the way, then the returns fall off a cliff. The observation is right and the number is close, but the usual diagnosis is not. The stall gets framed as a skill gap, or as a capability ceiling, as if the model runs out of road at some threshold. That's not what I keep hitting. What I keep hitting is a specification boundary, and it's the convergence point for the discipline Posts 2 and 3 described. The spec-first work from those posts exists partly for this reason: without an authoritative spec, the 75% mark looks like completion, because there's nothing to tell you what's still missing.
The scaffolding phase is fast. AI tools generate boilerplate, translate requirements into code structure, and handle the high-frequency patterns, including CRUD operations, standard API integrations, and component wiring, with impressive accuracy. The speed creates a real benefit: you can validate an approach in hours rather than days.
What it also creates is a particular calibration error. When the first 75% takes two hours, the temptation is to assume the remaining work is proportional. It almost never is.
---
## Where domain judgment lives
The last 25% of most software problems is where the domain-specific judgment lives. Which failure modes are distinct enough to handle separately, and which can be collapsed into a single error state? What does "incomplete" look like versus "failed," and do those mean the same thing to the downstream consumer? What's the correct retry policy for this specific rate limit on this specific API, not in general but here?
These are not questions a language model answers well from general training data. They're questions about your system, your constraints, your users, and your operational context. The AI has no access to those things except through what you tell it explicitly.
---
## The actual problem isn't capability
There's a common misconception embedded in how teams talk about this stall: that the issue is the AI stopping, as if the model runs out of capability at some threshold and the human has to take over. That's not what's happening. The model can generate code for authentication edge cases, error handling, performance optimization, and integration nuance. What it generates will be plausible, it will compile, and it will address the obvious version of each problem.
The actual issue is that plausible and correct diverge as you approach the boundary of what's specified. In the scaffold phase, the specified territory is large: "build a component that fetches user data and renders a list" has enough signal for a capable model to produce something useful. In the finishing phase, the territory is narrow and domain-specific: handle the case where the API returns a 206 Partial Content for this endpoint in particular, because the consuming client has a race condition in its retry logic that our retry policy needs to account for. That's not a prompt the model can answer from training data. It's a prompt that requires information the model doesn't have.
The 75% problem is not a capability ceiling. It's a specification boundary. The model works reliably within the specified territory and infers, plausibly but not correctly, outside it.
---
## How spec quality changes the shape of the last 25%
This is where the connection to spec quality becomes concrete. A complete design spec that names failure modes, edge cases, integration constraints, and operational requirements doesn't only improve the first 75%. It also extends the specified territory into the last 25%. Exit codes named for their specific shell automation context rather than numbered generically. Retry policy bounded to a specific number of attempts with specific backoff behavior rather than "handle transient failures." Authentication behavior described for both the token-valid and token-expired path rather than assuming the happy path.
When the spec covers that territory explicitly, the AI can operate in it reliably. When the spec leaves it open, the AI fills it with probability, and the 25% that was already hard becomes harder, because you're now debugging guesses rather than implementations.
Teams that front-load this work consistently find that the last 25% takes about the same amount of time, but looks different: it's integration, verification, and alignment work rather than discovery and correction. The total time shifts, but the distribution of effort shifts more, away from unexpected rework late and toward planned specification early.
---
## The part that's harder than it looks
The specification work that would close the 25% gap is often the work you don't know to do until you've seen the 75%. The authentication edge case isn't obvious until the prototype is running against a real auth service. The schema misalignment isn't visible until the consuming script tries to parse the output. The performance characteristic isn't measurable until you have real data volume.
This is a genuine constraint, not a failure of planning discipline. The practical response is the alignment pass: after the first implementation, compare the output against the spec, identify where the spec was silent on things that turned out to matter, update the spec, and close the gaps with a second pass. It's not a failure to find gaps here. That's the mechanism you need. The spec is authoritative enough to audit against, and the audit is where the 25% gets addressed deliberately rather than discovered accidentally in production.
---
## Planning and evaluation heuristics
As a planning heuristic: when estimating AI-assisted work, don't discount the last 25%. If anything, add time there rather than removing it, because that's where domain expertise matters most and AI assistance is least reliable. The speed you gain in the scaffold phase is real. Treat it as time freed for the finishing work, not as a reason to shrink the overall estimate.
For developers evaluating AI tools: the question to ask is not "how much code does this generate?" but "how far into the specified territory can this operate reliably?" A tool that generates excellent scaffold code quickly is valuable. A workflow that ensures the specified territory extends into the domain-specific finishing work is what turns that value into shipped software.
The 75% mark is not a failure. The scaffold is real, the speed was real, and the AI did exactly what it was specified to do. What it couldn't do is fill in what the spec left out, and in the last 25%, what the spec left out is everything that requires knowing your system, your constraints, and your users. That's not a gap the model can close. That's the work.
---
*Part 4 of 6 in the [Agentic Product Development Workflows](/articles/agentic-workflows/) series.*
*Next: [How to Know If Your Agentic Workflow Is Actually Working](/articles/technical-deep-dives/evaluating-agentic-workflows/) →*
---
# The Spec Is the Work: PRD-First AI Development
_Published 2026-05-11_
_1,517 words, about 8 min read._
I thought I was watching the model fail. The output was close, functionally close and in some cases working, but off in specific ways I kept having to patch. The error handling didn't distinguish between failures that should halt the program and ones that should retry. The schema had fields that made sense in isolation but didn't align across the three scripts consuming the output. The CLI surface was intuitive in a generic sense but wrong for the specific automation I was building it for.
I filed these as AI limitations. Models hallucinate. They generate plausible code rather than correct code. You have to review everything. I wrote better prompts. I ran more corrections. The problems kept appearing, slightly differently, reliably.
The actual diagnosis, once I could see it: I hadn't made the decisions. Every gap in the output traced back to a decision I hadn't made before writing the first prompt, about what a failed fetch meant versus an incomplete one, about which fields were required versus optional, and about who owned retry behavior in the module graph. The AI wasn't hallucinating. It was filling in blanks I had left open. It was making design decisions on my behalf, using the most statistically probable answer, and handing me code that was confidently wrong in ways I had inadvertently authorized.
---
## Why prompting before deciding is the real failure mode
Software development has a documented failure mode called design-by-implementation: you write the code, then write the design doc as a description of what you built. The doc captures nothing useful because all the hard choices were already locked in. What got built is what got built, and the doc is archaeology.
The same failure mode appears in AI-assisted development, with worse consequences. When you start prompting before you've made the architectural decisions, the "spec" that emerges is shaped by whatever the agent started generating. You've lost the decision-forcing function entirely. You're not writing a spec. You're narrating the code that already exists.
The discipline is: spec before code, not spec alongside code, not spec derived from code. This sounds like documentation advice. It isn't. Writing a spec before you open an AI tool is a design act, and the spec is where the real work happens. The context file from Post 2 handles how the agent behaves across any project. The spec handles what it's building in this one. Both need to exist before the first prompt.
---
## What a spec does that prompting cannot
The spec does three things that prompting cannot replicate.
First, it forces architectural decisions before code exists. You cannot write a data schema without committing to a data model. You cannot specify exit codes without deciding which failure modes are distinct enough to distinguish. You cannot define module boundaries without making coupling decisions. Writing "the CLI will produce JSONL and JSON array output with the following fields: item_type ("issue" | "pull_request"), id (integer), node_id (string), created_at (ISO8601 string), html_url (string), api_url (string), title (string)" is not documentation. It is a design act. Every field is a choice, and every omitted field is also a choice. The spec is where those choices happen deliberately, before the pressure to ship makes them implicit.
Second, it functions as a communication protocol between your intent and the agent's execution. A vague spec leaves the agent with a large space of plausible interpretations, and the agent fills that space with probability rather than precision. Every field named and typed, every edge case annotated, every non-goal made explicit: these replace hundreds of tokens of back-and-forth clarification with a single authoritative document the agent can reference across sessions.
Third, and most underappreciated: the spec makes deviation detectable. When the implementation doesn't match the spec, one of two things is true: the implementation is wrong, or the spec needs updating. Either way, you're making an explicit decision with a clear record. Without the spec, you're looking at code and deciding whether it feels right, which is an implicit decision with no record, no accountability, and no way to reconstruct later what you intended.
---
## The misconception about who the spec is for
Here's the misconception that's common in descriptions of AI-assisted development: the spec is primarily about guiding the agent. It isn't. The agent guidance is a side effect. The spec is primarily about forcing you to make design decisions before the pressure of a running system makes them for you. By the time the code exists, all the trade-offs are already embedded, whether in the module structure, the schema, or what got elided for simplicity. Writing after the fact doesn't change those decisions. It only documents them. The spec written before the code is where the decisions actually happen, which is why it produces better code, not because the agent follows directions well, but because the directions forced the author to think.
---
## A concrete case: gh-monthly
`gh-monthly` is a Rust CLI I built to collect GitHub organization activity through the unauthenticated search API. Before writing a single line of application code, three planning documents existed.
The first was a design spec. It committed to CLI subcommands and all their flags, named exit codes for each failure category rather than arbitrary numbers guessed at by the model, a canonical record schema for JSONL and JSON array output with every field typed and annotated, a fetch strategy (sequential rather than concurrent, streaming to disk rather than buffering, and unauthenticated rate-limit budget calculated in advance), preflight behavior (probe total_count before fetching, refuse with exit code 3 if the budget is insufficient), an error handling policy distinguishing fatal from retryable failures, and explicit module boundaries.
The second was an implementation plan of 18 tasks derived directly from that spec. Each task had a file target, a failing-test-first step, a verification command, and a commit message.
The third was the implementation itself.
After the first implementation pass, I ran an alignment review: a gap analysis comparing the spec to the code. It found four significant deviations: `node_id` and `api_url` missing from the canonical schema, the full metadata contract incomplete, JSONL buffered in memory rather than streaming to disk, and the retry-aware HTTP client behavior absent. Every one of those was specified in the design doc. The gap wasn't between the spec and reality. It was between the spec and the first implementation pass.
The canonical record schema for gh-monthly: seven fields specified, five delivered in the first implementation pass, two missing.
Field
Type
First pass
item_type
"issue" | "pull_request"
delivered
id
integer
delivered
node_id1
string
missing
created_at
ISO8601 string
delivered
html_url
string
delivered
api_url2
string
missing
title
string
delivered
7 specified
5 delivered + 2 missing
Fig. 01 The canonical record schema, as specified and as delivered. Source: the design spec and the alignment review, both quoted in this post, read 2026.07.28. Callouts 1 and 2 mark the two fields the review found missing. The review found four deviations in total and this table covers only the first of them. The other three were the incomplete metadata contract, JSONL buffered in memory rather than streamed to disk, and the absent retry-aware HTTP client.
The alignment review's existence, the ability to write one at all, was only possible because the spec was precise enough to be falsifiable. The first implementation wasn't a failure. The alignment pass is the mechanism by which the spec stays authoritative. It worked because there was something authoritative to align against.
---
## How to apply this: developers and teams
For developers, the sequence is four steps. Write the design doc first, covering goals, non-goals, the external interface (CLI surface, API endpoints, or UI flows), data schemas, error states, module boundaries, and testing strategy. Derive a numbered implementation plan from the doc, where each task has a file target and a verification step so it's mechanically checkable. Work through the tasks in order. When the agent diverges from the spec, that's a signal that either the spec needs updating or the implementation needs correcting. After the first pass, run an alignment review: compare the output to the spec, write up the gaps, and close them in a second pass. The alignment pass is not a failure. It's the mechanism.
For teams evaluating whether to invest in this practice: the spec is not additional overhead on top of the AI-assisted workflow. It's what determines whether the workflow produces trustworthy output or approximately correct output that slowly diverges from intent. The investment is front-loaded. The payoff is that the last 30% of an AI-assisted build, covering integration, edge cases, and behavior at the boundaries, stops being the place where everything slows down. Post 4 shows exactly where that wall appears and why teams without specs consistently hit it.
---
## The difference that actually matters
The spec is not documentation of what you built. It is the record of what you decided before implementation pressure made the decisions for you. The difference between those two things is the difference between a codebase you can audit and one you can only approximately trust.
---
*Part 3 of 6 in the [Agentic Product Development Workflows](/articles/agentic-workflows/) series.*
*Next: [The Specification Boundary: Why AI-Assisted Builds Stall at 75%](/articles/technical-deep-dives/the-specification-boundary/) →*
---
# LLM Context Files Are Deliverables, Not Config
_Published 2026-05-04_
_1,498 words, about 8 min read._
I wrote my first CLAUDE.md in about fifteen minutes. It covered the tech stack, a few notes about TypeScript over JavaScript, a reminder about commit message format. It felt thorough. I didn't touch it for three months.
Every session, I kept adding the same clarifications. The same note about the error handling pattern we'd settled on. The same reminder that the API client was the single integration point, not a thing to be bypassed in one-off routes. The same correction about console.log statements before committing. I was rebuilding context with the agent from scratch every session, by hand, in the chat, because the file that was supposed to prevent that was frozen in week one.
The misdiagnosis I was making: I thought CLAUDE.md was configuration. Something you set up properly once, like an `.editorconfig` or a linter ruleset, and then let run. That framing was why it wasn't working. Configuration files describe how tools should behave. A CLAUDE.md describes what the project is, what decisions have been made, and what the agent needs to navigate it without guessing. Those are fundamentally different things, and the second one doesn't stay accurate on its own. That distinction is the first practical consequence of the AX shift: the file that shapes agent behavior every session is a first-class engineering artifact, not a config toggle.
---
## Global versus project context files
Before getting into structure, there's a distinction worth making explicit. A global CLAUDE.md, typically at `~/.claude/CLAUDE.md`, is the behavioral layer. It shapes how the agent works regardless of project: answer first, flag speculation, ask before diving into large changes, don't fabricate. This is personality and work style, and it persists across projects because the underlying preferences do. A project CLAUDE.md, at the repo root, is the contextual layer. It shapes what the agent knows about this codebase: stack, folder structure, conventions, and explicit boundaries. Both matter, and they do different jobs.
Most people think about only one of them. The teams who treat AI-assisted development as a serious workflow maintain both and keep both current.
## What a useful context file actually contains
Within the project layer, what separates useful context files from vague ones comes down to two categories before you get to the boundary tier. The first is invocation details. The second is process conventions.
Invocation details means commands with full flags, not tool names. "Run the tests" is a description. `npm test -- --coverage --watch=false` is an instruction. The agent needs the exact invocation, and the same applies to build, lint, type-check, and any CI-relevant commands. If the command requires environment setup, note that too. It also means explicit file paths: not "the API layer" but `src/lib/api/`, with a note that all external calls go through `src/lib/api/client.ts`. Not "the components folder" but `src/components/` with whatever structural rules apply. Agents pattern-match from concrete anchors. One canonical code snippet beats three paragraphs describing a pattern: if your error handling follows a specific shape, paste a real example. If your components use a particular structure, paste the shell. Description and demonstration both work at first, but demonstration is what the agent will actually match against.
Process conventions covers testing and git. For testing: not "we use Jest" but where tests live (`__tests__/`, or `*.test.ts` colocated, or `tests/integration/` for integration tests), what coverage threshold is enforced, and whether there's a separate command for watch mode versus CI. "We use Jest" leaves the agent guessing at all of this. For git, note the branch naming prefixes, commit message format, PR requirements, and any review expectations. Without this, the agent will invent a convention. It will be internally consistent and wrong — and that wrongness compounds, because each session starts from the invented baseline rather than the actual one.
The area most consistently absent from real-world context files is the third category.
## The boundary tier most teams skip
The boundary tier is where most context files fall short. Telling the agent what the project does isn't enough. It needs to know what it can do autonomously, what requires a check-in, and what's off the table regardless of context.
The always tier is the baseline: run tests before committing, include a link to the related issue in PR descriptions.
Ask-first is what requires a pause before proceeding: adding new dependencies, deleting files (even empty ones), modifying database migrations.
Never is the hard stop: no secrets or credentials even as placeholders, no direct pushes to main, no modifications to the generated files in `src/generated/`.
That third tier is the one most teams forget. "Never commit secrets" is the most consistently repeated constraint across every context file I've reviewed in production use, which means it's also the constraint most teams don't write until they've had a close call. The fact that it needs to be explicit is itself telling about the failure mode.
## Decisions versus state descriptions
Here's the misconception that makes most context files decay: teams write them as a reflection of what the project is right now, rather than as a record of decisions that have been made. The distinction sounds subtle, but it determines whether the file compounds or decays.
A project description captures state. A decision record captures commitments. "We use React 18" is a state description. It'll stay true until you upgrade. "We use function components only; no class components" is a commitment. It was a decision made at a specific point for a specific reason, and it tells the agent something that reading the code might not. State can be inferred. Decisions need to be explicit.
Every vague line in a context file is a decision you haven't recorded, and the agent will make that decision for you. It will do so predictably, defaulting toward whatever pattern appears most often in its training, not necessarily the one your codebase has established. Those defaults are where drift originates.
## The self-improving pattern
The self-improving pattern is what makes the difference between a context file that compounds and one that decays. When the agent uses a stale API path or wrong pattern, the correction should be: "No, use `v2/users`. Update CLAUDE.md to reflect this, then continue." The agent updates the file in-session. Future sessions inherit the correction. The file grows more accurate through use rather than despite it.
This requires treating the context file as something you actively maintain rather than something you set and revisit when things break. The practical cadence: when you catch yourself re-explaining the same thing to the agent a second time, stop and add it to the file. That correction is information about what's missing. Don't let it live only in the session history where no future session can reach it.
The opposite failure mode is context bloat. A 400-line CLAUDE.md with outdated migration notes, deprecated API paths, and decisions from a tech stack you replaced six months ago is worse than a 100-line one that's current. The agent reads the whole file on every session. Every stale line is a small injection of noise into every interaction. Prune regularly. Archive rather than accumulate.
## The diagnostic the file actually runs on you
The genuinely non-obvious part, the part that isn't visible until you've maintained one of these files through a real project, is that writing a useful context file requires having made the decisions it's supposed to record. You can't write "all external calls go through `src/lib/api/client.ts`" unless you've established and enforced that as a real convention. CLAUDE.md surfaces where project conventions are implicit rather than explicit. Every vague line is a decision still open. The file is, in a strange way, a diagnostic: if it's hard to write precisely, the project architecture may be harder to describe precisely than you thought.
For teams evaluating AI-assisted development at scale, this is the discipline worth investing in ahead of model selection or prompt engineering. The context file shapes every session, and its quality determines the quality of the work that comes out of it. The model you're running, the prompt you've written, the workflow you've built around it — all secondary. Treat the context file with the same rigor as a spec. Review it the way you'd review a significant PR. It sets behavioral constraints: how the agent works. The spec that Post 3 introduces sets scope and decisions: what it builds. Both need to exist before the first prompt.
A CLAUDE.md that hasn't been updated since you wrote it isn't config that's working quietly. It's a gap between how you think the project is and how the agent is navigating it. That gap compounds. The sessions that feel inexplicably off, where the agent makes choices that don't quite fit and where you correct the same thing again, are often the file speaking.
---
*Part 2 of 6 in the [Agentic Product Development Workflows](/articles/agentic-workflows/) series.*
*Next: [The Spec Is the Work: PRD-First AI Development](/articles/technical-deep-dives/the-spec-is-the-work/) →*
---
# The AX Shift: You're Still Designing for Yourself
_Published 2026-04-27_
_1,254 words, about 7 min read._
I kept improving my prompts.
In every session that drifted, the agent generated code in a style we'd already moved away from, missed a pattern we'd established in the previous session, or made the same class of mistake I'd corrected twice before. I diagnosed this as a prompt quality problem. The fix was to write better prompts, include more specifics, add more examples, and bundle more context into each message. Even after I got quite good at prompting, the drift kept happening.
After six months, I grew not only a library of refined prompts but also a post-code-generation corrections list that looked nearly identical to the one from month two. The thing I kept adjusting wasn't the problem. The problem was somewhere I hadn't looked yet.
I was still designing everything for myself. The human in the loop. This is what I kept missing. The prompts, the context files, the project notes, the spec: all of it was written for a human reading it later, not for an agent trying to infer intent from it right now. Inference at the edges of the AI session is exactly what was causing the drift.
---
## From UX to DX to AX
Software interfaces have followed a consistent pattern. UX redesigned those interfaces around their users rather than the people who built them. DX redesigned APIs around their developer-consumers rather than their designers. AX (Agent Experience) is the next iteration. The primary consumer of your documentation, API responses, project context files, and specifications is increasingly an AI agent operating autonomously between sessions. Almost everything we've built so far was designed for humans who can read between the lines, ask colleagues, and remember what we talked about last week.
Agents cannot reliably read between the lines, nor do they phone a friend. They fill ambiguity with the most statistically probable answer. They're never 100% correct, and those near misses compound.
---
## The gh-monthly case study
The clearest moment I have of seeing this is a Rust CLI I built called [`gh-monthly`](https://github.com/globalpayments-samples/gh-monthly). It's a tool that collects activity across one or more GitHub organizations via the unauthenticated search API, normalizes it to JSON, JSONL, and CSV, and produces local terminal reports. It's not a complicated project, but the implementation session started before the real problem decisions were made.
The exit codes came out arbitrarily, whatever the model thought was reasonable. The JSON and JSONL schemas had fields that made sense in isolation but weren't composable across the four scripts consuming the output. The retry behavior was guessed. The metadata contract was incomplete. None of these were the agent's failures. The agent was doing exactly what you'd expect a capable text generator to do with vague requirements. It filled in the gaps as best as it could.
Afterward, I restarted, writing a design spec first, and I realized halfway through that the spec process was forcing me to make decisions I'd been deferring. The exit codes needed to be named and scoped for shell automation rather than numbered and intuited. The metadata needed `run_started_at`, `run_completed_at`, `completed`, and `request_count` included to evaluate the results later rather than whatever felt sufficient in the moment. The output had to be streamed immediately to disk during fetch rather than live in memory until completion, because the whole point was to handle potentially large result sets.
Every one was a design decision I'd delegated to the model. It had made choices, but they were wrong.
After writing the spec, I ran an alignment pass against the first implementation, generating a document listing the gaps between what the spec said and what the code did. Four significant deviations came up. All of them traceable to places where the spec hadn't been written yet when the code was generated.
The alignment pass is the honest part of the story. Finding gaps here isn't a failure. That's the mechanism you need, but the mechanism only works if you wrote the spec first, before implementation pressure fills in the blanks.
---
## Why prompts don't fix structural drift
Here's the misconception about AX that most descriptions get backward. It's usually framed as "make your system work well for agents," and that's true in practice. This names the effect rather than the discipline, though. The more productive shift is making the important planning decisions before you build, rather than during or after.
When you design for a human reader, ambiguity is recoverable. A developer reading unclear documentation can infer intent, check the GitHub issues, or ask a colleague. An agent operating across sessions cannot do any of those things with any certainty. Every ambiguous line in your spec or context file is a decision you haven't made. The agent will make it for you, in the direction of statistical probability.
The spec is not primarily about guiding the agent. It's about forcing you to make design decisions before the pressure of a running system makes them implicit. The agent's reliable behavior is a byproduct. What you actually get first is clarity about what you're building, which turns out to be the same thing that makes the agent reliable.
This is why improving prompts doesn't fix structural drift. Prompts are interaction-level adjustments. Drift is artifact-level. You can have excellent prompts running against an underdeveloped CLAUDE.md and a vague spec, and the agent will still be making design decisions on your behalf, albeit with slightly better manners.
---
In practice, this is harder to shift than it sounds. The instinct, especially with a capable AI tool available, is to start building and let the shape emerge. Scaffolding is fast. It's tempting to interpret that speed as permission to defer thinking, but there's one thing you lose when you defer: your ability to audit the work. Without a spec, there's no standard to close a task against. You review the output, decide it feels approximately right, and move on. When you hit a wall three sessions later, you can't tell whether the spec changed, the implementation drifted, or you never decided what needed to happen. Developers working this way inevitably realize the codebase is approximately correct, but not definitively right.
For teams evaluating whether to focus and invest more in AI-assisted development, this is where they should put their effort. Not into model selection or prompt engineering, but the artifacts that give their agents reliable context across sessions. A CLAUDE.md that's updated after every session. Design specs that are authoritative enough that gap analyses can be written against them. Implementation tasks derived from the spec rather than improvised from the conversation. The next two posts dig into both: what makes a context file actually work, and what a spec needs to do that prompting cannot.
The same artifact mindset applies to API documentation. An older example is [Writing API Wrappers with Elixir](/archive/2013/writing-api-wrappers-with-elixir/), where the interface contract has to be made explicit before the code can be useful.
The shift from DX to AX isn't a new tool or a new model. It's a new relationship to the artifacts you already produce, treating them as the primary communication layer with the agent rather than the documentation you write afterward. The answer to "why does the agent keep getting this wrong?" has rarely been "better prompts." It's been "better decisions, made earlier, written down somewhere the agent can find them."
---
*Part 1 of 6 in the [Agentic Product Development Workflows](/articles/agentic-workflows/) series.*
*Next: [LLM Context Files Are Deliverables, Not Config](/articles/technical-deep-dives/llm-context-files-are-deliverables-not-config/) →*