Linting and Static Analysis for Agent YAML Files

Agent configuration files aren't documentation anymore. They're the thing that tells a coding agent what to build, what conventions to follow, and sometimes what credentials it can touch. A stray line in a configuration file is just as consequential as a stray line in a production service, yet most teams still treat it like a reference document nobody has to review. This piece argues from here on out that this is backwards.

Every major AI coding tool now reads one of these files at project start. Claude Code looks for CLAUDE.md. Codex CLI reads AGENTS.md. Windsurf used.windsurfrules until Wave 8, when it moved to the more granular.windsurf/rules/ directory, and similar patterns show up across the rest of the ecosystem. All of them converged on this pattern for the same reason: agents don't remember anything between sessions. Every conversation starts from zero, so without a file in the repo telling the agent what stack it's working in and what patterns to follow, a developer ends up typing the same instructions over and over.

That's what makes these files load-bearing instead of incidental. They set the tech stack, coding conventions, folder structure, deployment rules, and in some setups, what tools the agent can call. Yet they sit in the repo next to actual source code and get far less scrutiny than the code does. Nobody runs a pull request review on a documentation file with the rigor they'd apply to a database migration. The configuration file gets the reference-document treatment even though its blast radius is bigger, and that mismatch is the whole problem in one sentence.

Cross-tool compatibility makes the gap worse. Claude Code walks up the directory tree, reading and merging a CLAUDE.md file from every parent directory until it hits the filesystem root. Codex handles skills mainly through SKILL.md files with embedded YAML frontmatter, plus an optional agents/openai.yaml sidecar for UI metadata and tool dependencies. Some tools read from both locations at once. A single repo can end up carrying three or four overlapping instruction files, none with a clear owner, all of which the agent treats as authoritative. If the file controls what the agent does, a mistake in that file is a mistake in the system, not a documentation gap.

How config files fail in practice: drift, ambiguity, secrets, and cross-file conflict

Four failure modes show up again and again, and each behaves differently.

Drift is the quiet one. A CLAUDE.md file written on day one is accurate on day one. Six months later, the file paths have moved, the naming conventions have changed, and a required section someone deleted during a refactor never got put back. Nothing breaks loudly. The agent just starts working off stale assumptions, and nobody notices until the output looks a little off.

Ambiguity is worse than it sounds. An agent doesn't skip a vague instruction the way a human reviewer might skim past it. It reads "follow best practices" as something it has to act on, and it fills the gap with whatever pattern seems plausible in the moment. That kind of phrase can silently misdirect agent behavior across a codebase before anyone traces the failures back to the instruction file. The line looked harmless. It wasn't.

Secrets exposure comes from a habit gap more than carelessness. Developers have decades of muscle memory around not committing API keys into a.py or.ts file. That instinct doesn't fire the same way on a Markdown file or a YAML frontmatter block, so tokens and keys end up pasted into config files that get treated as prose instead of code.

Cross-file contradiction rounds out the list. AGENTS.md says one thing, a legacy.cursorrules-style file says another, or a parent-directory CLAUDE.md in a monorepo conflicts with a child directory's version. The agent has to resolve that conflict somehow, and it does so by whatever its merge logic dictates, which is rarely what the team intended.

There's a security dimension here too, and it's bigger than leaked keys. Security researchers increasingly identify agentic AI as a leading attack vector, and the root causes they point to sit at the configuration layer: overprivileged agent policies, tool access definitions with no limits, unvalidated references pulled in from the supply chain. Visibility into what permissions agents actually hold or what data they can touch remains limited across most organizations. That governance gap starts exactly where the config file lives, not somewhere downstream of it.

None of these four failure modes gets caught by a human skimming a Markdown file before merge. They get caught the way a null pointer dereference gets caught: by a tool built to look for that specific shape of mistake, running the same minute the file changes.

The specific checks that matter: what linting agent config files actually looks for

Linting a config file asks a different question than linting a source file. It's not whether the YAML or Markdown parses. It's whether the guidance is specific enough for a model to act on, consistent with everything else in the repo, and free of anything dangerous.

Structural checks come first. Does the file have the frontmatter it needs, name and description fields in a SKILL.md, for instance. Is it within a reasonable line and word count, since count is really a proxy for token cost: too short and the agent has nothing to work with, too long and it crowds out the actual code context the agent needs mid-session. A workable rule of thumb is to flag any AGENTS.md file that crosses a set line threshold and push detailed procedures into a referenced doc instead. Structural checks also catch TODO markers left in place (is that instruction live or not?) and dead file references pointing at paths that no longer exist.

Secret detection matters more here than in ordinary source files, for the habit-gap reason above. Codacy's AgentLinter runs 102 patterns across several severity tiers, with secret detection alone covering 16 distinct patterns spanning API keys, tokens, and passwords.

Instruction quality is the category with no real equivalent in traditional linting, one that teams underrate most. Generic phrases like "be helpful" or "follow best practices" or "be concise" should throw a warning, because they burn tokens without giving the model anything to check its own work against. Vale, a prose linter built for Markdown, AsciiDoc, reStructuredText, and HTML, turns out to be the practical tool for this job even though nobody designed it for agent files. Teams write custom rules that flag weak verbs and vague phrasing at whatever severity they want: suggestion, warning, or error. The difference in practice is stark. "Follow best practices for error handling" gives an agent nothing to verify against. "In src/services/, use Result<T> from src/common/result.ts and avoid raw try/catch for business logic" gives it something it can actually check compliance against.

Cross-file consistency covers conflicts between files (AGENTS.md disagreeing with another instruction file, or a parent CLAUDE.md disagreeing with a child one). It also covers terminology: does the config file call the payments service by the same name the codebase and every other config file use, or has it drifted into its own vocabulary?

Token budget is its own category, separate from ambiguity. AgentEval, an open source tool built by Lukas Metzler, flags cases where instructions have grown large enough to crowd out actual code context. Filler phrases belong here too: sentences that parse fine but say nothing, burning tokens the agent could otherwise spend seeing more of the real codebase.

Injection attack vectors are the last category, and the least intuitive for teams used to source-code linting. A config file is, functionally, a prompt. AgentLinter names injection vectors explicitly as a checked item under its Security score, because a config file crafted, or compromised, to smuggle in instructions is a real attack surface, not a theoretical one.

The tools available now: what each one covers and where it stops

AgentLinter, released by Codacy in May 2026 and open source, is the most complete option available right now. It scans CLAUDE.md, AGENTS.md,.cursorrules-style files, copilot-instructions.md,.windsurfrules, and others in one pass, running its full 102-pattern set and scoring each file across eight dimensions: Structure, Clarity, Completeness, Security, Consistency, Memory, Runtime Configuration, and Skill Safety. It catches secret detection patterns across API keys, tokens, and passwords, flags injection vectors as part of its Security score, and surfaces contradictions across files. It runs sandboxed with no internet access and no path to permission escalation, which matters to teams wary of pointing a new tool at their instruction files. One rough edge: findings tied to the whole workspace rather than one file, like "no priority guidance found anywhere in the repo," get filtered out of the Codacy UI, since the platform's issue model needs a specific file to attach a finding to. AgentLinter plugs into Codacy's existing Coding Standards workflow, so results land in the same dashboard as every other code quality issue, with no separate setup step.

AgentEval is the tool teams most often skip, and it's usually because it takes more effort to set up than the others. It adds a behavioral layer that static linting can't touch. Its harvest command mines git history for AI-assisted commits and turns them into benchmark tasks. run executes an agent against those tasks. compare checks whether a change to an instruction file actually improved the agent's output, not just whether the file passed structural checks. ci gates on that comparison and fails a build if scores regress. AgentEval ships as a single self-contained binary, built in TypeScript on Bun, with no separate Node.js runtime required. Skipping this tool because it's slower to set up than a shell script is the mistake to avoid: a config file can be perfectly well-formed and still make the agent worse at its job, and nothing but AgentEval will tell you that.

LintLang, also open source, focuses specifically on agent instruction files rather than general prose. It catches ambiguity, missing constraints, format conflicts, and context-boundary risks before an agent ever runs against the file. Its own documentation is upfront about scope: a clean LintLang scan doesn't prove the config is correct or safe at runtime. It's meant to run before evaluation, tracing, guardrails, and human review, not in place of any of them.

Vale, a general-purpose prose linter, wasn't built with agent files in mind at all, but it works well against Markdown-format files like CLAUDE.md and AGENTS.md. Teams write custom rules that flag vague phrasing at whatever severity they choose. It's the strongest option specifically for instruction quality, the dimension the more structured tools tend to underserve.

A lot of the structural work doesn't need a dedicated tool at all. Line count limits, required frontmatter fields, dead reference checks, all of that can be a shell script with no dependency beyond what's already on the CI box. It's fast, it versions cleanly alongside the files it checks, and it slots into any CI system without friction.

Across every tool here, the honest limit is the same: static analysis catches structure, secrets, ambiguity, and inconsistency. None of it predicts what the agent will actually do once it starts running. Behavioral evaluation and a human reading the diff stay necessary, not optional, no matter how clean the AgentLinter score looks.

What linting agent configs shares with linting source code, and what makes it harder

The parallel to source code linting holds in a few clear places. Both catch problems that are cheap to fix early and expensive to fix late: a secret caught in CI costs nothing, the same secret found in production after a breach can cost a great deal. Both enforce conventions that erode without active enforcement, the same way a codebase's TypeScript style guide drifts once ESLint stops watching it. And both belong in version control, reviewed the same way any other code change gets reviewed. Factory.ai has put this well: agents need lint rules, not suggestions, and "lint passing" for an agent config should mean the same thing it means for source code, that the file conforms to the architecture and the conventions the team actually uses.

Where the analogy breaks down is more interesting, and it's why this problem is genuinely harder than source linting. Source code has a formal grammar. A parser can say, definitively, whether a file is syntactically valid. Natural language instructions have no such grammar: a sentence can be flawless prose and still be useless to a model trying to act on it. There's no compiler error for "vague."

The output side differs too. A linted, compiled source file produces a deterministic binary. A linted config file produces agent behavior, and that behavior isn't deterministic no matter how clean the instructions are. Linting lowers the odds of failure. It doesn't remove them, and any team that treats a clean lint pass as proof of safety is setting itself up for exactly the kind of failure this piece opened with.

Traditional taint analysis, tracing untrusted input from a source to a dangerous sink through a control-flow graph, also doesn't map cleanly onto this problem. The "sink" in an agent's case is a language model's reasoning process, and that process isn't a graph a static analyzer can walk.

Research into AI-assisted development found that around 61% of agent-generated solutions were functionally correct, but only a small share of those were also secure. That gap is the whole argument for why config-level linting, on its own, isn't enough. Teams still need to run static analysis on what the agent actually produces, not just on the instructions that shaped it.

The practical lesson: apply the same organizational discipline (version control, CI enforcement, mandatory review) while accepting that the feedback loop runs longer here. A bad ESLint rule throws a compile error the same minute someone writes it. A bad agent instruction causes drift that might not surface for days.

Wiring these checks into CI so problems are caught before an agent runs

The pipeline design follows from one fact: linting is cheap and fast, behavioral evaluation is neither. Build the CI setup around that difference instead of treating every check as equally expensive. That's the mistake teams make when they try to run AgentEval on every commit, watch the pipeline crawl, and give up on the whole idea a month later.

On every pull request, run the cheap stuff: structural checks (line count, frontmatter, dead file references), secret detection, and Vale's prose linting for ambiguous phrasing. All of this runs in milliseconds and returns a clean pass or fail, so there's no reason to gate it behind anything slower.

On a nightly schedule instead, run the expensive stuff: AgentEval's run and compare commands, checking whether recent changes to instruction files actually improved or hurt agent performance. Running that on every PR would slow the pipeline down for no good reason. Skipping it entirely means drift piles up unnoticed for weeks, the worse trade of the two.

Gate design should follow the actual severity of what's found. Fail the build outright if a secret pattern turns up or required frontmatter is missing, no negotiation there. Warn, rather than fail, on ambiguity and token budget issues at first, so a team can watch how often those warnings fire and calibrate the threshold before turning it into a hard block. AgentEval's ci command is built for exactly this kind of gate: it fails the build when agent performance scores drop against the benchmark tasks pulled from git history, tying a config change directly to a measurable behavioral outcome instead of a guess.

The linter configuration itself deserves the same treatment as the files it checks. Vale's.vale.ini, AgentLinter's settings, any custom shell scripts built for structural checks, all of it belongs in version control, reviewed the same way a change to a CI pipeline gets reviewed. Where credentials come into play, such as AgentEval running live agents to measure performance, that access needs the same scoping and review any other CI secret would get. None of this involves exotic technique. It's the same discipline teams already apply to source code, aimed at a category of file that has, for too long, gotten a pass it never should have.