Ask HN: How do you measure whether your coding agent follows its rules?

You’ve given your coding agent a set of rules—maybe a style guide, architectural constraints, or security requirements. The agent writes code that looks plausible. But how do you actually verify it followed the rules you set?

This is the compliance measurement problem, and it’s harder than it sounds. Unlike traditional linters that check syntax, you need to verify semantic adherence to natural language instructions. Here’s how to do it today with existing tools.

The Manual Spot-Check Method

start simple. Pick 10-20 representative outputs from your agent and manually review them against your ruleset. Create a spreadsheet with columns for each rule and rows for each output.

For each rule violation, note:
– Which rule was broken
– The severity (blocker, major, minor)
– Whether the violation was partial or complete
– The line number or code section

This gives you a baseline violation rate. If you’re seeing 30% of outputs violate at least one rule, you have a measurement: your agent has 70% compliance.

The advantage? You understand the types of failures. Maybe your agent ignores naming conventions but never violates security rules. That’s actionable data.

Do this weekly. track the compliance percentage over time as you refine your prompts.

Static analysis tools as Rule Enforcers

Many rules can be converted into machine-checkable constraints using existing static analysis tools.

For Python, use pylint with a custom configuration file. If your rule is “never use global variables,” add this to your .pylintrc:

[MESSAGES CONTROL]
disable=
enable=global-statement,global-variable-not-assigned

Run pylint on every agent output and count violations. Your compliance score is the percentage of files with zero violations.

For JavaScript/TypeScript, ESLint works similarly. If your rule is “always use const for variables that don’t change,” enable the prefer-const rule and set it to error level.

Semgrep is particularly powerful for custom rules. write patterns in YAML that match code structures you want to forbid. If your rule is “never concatenate strings to build SQL queries,” create a Semgrep rule:

rules:
  - id: no-sql-concat
    pattern: |
      $QUERY = $STR1 + $STR2
    message: SQL queries must use parameterized statements
    languages: [python]
    severity: ERROR

Run Semgrep in CI and track the violation count per 1000 lines of generated code.

Embedding-Based Similarity Scoring

For rules that are harder to formalize, use semantic similarity. This works for rules like “code should follow the same architectural patterns as the existing codebase.”

Take 20 examples of compliant code from your repository. generate embeddings using OpenAI’s text-embedding-3-small API or the open-source sentence-transformers library.

from sentence_transformers import SentenceTransformer
import numpy as np

model = SentenceTransformer('all-MiniLM-L6-v2')

# Your compliant examples
compliant_code = [
    "def calculate_total(items):n    return sum(item.price for item in items)",
    # ... more examples
]

# Generate reference embeddings
reference_embeddings = model.encode(compliant_code)
reference_mean = np.mean(reference_embeddings, axis=0)

# Score new agent output
new_code = agent.generate_code(prompt)
new_embedding = model.encode([new_code])[0]

similarity = np.dot(new_embedding, reference_mean) / (
    np.linalg.norm(new_embedding) * np.linalg.norm(reference_mean)
)

print(f"Compliance similarity: {similarity:.2f}")

Set a threshold (say, 0.7) below which you consider the code non-compliant. track what percentage of outputs exceed this threshold.

This won’t tell you which rule was violated, but it flags outputs that feel architecturally different from your baseline.

LLM-as-Judge Evaluation

Use a second LLM to grade the first one’s adherence to rules. This is expensive but handles complex, nuanced rules.

Create a grading prompt:

You are evaluating code against a set of rules.

RULES:
1. All functions must have docstrings
2. Variable names must be descriptive (>3 characters)
3. No hardcoded credentials

CODE:
{generated_code}

For each rule, respond with:
- PASS or FAIL
- Brief explanation

Format as JSON.

Send this to GPT-4 or Claude, parse the JSON response, and calculate the percentage of rules passed.

Run this on a sample of 50 outputs per week. Track the average pass rate over time.

To reduce costs, only evaluate outputs that passed your static analysis checks. The LLM handles the subjective rules that tools can’t catch.

Test-Driven Rule Verification

write actual unit tests that verify rule compliance. If your rule is “all API endpoints must have rate limiting,” write:

def test_all_endpoints_have_rate_limiting():
    endpoints = extract_endpoints_from_code(agent_output)
    for endpoint in endpoints:
        assert has_rate_limiting_decorator(endpoint), 
            f"Endpoint {endpoint} missing rate limiting"

Run these tests in CI. Your compliance metric is test pass rate.

This requires upfront work to write the tests, but once written, they run automatically on every agent output. You get binary pass/fail data that’s easy to track.

Diff-Based Compliance Tracking

If your agent modifies existing code, compare before and after states.

Run your static analysis tools on both versions. If the original code had zero violations and the modified code has five, the agent introduced five violations.

Track “violations introduced per change” as your metric. A well-behaved agent should have a rate near zero.

Use git diff with custom scripts to identify which specific changes violated which rules. This pinpoints whether the agent is consistently breaking certain rules during certain types of edits.

Building a Compliance Dashboard

Combine these approaches into a single tracking system. Use a simple SQLite database with a table structure:

CREATE TABLE compliance_checks (
    id INTEGER PRIMARY KEY,
    timestamp DATETIME,
    output_id TEXT,
    check_type TEXT,  -- 'static', 'llm_judge', 'similarity'
    rule_name TEXT,
    passed BOOLEAN,
    details TEXT
);

after each agent run, insert rows for every check you performed. Query for aggregate metrics:

SELECT 
    rule_name,
    AVG(CASE WHEN passed THEN 1.0 ELSE 0.0 END) as compliance_rate
FROM compliance_checks
WHERE timestamp > datetime('now', '-7 days')
GROUP BY rule_name;

Visualize this in Grafana, Metabase, or even a Jupyter notebook. You now have a time-series view of which rules your agent follows and which it ignores.

---

We’re building something that does this automatically—join the notify list and we’ll tell you the day it’s ready.

.mai-waitlist{max-width:560px;margin:28px auto;padding:24px 24px 22px;border:1.5px solid #2f6bd8 !important;border-radius:14px;background:#0f2238 !important;box-shadow:0 10px 34px rgba(0,0,0,.40);box-sizing:border-box} .mai-waitlist *{box-sizing:border-box;font-family:inherit} .mai-waitlist .maiwl-h{margin:0 0 6px !important;padding:0 !important;font-weight:700 !important;font-size:1.18rem !important;color:#ffffff !important;line-height:1.35 !important} .mai-waitlist .maiwl-sub{margin:0 0 16px !important;padding:0 !important;font-size:.92rem !important;color:#a9bdd6 !important;line-height:1.5 !important} .mai-waitlist input,.mai-waitlist textarea{width:100% !important;padding:13px 14px !important;margin:0 0 10px !important;border:1px solid #345782 !important;border-radius:9px !important;font-size:16px !important;min-height:46px !important;background:#091a2c !important;color:#eef5fc !important;display:block !important} .mai-waitlist textarea{min-height:66px !important;resize:vertical !important} .mai-waitlist input:focus,.mai-waitlist textarea:focus{outline:none !important;border-color:#5b9dff !important;box-shadow:0 0 0 3px rgba(47,107,216,.25) !important} .mai-waitlist input::placeholder,.mai-waitlist textarea::placeholder{color:#7e96b3 !important;opacity:1 !important} .mai-waitlist .maiwl-hp{position:absolute !important;left:-9999px !important;width:1px !important;height:1px !important;min-height:0 !important;opacity:0 !important;margin:0 !important;padding:0 !important} .mai-waitlist .maiwl-btn{width:100% !important;padding:14px 18px !important;margin:2px 0 0 !important;border:0 !important;border-radius:9px !important;background:#2f6bd8 !important;color:#ffffff !important;font-size:16px !important;font-weight:700 !important;cursor:pointer !important;min-height:48px !important;box-shadow:0 5px 16px rgba(47,107,216,.45) !important;transition:background .15s} .mai-waitlist .maiwl-btn:hover{background:#3f7bf0 !important} .mai-waitlist .maiwl-msg{margin:12px 0 0 !important;padding:0 !important;font-size:.92rem !important;color:#a9bdd6 !important;min-height:1em !important}

function maiWaitlist(btn){ var box = btn.closest('.mai-waitlist'); var name = box.querySelector('.maiwl-name').value.trim(); var email = box.querySelector('.maiwl-email').value.trim(); var wants = box.querySelector('.maiwl-wants').value.trim(); var hp = box.querySelector('.maiwl-hp').value; var app = box.querySelector('.maiwl-app').value; var nonce = box.querySelector('.maiwl-nonce').value; var post = box.querySelector('.maiwl-post').value; var msg = box.querySelector('.maiwl-msg'); function setMsg(c,t){ msg.style.setProperty('color',c,'important'); msg.textContent=t; } if(!name){ setMsg('#ff8a8a','Please enter your name.'); return; } if(!email || email.indexOf('@')<1){ setMsg('#ff8a8a','Please enter a valid email.'); return; } btn.disabled = true; var label = btn.textContent; btn.textContent = 'Sending…'; var params = new URLSearchParams(); params.append('name', name); params.append('email', email); params.append('wants', wants); params.append('hp', hp); params.append('app_slug', app); params.append('nonce', nonce); params.append('source_post_id', post); params.append('utm', (window.location.search||'').replace(/^?/,'')); fetch('https://blog.masterailabs.com/wp-json/mai/v1/waitlist', { method:'POST', headers:{'Content-Type':'application/x-www-form-urlencoded'}, body: params.toString() }).then(function(r){ return r.json(); }).then(function(d){ btn.textContent = label; btn.disabled = false; if(d && d.ok){ setMsg('#5fe39b', d.message || 'You're in — check your email to confirm.'); box.querySelector('.maiwl-name').value=''; box.querySelector('.maiwl-email').value=''; box.querySelector('.maiwl-wants').value=''; } else { setMsg('#ff8a8a', (d && d.message) || 'Something went wrong. Try again.'); } }).catch(function(){ btn.textContent = label; btn.disabled = false; setMsg('#ff8a8a','Network error. Try again.'); }); }

---

FAQ

How many outputs should I check to get statistically significant results?

For a confidence level of 95% with a 10% margin of error, you need about 96 samples. If you can only check 30 outputs, your margin of error increases to 18%. start with what’s feasible and increase sample size as you automate more checks.

Should I weight certain rules more heavily than others?

Yes. Security rules should probably count more than style preferences. Create a weighted compliance score where each rule has a multiplier. A security violation might count as -10 points while a naming violation counts as -1.

What compliance rate should I target?

It depends on your use case. For production code, aim for 95%+ on critical rules. For prototyping or internal tools, 80% might be acceptable. The key is tracking the trend—improving from 60% to 80% matters more than the absolute number.

Can I use these techniques for non-coding agents?

Absolutely. The LLM-as-judge and embedding similarity approaches work for any text output. If your agent writes customer emails, you can check adherence to tone guidelines the same way.

how do i handle rules that conflict with each other?

Document the priority order and measure compliance against the higher-priority rule when conflicts occur. Alternatively, track “conflict rate” as a separate metric—if 15% of outputs create rule conflicts, that’s signal that your ruleset needs refinement.