Output Quality Audit monitors every response your automated systems produce. Detect mistakes, silent rewrites, factual errors, and compliance violations — automatically.
Get Agent Audit — £39⚡ 1,649-line Python tool • Works with any system • 5-minute setup
Agents silently edit figures in summaries every day. That invoice said £4,500 — your agent told the client £4,000. You won't catch it manually.
97% of AI agents hallucinate facts at least once per 100 responses. Your customers trust them. Your legal team won't when they find out.
Compliance violations happen silently. One agent response promising something your business can't deliver — and you're liable.
Over weeks, responses get shorter, snarkier, or stray from your brand voice. You don't catch it until churn spikes.
Agents get stuck repeating the same phrases or questions, frustrating users. Manual review is too slow.
When something goes wrong, you have no record of what the agent said, when, or why. Compliance teams panic.
Six audit checks that run against every agent response
Cross-references claims against source material. Flags unverifiable facts, invented statistics, and fabricated citations.
Compares agent output to raw LLM response. Catches when middleware or post-processing changes content without logging it.
Define forbidden phrases, required disclosures, and regulatory patterns. Violations trigger immediate alerts.
Tracks sentiment, reading level, and response length over time. Alerts when quality degrades beyond your thresholds.
Generates structured JSON audit reports — pass/fail per check, severity scores, and actionable fix suggestions.
OpenAI API, Anthropic API, or custom JSON logs. One Python script, no dependencies beyond requests.
Single Python file. Runs anywhere — your server, CI pipeline, or cron job.
OpenAI logs, Anthropic logs, or any JSON file with agent responses. One config line.
Set forbidden phrases, compliance requirements, and quality thresholds. Or use defaults.
Run on-demand or schedule via cron. Every response scored. Every violation flagged.
🔒 Secure payment • Instant download
You can't manually review every response. Let Output Audit do it — automatically, every time.
Get Agent Audit — £39Only if you want to audit those providers' outputs. The tool itself runs locally — it reads your existing agent logs. No additional API costs to run the audit.
Yes. The custom JSON log input accepts any structured agent output — Claude via AWS Bedrock, open-source models, even non-LLM chatbots. Just format your logs as JSON.
Daily is recommended for production agents. The tool is cron-friendly — schedule it alongside your other Hermes Agent cron jobs. Each run takes seconds for typical log volumes.
It's a self-hosted Python script. You own it, you run it, your data never leaves your machine. No monthly fees, no vendor lock-in.
14-day money-back guarantee. If it doesn't catch issues in your agent outputs, email for a full refund.
Setup time: 5 minutes. Point it at your agent's output and run.
# Point the audit at your agent logs directory ls -la ~/agent-logs/ # Each log file should contain the full agent response
hermes run --prompt "Audit all agent outputs in ~/agent-logs/ for the last 24 hours. Check for: factual errors, hallucinations, compliance violations, and tone drift. Score each output 0-100 and flag anything below 70. Deliver a summary report to ~/audit-report.md"
hermes cron create \ --name "daily-agent-audit" \ --schedule "0 8 * * 1-5" \ --prompt "Run agent output audit on ~/agent-logs/. Check for errors, hallucinations, and compliance. If any output scores below 70, alert me on Slack with the details."
No fluff. No demo request form. Just a quick look at what this does.
⚠️ This is NOT for you if:
you want get-rich-quick schemes, expect overnight results, are looking for "hype" AI tools, or can't be bothered to follow a 5-minute setup.
✅ This IS for you if:
you want boring automation that quietly makes money, you value reliability over flash, and you believe one person with the right systems can build a real business.
Real products, real revenue, real problems solved. No hype — just boring reliability.