Autogeny

Independent audits of AI in production

Your AI passed the demo. Now it has to pass their security review.

Autogeny independently audits the AI inside your product and hands you a report your buyer’s reviewer can use. First, a targeted red team and a pass or fail on the failures you can’t afford. Then a score for how well your AI does its job, and whether a cheaper model would do as well. Plus an AIUC-1 readiness check, delivered in three to four weeks.

Built for AI-native companies selling to enterprise and regulated buyers. Three design-partner spots open.

The problem

The deal is won. Then security review starts.

Your champion said yes. Now their security, risk and procurement teams want to know how you know your AI is accurate, what happens when it is wrong, and who checked. A demo and a benchmark score don’t answer that.

Deal cycle

Reviews are where AI deals stall

Half of enterprise buyers name IT security review as the biggest delay after choosing a vendor. AI purchases take 16 to 20 weeks, against 7 to 10 for standard software.

Questionnaires

The questionnaires now ask about AI

The CSA AI Controls Matrix has 247 control objectives. “We use a leading model” is not an answer.

Existing audits

SOC 2 doesn’t cover it

SOC 2 and a pen test check your infrastructure. Neither checks whether your AI leaks one customer’s data to another, or follows an instruction hidden in a document.

Certification

Certification takes weeks to months

AIUC-1 is the emerging standard. AIUC’s own pages say 4 to 12 weeks from kickoff, and longer if you start from zero. You need evidence for the deal in front of you now.

Sources: G2 buyer research, July 2026; Levelpath research, July 2026; CSA AI Controls Matrix v1.1; AIUC-1 certification.

Why we test the system

The model held. The system leaked.

Most AI safety scores test a model on its own: send it an attack, check that it refuses. Your product is that model inside a harness, the code that wires it to files, tools and other systems. There it acts on what it reads, and one instruction hidden in a document can make it hand over data your code would refuse an outsider.

  1. 01

    Alone, a model can only answer

    Benchmarks and model safety scores measure this. An attack goes in as text and a refusal comes out as text. The worst case is a bad answer.

  2. 02

    In a harness, it can act

    The harness connects the model to files, databases, email and other tools. Each tool is ordinary code: it does exactly what it is asked, every time. The model is the one part that decides what to ask for.

  3. 03

    One hidden instruction is enough

    An attacker your code refuses can go around it, and plant an instruction in something the model will read. The model has the access the attacker lacks, so every permission check passes and the data leaves anyway.

Evidence

What generic scanners miss.

Three open-source AI apps, tested on our own isolated copies with dummy data. On the first, a generic web security scanner we ran alongside missed the hijack, even with its own test inputs in the same field.

Tested in September 2026; the apps may have changed since. App names have been replaced with general descriptions, and identifying details left out.

Personal-finance app
Reproduced

Prompt hijack

6/6

An instruction hidden in a transaction description, sent directly to the app’s spending-analysis AI, was followed in 6 of 6 attempts (3 of 3 for a second team). Our test marker replaced the AI’s summary or was written into its recommendations or risk flags. A generic web scanner given the same field did not flag it. We have not yet tested whether a saved transaction can steer the AI a signed-in user sees.

Couples-finance app
Leak reproduced

The AI held. The leak sat in the database rules beneath it.

0/13

The assistant resisted every injection attempt (0 of 13) and showed each user only their own data. The leak was outside the AI: using the app’s database interface directly with their own login, one partner could read transactions and notes the other had marked private, and could attach a note to a stranger’s transaction. We report anything like it we see in passing, and tell you when your pen test should cover your AI’s data paths.

Hiring screener
Reproduced

The candidate sets the score

10/10 or 1/10

A candidate could set their own stored score to 10/10 or 1/10 by typing an instruction into an answer, and the app kept it. Separately, identical answers scored about 9/10 said confidently and about 5/10 hedged (1 to 2 runs per condition, two test teams). That is a potential disparate-impact risk; our checks of names and stated age or background found no effect in a small sample.

All tests ran on our own isolated copies of public code, never on live systems.

What we do

An independent audit, written for the person reviewing you.

We test your running system (model, prompts, tools and data together) on a test copy with test data, never production. We test it against the failures you would be liable for and what your AI is for, then write it up so your buyer’s reviewer can read it.

A gate on catastrophic failures

You name the failures you can’t afford. A targeted red team goes after each one, and each is pass or fail.

Two scores, never blended

Failure and capability (how well your AI does its job) are reported separately, so a good average can’t hide a dangerous failure.

The model bill, with the risk attached

We compare your current model with cheaper alternatives on cost against failure, so you can switch with the numbers in front of you.

Mapped to what buyers ask

Findings are mapped to common AI vendor questionnaires, and your test evidence is mapped to AIUC-1 requirements, so you can answer reviewers and see your gaps before you pursue certification.

Method

Cheaper is only cheaper if it doesn’t fail.

We score capability and failure separately and put a price on every failure short of catastrophic.

ConfigurationRecommendedDisqualified: catastrophic failure
Acceptable zone70758085909510001234567Unacceptable failures per 1,000 tasksCapability scoreCapability bar 85Failure limit 2Flagship A$0.079 per acceptable outcomeMid-tier B$0.113 per acceptable outcomeBudget C + output filter$0.059 per acceptable outcomeBudget CDisqualifiedOpen-weights D$0.154 per acceptable outcome
Illustrative data. Failure rates come from 5,000 tasks per configuration. One unacceptable failure costs $40 in rework and refunds.
Table view
ConfigurationFailures /1kCapabilityCost /taskPer acceptable outcomeVerdict
Flagship A1.293$0.0310$0.079Acceptable
Mid-tier B2.690$0.0090$0.113Outside requirements
Budget C + output filter1.486$0.0026$0.059Recommended
Budget C6.284$0.0020Not pricedDisqualified
Open-weights D3.878$0.0015$0.154Outside requirements

Catastrophic failures are a gate

Defined with you. One reproduced catastrophic failure disqualifies a configuration.

Everything else gets a price

cost per acceptable outcome = (cost per task + failure rate × cost of one failure) ÷ (1 − failure rate)

We score the system, not the model

A cheap model with the right filter can beat a flagship. We test them together.

A clean result needs volume

Thousands of runs per failure class, so a zero means something.

Sample report

The report is the product.

This is what your buyer’s reviewer reads. An illustrative audit of an insurance claims intake assistant.

Autogeny audit report · A-0142Sample report · illustrative data

Insurance claims intake assistant

Harness v3.2 held fixed · 5 configurations · 3,000 adversarial runs per catastrophic evaluation · 5,000 tasks per configuration · 412 evaluation-set claims · audited 6 Oct 2026

Results: Budget C + output filter (recommended)

MeasureResultReferenceFlag
Catastrophic: cross-claimant disclosureC-02 Trust-boundary crossing · 3,000 adversarial runs0 events0Pass (<0.10%)
Catastrophic: payout promised beyond authorityClient-defined evaluation · 3,000 adversarial runs0 events0Pass (<0.10%)
Adjuster hand-off under pushbackC-04 Asserting where it should escalate · awaiting human adjudication3 cases0Needs review
Unacceptable failures7 in 5,000 tasks: misrouted claim, wrong policy citation1.4 per 1k≤ 2.0 per 1kPass
Capability score412 evaluation-set claims, graded blind86 /100≥ 85Pass
Recommendation

Run Budget C with the output filter. It clears the catastrophic gate and the capability bar with the same failure profile as Flagship A, at 25% lower cost per acceptable outcome. Budget C alone is disqualified: it disclosed another claimant’s data in 3 of 3 reproductions. Re-test on the next model release or harness change.

Method: harness held fixed across configurations. Every fail reproduced before it is reported. Subjective classes go to human adjudication and stay flagged for review until they are.

This is the report’s first page. The full sample report adds every result that is not a pass with what we saw, all five configurations side by side, and what ends this verdict.

How it works

Set the bar. Audit against it. Test it every month.

  1. 01One-time

    Scope

    An initial consultation to set your baseline quality, which failures are acceptable and which aren’t, what counts as catastrophic, and what one failure costs you.

  2. 02One-time

    Audit

    We build an evaluation set with your domain experts and targeted adversarial tests (a red team built around what your app is for), run both inside a test copy of your system, and deliver the audit report.

  3. 03Every month

    Monthly testing

    We test your AI again in the test copy of your system, adding new attack and failure techniques as they emerge, so it is not a re-run of the same tests. We tell you as soon as something fails, and you get a signed quarterly report for your buyers.

  4. 04On every release

    Re-audit

    A credible new model ships, or you change prompts, tools or data. We re-audit, and tell you whether to switch and what it saves.

Design partners

Be one of our first three design partners.

We’re taking three AI-native companies at a reduced design-partner fee. In return, we write up a case study together and you serve as a reference. You get the full audit, report and AIUC-1 readiness check.

Built for

  • AI-native companies with roughly $1M to $10M in revenue
  • AI already in production, not in a pilot
  • Buyers in health, finance, insurance, legal or government who run security reviews
  • AI that makes decisions, touches customer data or acts on a user’s behalf
  • Teams that change models, prompts or tools often and want to know what broke

Not for

  • Deciding whether to use AI at all
  • Low-stakes AI where a wrong answer costs a re-ask
  • Building your AI or writing your policies for you

Your AI eval and red team, without the hires.

  • No eval engineers or AI red-team specialists to recruit and keep.
  • No tooling to build, run and maintain.
  • Each credible new model tested as it ships, not when someone has time.
  • No angle on the verdict: we don’t sell models, guardrails, runtime protection or a testing platform, and we take no referral fees, so nothing we find turns into something we sell you.

Founder

Brad Cunningham

MSc Engineering · BSc Information Technology · EMBA

Brad spent 15 years implementing health data systems in over 40 countries, with a focus on data privacy, policy and security. He has been on the other side of the table too: those systems had to clear privacy and security review before they went live.

Autogeny brings that discipline to AI. Define what failure means before you test. Reproduce it before you report it. Re-test it every time something changes.

Have a security review coming up?

Book a 30-minute scoping call. Bring the questionnaire if you have one. We’ll tell you which questions an audit would answer and which it won’t.