AI audit

Also known as: algorithmic audit · bias audit

On this page 6

An AI audit is a structured, evidence-based examination of an AI system, its data and the organization that operates it, carried out to determine whether the system meets defined criteria for fairness, accuracy, robustness, transparency, privacy and legal compliance. In AI ethics, it is the main mechanism for turning abstract principles into verifiable findings that someone can be held accountable for.

The practice is also called an algorithmic audit, and when it focuses narrowly on discrimination it is often called a bias audit. Unlike an internal model evaluation, an audit is defined by independence, documented criteria and a formal report, and its conclusions are meant to be relied on by regulators, customers or the public.

What an AI audit examines

An AI audit examines the whole sociotechnical system, not only the model file. Auditors look at how training data were collected, how the model was built and validated, how outputs are used in decisions and what governance controls surround the system throughout its lifecycle.

The scope is usually divided into layers, each requiring different evidence. The table below shows the common audit layers.

Audit layerKey questionsTypical evidence
Data auditIs the data representative, lawful and documented?Datasheets, sampling reports, consent records
Model auditDoes the model perform fairly and reliably?Test results by subgroup, robustness tests
Process auditAre development and deployment controlled?Policies, approvals, change logs
Outcome auditWhat effects does the system have on people?Decision logs, complaint records, impact data
Compliance auditDoes the system meet legal obligations?Mapping to regulations and standards
Layers of an AI audit and the evidence they require

A process audit checks whether controls exist and are followed, while a technical audit tests the system directly. Mature audits combine both, because a well-documented process can still produce a discriminatory model, and a fair model can still be misused in deployment.

How an AI audit is conducted

An AI audit follows a defined sequence similar to financial or security audits: scoping, evidence gathering, testing, evaluation against criteria and reporting. The most consequential step is choosing the criteria, because the same system can pass one fairness metric and fail another.

  1. Scoping: define the system, its purpose, affected groups and the decision it supports.
  2. Criteria selection: choose metrics, thresholds and legal or ethical benchmarks before testing.
  3. Access and evidence: obtain data, model access, documentation and interviews with staff.
  4. Testing: run disaggregated performance tests, stress tests and explanation analyses.
  5. Evaluation: compare results with criteria and assess the severity of each gap.
  6. Reporting and remediation: document findings, recommendations and a follow-up plan.

Access level strongly shapes what an audit can prove. The following table compares the main levels of access an auditor may receive.

Access levelWhat the auditor seesWhat can be testedMain limitation
Black-box accessInputs and outputs onlyOutcome disparities, consistencyCauses of bias remain hidden
Gray-box accessOutputs plus documentation and featuresProxy variables, data lineageDepends on documentation quality
White-box accessWeights, code, training dataExplanation methods, root causesCostly and requires confidentiality
Outside-the-box accessOrganizational context and recordsGovernance and accountabilityDoes not test the model itself
Auditor access levels and what they allow

Typical technical tests include disaggregated error rates by protected attribute, checks of demographic parity and equalized odds, and the four-fifths rule for selection rates. Explanation tools such as SHAP or counterfactual analysis help locate proxy discrimination, for example a postal code standing in for ethnicity.

Internal, external and regulatory audits

AI audits differ by who performs them and who relies on the result. Independence increases credibility but usually reduces access, which creates a persistent trade-off between independence and depth.

Audit typePerformed byIndependenceTypical purpose
First-party auditInternal team or internal audit functionLow to moderateQuality assurance before release
Second-party auditContracted external firmModerateAssurance for management or clients
Third-party auditResearchers, journalists, civil societyHighPublic accountability
Regulatory auditSupervisory authority or notified bodyHighEnforcement and conformity assessment
Comparison of audit types by independence

Regulation increasingly makes audits mandatory. New York City Local Law 144 requires annual independent bias audits of automated employment decision tools, the EU AI Act requires conformity assessments for high-risk systems, and the Digital Services Act requires independent audits of very large online platforms. ISO/IEC 42001 provides a certifiable AI management system standard, and the NIST AI Risk Management Framework is widely used as a reference for criteria.

An audit should not be confused with an algorithmic impact assessment. An impact assessment is usually performed before deployment to anticipate harms, whereas an audit verifies actual behavior against criteria, often after deployment.

Limitations and common pitfalls

AI audits are only as strong as their criteria, access and follow-through. Experienced practitioners know several recurring failure modes that weaken the value of an audit report.

PitfallCauseRemedy
Audit washingNarrow scope chosen to guarantee a passPublish scope and criteria in advance
Snapshot biasSingle test of a model that later changesContinuous monitoring and periodic reaudits
Missing demographic dataProtected attributes not collectedLawful inference methods or targeted data collection
Metric conflictFairness metrics are mathematically incompatibleJustify the chosen metric for the context
No remediationFindings are not linked to ownersAssign accountable owners and deadlines
Common pitfalls and how to address them

A fundamental limit is the impossibility result in fairness research: when base rates differ between groups, a classifier cannot satisfy calibration and equal error rates at the same time. Auditors therefore cannot certify a system as fair in every sense; they can only state which criteria were tested and met.

Audits of generative AI add further difficulty, because outputs are open-ended. Auditors rely on red teaming, structured prompt sets and representational harm analysis, and results depend heavily on which prompts were chosen.

In everyday language

An AI audit is like a safety inspection for a decision-making machine. Just as an inspector checks a building against fire codes, an auditor checks an AI system against agreed rules: does it treat people fairly, does it make too many mistakes, and does the company know how it works?

The inspector writes down what was tested and what failed, so others can trust the result. An audit cannot prove a system is perfect, but it shows clearly where problems are and who must fix them.

Seen in the real world

A mid-sized employer in New York uses an AI tool that ranks job applicants for customer service roles. Because the tool qualifies as an automated employment decision tool, the company hires an independent auditor before the annual renewal. The auditor receives anonymized records of about 12,000 applications together with selection outcomes and self-reported demographic categories.

The audit calculates selection rates and impact ratios for each sex and race or ethnicity category. One group shows an impact ratio of 0.72, below the four-fifths benchmark. Feature analysis shows that the model rewards uninterrupted employment history, which penalizes applicants with caregiving gaps. The employer removes that feature, retrains the model and publishes the audit summary as the law requires. A follow-up test three months later shows all impact ratios above 0.85.