Pentesting after AI demands faster work and stricter proof

Pentesting has not become a prompt-writing exercise. It remains the disciplined attempt to discover whether a defined system resists an authorized adversary, and to show the business consequence of a confirmed weakness. NIST describes penetration testing as testing that verifies the extent to which a system, device, or process resists active attempts to compromise security. That definition still matters because it separates a security assessment from a fluent list of possible defects. AI has shortened parts of the work, not removed the obligation to prove a result.

Pentesting’s change starts with the work, not the tools

The practical change is visible before an engagement begins. A tester can use a language model to turn an asset inventory into hypotheses, compare public documentation with a scope statement, summarize authentication flows, and draft questions for an application owner. In a well-run engagement, those outputs are treated as leads. The tester checks them against the authorized target, the actual traffic, source code, logs, configuration, and reproducible behavior. A plausible explanation is not a finding; a real finding needs evidence that another qualified person can review.

That distinction has become more important because models produce confident prose even where the underlying facts are absent or stale. An assistant might identify a framework from a partial response, infer a cloud product from a hostname, or describe a known weakness whose preconditions are not present. Those errors can cost time, but they can also contaminate a report or send engineers toward the wrong fix. The new bottleneck is often verification rather than idea generation. The fastest researcher in the room is no longer necessarily the person who types queries quickest; it is the person who can reject attractive but unsupported answers early.

AI has also broadened the definition of what must be tested. A conventional web assessment still examines identity, authorization, input handling, session management, cryptography, configuration, business logic, dependencies, and operational exposure. When an application contains a model, retrieval system, tool connector, or agent, the assessment has to examine the paths that turn untrusted text into model context and model output into action. OWASP’s web testing guidance remains useful because AI features sit inside ordinary software, not outside it. The same service may still fail on access control, secrets, logging, and unsafe deployment even when its most visible feature is a chatbot.

The second change is economic. Research, sorting, normalization, and report drafting are activities that reward breadth. Models are good at producing a first pass across repetitive material, especially when the material is already available to the engagement. That can make a small team cover more endpoints or spend more time on edge cases. It also creates pressure to generate more output than a client can use. More hypotheses do not automatically mean more security. A crowded backlog of unverified issues can make a program look active while delaying remediation of the few paths that truly matter.

The third change is adversarial. Security teams are no longer merely using AI; they are testing systems that consume it and defending against attackers who use it. NIST’s generative-AI risk profile identifies information-security risks in both directions: models can lower barriers to offensive cyber activity, and they add their own exposure through data, components, interfaces, and deployment choices. This is why a modern assessment needs two lenses at once. One asks whether AI helps the tester reason and communicate. The other asks whether the deployed AI feature creates an unsafe authority path.

The result is not a replacement of craft with automation. Good pentesters still need to understand protocols, applications, systems, identities, cloud controls, and the organization’s real tolerances for disruption. They need to know when a test should stop, when evidence is incomplete, and when an apparent vulnerability is actually an expected control. AI changes the pace of the engagement; human judgment still determines its credibility. That is the frame for every change that follows: faster research, a larger target surface, and a higher standard for evidence.

One practical safeguard is to require a reviewer to identify the source behind every statement of impact before it is assigned a severity. That habit makes the team ask whether the model merely suggested a path or whether the environment actually permitted it. Evidence-led speed is the real advantage.

It also makes post-engagement review easier. When a client challenges a finding, the team can return to an observation rather than defending an opaque chain of model reasoning. That discipline protects trust after delivery.

The conventional engagement still sets the boundary

An AI-assisted engagement still begins with authorization, scope, safety limits, and a shared definition of evidence. That may sound procedural, but it is the protection that keeps speed from becoming carelessness. A model can draft an engagement plan in seconds, yet it cannot determine whether a production dependency is fragile, whether a data set contains personal information, or whether a test might trigger a contractual obligation. Scope remains the control that makes a penetration test legitimate. NIST’s technical testing guide places planning, execution, analysis, and mitigation in a process rather than treating technical activity as an isolated event.

Traditional methodology also provides a useful antidote to model enthusiasm. OWASP’s testing framework covers work before development, during design and development, at deployment, and through maintenance. That lifecycle view is valuable because a weakness found in production often began as a missing requirement, an unreviewed architecture choice, or a deployment shortcut. AI can speed the conversion of notes into test cases, but it does not change the fact that different phases expose different evidence. A generated checklist cannot replace a threat model tied to a real system.

The same principle applies to rules of engagement. An assistant may propose broad discovery tasks, but the tester must make sure that data collection is authorized and proportionate. Publicly visible information may still be sensitive when combined with internal details. Requests that look harmless in isolation may burden an environment when automated at scale. Testing teams therefore need explicit decisions about rate limits, credential use, test accounts, sensitive-data handling, escalation contacts, and stop conditions. AI makes it easier to write those rules down. It does not make them optional.

A clear evidence standard is equally important. For a conventional defect, the tester should be able to describe the affected asset, preconditions, observed behavior, security impact, and a safe reproduction path. For an AI-related defect, the report also needs to identify the source of the untrusted instruction or data, the route into context, the model or agent behavior, the guardrails present, the tool or data access involved, and the consequences. The test record must show the boundary that failed, not merely the words that appeared on screen. That makes the result useful to engineers rather than merely alarming to executives.

This approach counters a common problem with AI-assisted work: tool outputs can appear finished before the underlying question has been resolved. A polished summary may hide uncertainty about an endpoint, an identity role, a dependency version, or a test condition. Teams should mark model-generated notes as provisional until a person has checked the primary evidence. That habit is not bureaucracy. It makes later review possible and helps prevent a model’s assumption from becoming an organizational fact.

The conventional engagement also retains a necessary asymmetry. Defenders frequently want a long list of potential issues, while a pentest must prioritize the paths with credible reach and consequence. A model can classify similar findings or suggest mappings to a framework, but the tester must evaluate exploitability in the actual environment: segmentation, identity controls, monitoring, business process, compensating controls, and the attacker’s starting point. Severity is a property of a real system, not a sentence generated from a vulnerability label.

For AI services, the scope conversation needs one extra question: what authority can the feature exercise? A chat interface with no access to sensitive data and no ability to act has a different risk profile from an agent that can search internal repositories, change records, send messages, or call operational APIs. The difference is not semantic. It determines the tests, the required telemetry, and the safe way to demonstrate impact. The old discipline of defining assets, trust boundaries, and acceptable actions therefore becomes more valuable, not less, in an AI-heavy environment.

This is also the point at which client approval processes earn their keep. A test team should know who can interpret an unexpected result, authorize a narrow follow-up, and decide whether a finding requires immediate containment. AI makes unusual hypotheses appear quickly; the response process must be able to sort them quickly without granting a model-driven guess the status of an emergency. Clear escalation protects both the environment and the credibility of the assessment.

It also prevents a common failure: treating access to a test credential as permission to test every system reachable through it. Credentials define a technical path; the engagement defines the permitted path. Authorization is never implied by connectivity.

Discovery expanded faster than validation

Discovery has always been a mixture of inventory work, documentation reading, observation, and disciplined curiosity. AI changes its tempo. A tester can ask a model to normalize hostnames, compare API descriptions, group similar responses, summarize lengthy configuration files, or derive questions from a design document. That saves attention for the places where the available evidence is contradictory or incomplete. The gain is breadth of review, not automatic truth. OWASP’s testing material still treats information gathering as a distinct part of testing because later decisions depend on knowing what is actually exposed.

The most useful model assistance appears where the input is bounded and reviewable. Suppose an engagement has an approved inventory of routes, cloud resources, or code repositories. A model can cluster related items, identify inconsistent naming, extract likely parameters, and generate a list of ownership questions. The tester can then validate each hypothesis against the system. This is a better use than asking a general-purpose model to invent an attack plan from a company name, because it keeps the work connected to authorized material and makes errors easier to detect.

The danger is false completeness. Models make it easy to turn an incomplete asset list into a confident narrative, especially when public records, repository references, and DNS data partly agree. A missing system is often more important than a badly described one. Teams should therefore retain independent inventory controls: authenticated cloud queries where available, owner confirmation, versioned scope lists, and reconciliation between development, operations, and security records. AI can expose inventory gaps, but it cannot certify that no gap remains.

Discovery also changes when the application itself uses AI. The tester needs to map more than URLs and ports. Relevant assets can include model providers, model versions, system prompts, retrieval stores, embedding pipelines, tool definitions, connector servers, sandboxing controls, agent memory, evaluation datasets, moderation services, logging destinations, and fallback workflows. Many of those components are invisible from a browser but can decide whether an untrusted document becomes an instruction or whether a model can act under a powerful identity. MITRE ATLAS organizes adversarial techniques against AI-enabled systems, which is useful as a prompt to map those new trust relationships rather than as a substitute for local analysis.

The distinction between data and instructions has become fragile. A document imported into a retrieval corpus may look like passive content to a business owner but be treated as meaningful context by a model. A web page fetched by an agent may contain text intended to influence the agent. A tool description may be more persuasive to a model than a user’s request. Testing therefore asks not only “what can enter the system?” but “where can that material be interpreted as a directive?” Every context boundary needs an owner and an explicit trust decision.

This is one reason discovery needs to remain evidence-led. A test team should collect artifacts that show the route from source to action: configuration, request logs, retrieval traces, tool-call records, identity policies, and controlled test data. Screen captures of a surprising model response are rarely enough. The material may be non-deterministic, the prompt may be absent, or the model may have reached an answer through a path the team has not reconstructed. Strong evidence makes remediation more precise because it tells engineers which boundary requires change.

Finally, AI changes the workload distribution. The tester spends less time reading every line of routine documentation and more time checking provenance, conditions, and exceptions. That is a productive trade if the engagement preserves review time. It is a bad trade if management assumes that automatic discovery means an assessment can be shortened without reducing scope. AI accelerates reconnaissance only when the saved time is reinvested in validation. Otherwise, it merely accelerates the production of uncertain claims.

Discovery records should therefore show dates and collection methods, not just conclusions. Systems change, DNS records move, documentation lags, and cloud resources are created or retired without a tester’s knowledge. A dated observation, tied to a source, is easier to revisit than a model-generated description of “the current environment.” Time-bounded evidence prevents an old clue from becoming a new claim.

The same record should capture exclusions. A system omitted because its owner could not confirm scope may deserve follow-up, but it should not silently disappear from the assurance story. Unknown assets are governance debt.

The inventory process should therefore be repeatable, with the same reconciliation steps applied before material retests and releases.

AI turned evidence review into a central task

Evidence review has become the center of gravity in AI-assisted security work. A model can produce a compact interpretation of source code, a proxy log, a cloud policy, or a set of test results much faster than a human can read every artifact from scratch. The advantage is real: the tester can inspect more material, ask better follow-up questions, and identify inconsistencies earlier. Yet the output is a narrative about evidence, not the evidence itself. A security finding should remain traceable to primary artifacts that a reviewer can inspect.

That rule matters because language models are optimized to continue text coherently, not to maintain an auditable chain of custody. They can combine two nearby facts into a claim that neither source established, lose an exception hidden in a configuration, or state a probable interpretation in definite language. In security work, those are not minor stylistic defects. They change remediation priorities. A mistaken claim that an identity has administrative authority, for example, can trigger a costly response or distract a team from a more urgent exposure.

A useful operating model has three layers. First, preserve the raw material: logs, captures, configuration exports, code revisions, command results, and approved test notes. Second, let the model assist with categorization, comparison, and drafting. Third, require a human reviewer to confirm every material assertion against the first layer. The model may summarize the case; it must not become the sole witness. This process also makes it easier to show a client which portion of a report is observed behavior and which portion is risk analysis.

The same discipline improves collaboration. Pentesting commonly involves developers, cloud engineers, identity teams, application owners, legal staff, and incident responders. Each group needs different detail. AI can generate tailored first drafts: a technical reproduction for engineers, a risk story for leadership, and a control mapping for governance staff. Those drafts still need one authoritative evidence set underneath them. Without it, different audiences may receive subtly incompatible accounts of the same issue.

AI systems add further evidence problems because outcomes can vary. A prompt that produces a risky response once may not produce it on the next run. A retrieval system can change as documents are indexed or ranked differently. An agent may take another path when a tool times out or a model is upgraded. The test record should therefore capture the model and configuration version, the relevant inputs, the observed tool calls, timestamps, environmental conditions, and any sampling or randomness settings that matter. Repeatability in AI testing may mean repeatable conditions and measured behavior, not identical prose.

NIST’s AI Risk Management Framework gives organizations a vocabulary for governing, mapping, measuring, and managing AI risk. Its generative-AI profile treats information integrity, data privacy, information security, and component integration as distinct risk areas. Those categories are useful for evidence review because they prevent a team from reducing every concern to a prompt. A factual error, a data leak, a compromised connector, and an unauthorized action may all look like a bad answer in a chat window, but they demand different remedies.

Review practices should reflect that difference. A report needs to state whether the weakness lies in model behavior, application logic, data governance, identity assignment, tool design, or monitoring. It should identify what the test did not establish. It should avoid treating a model refusal as proof that a control works, because another pathway or configuration may remain exposed. Honest uncertainty is part of technical quality.

The deeper change is cultural. In the past, a tester might demonstrate expertise by remembering obscure techniques or finding a hidden endpoint quickly. In an AI-assisted environment, expertise also appears in disciplined rejection: refusing a neat explanation until the system confirms it, preserving the record of the test, and writing a finding that engineers can act on. That is not slower work. It is the work that makes accelerated analysis trustworthy.

A useful review question is: could a skeptical engineer reproduce this claim without access to the model conversation that produced it? If the answer is no, the finding needs better artifacts, clearer conditions, or narrower language. Reviewability is the test that turns analysis into evidence.

That question also reduces dependence on a particular tool. Evidence that survives an independent review remains useful when the model, vendor, or workflow changes. Security knowledge should outlive the assistant that helped find it.

That standard also makes later dispute resolution less costly.

Code analysis moved closer to security testing

AI-assisted coding has moved security testing closer to the daily development loop. Developers use models to explain unfamiliar code, create scaffolding, modify tests, write integration glue, and translate between languages. Testers can use similar tools to locate authorization checks, trace data flows, compare routes, or identify risky patterns for manual review. The benefit is faster orientation inside large codebases, not an exemption from code review. The NCSC’s secure-development guidance continues to emphasize practices such as version control, peer review, and automated testing because delivery speed creates risk when those controls are bypassed.

For a pentester with approved source access, a model can be particularly useful when the codebase contains several services and languages. It can propose where a request is authenticated, flag an inconsistent authorization helper, or summarize a serialization pathway. The tester then checks the actual call graph, unit tests, runtime behavior, and deployment configuration. This is similar to using a code search tool, but more conversational. The model reduces the cost of forming questions; it does not answer them conclusively.

Generated code creates a second concern. A feature may appear to work and still carry weak defaults, excessive permissions, unsafe input handling, obsolete dependencies, or missing error checks. Security teams should resist the temptation to treat “AI-generated” as either uniquely dangerous or automatically reviewed. The useful question is ordinary and concrete: what does this code do, which trust boundary does it cross, and what happens when assumptions fail? The origin of code changes the review workflow; it does not change the need for a security standard.

AI also produces a subtle form of codebase drift. Small generated changes can multiply because they are cheap to make and easy to accept. A temporary bypass, a copied sample connector, or a broad service permission may survive longer than intended. Traditional controls such as code ownership, pull-request review, dependency management, secrets scanning, infrastructure review, and release gates become more important when the rate of change increases. A test team should examine whether the organization has adjusted those controls to the amount of generated or assisted code entering production.

The same pattern appears in test automation. Models can draft unit tests and negative cases, but they may mirror the implementation’s assumptions or invent expected behavior. A pentester can use generated tests as a source of ideas, then seek independent validation. Faults that matter often live at the edges: a role added after the original design, a backward-compatibility route, a feature flag, an error path, a data migration, or a request that combines fields in a way the test author never considered. Independent test design remains the point of security testing.

For AI applications, code review also has to include the orchestration layer. The risk may not be in the model API call itself but in the helper that builds context, the policy that selects a tool, the converter that interprets structured output, or the service identity attached to an action. A secure design keeps policy enforcement outside the model’s prose wherever possible. The application should validate parameters, authorize actions, constrain destinations, and log decisions with ordinary deterministic controls rather than asking the model to police itself.

The test team can strengthen this work by maintaining a library of observed patterns, not canned conclusions. Examples might include unchecked tool arguments, retrieval sources treated as trusted instructions, overly broad connector permissions, missing tenant filters, and prompts used as the only barrier to a destructive action. Each pattern should lead to a specific review question and a safe verification method. That makes AI-assisted code review cumulative without becoming mechanical.

AI has made code reading cheaper; it has not made systems simpler. The strongest programs use the saved time to inspect data flows, authorization, deployment conditions, and error handling more deeply. They do not replace those activities with a model’s confidence score.

Security leaders should also watch for a change in defect shape. Assisted development can create many small, similar mistakes across services when a sample pattern is copied widely. That makes root-cause analysis and code-search-based remediation more important than fixing a single instance. The unit of remediation may be a generation pattern, not one file.

Reviewers should look for shared prompts, copied libraries, reused service accounts, and common templates. Fixing the pattern and adding a targeted regression test prevents the same flaw from returning through the next generated change. Scale requires pattern-level assurance.

This lets teams correct the source of repetition rather than its visible symptoms.

Test planning became a model-assisted discipline

A good test plan begins with decisions that a model cannot make alone: which business processes matter, which assets are in scope, which users or customers could be harmed, and which forms of disruption are unacceptable. AI is useful after those decisions exist. It can turn architecture notes into candidate abuse cases, compare a design against a testing standard, organize questions for workshops, or identify missing details in a scope document. Planning becomes faster when the model is constrained by a real threat model.

NIST’s testing guidance describes assessment methods as part of an organized process for planning, conducting, analyzing, and mitigating technical tests. That framing is still practical. A tester can ask an assistant to structure an engagement around authentication, authorization, input validation, logging, and resilience, but the plan must reflect the organization’s actual assets and controls. A payment workflow, a health-data portal, a developer platform, and an operational technology interface may all use the same cloud services while demanding very different test priorities.

AI helps most when it makes assumptions visible. For example, a model can list what it needs to know before evaluating an agent: what tools it can call, under which identities, which sources can enter its context, what confirmations exist, what is logged, and how a failed action is reversed. The tester can bring that list to system owners and identify unanswered questions. This is better than treating an AI feature as an opaque box, because ambiguity itself is a security signal. An undocumented authority path is a test finding waiting to happen.

Planning also benefits from scenario design. A conventional assessment often builds paths from external entry points toward sensitive systems or data. An AI assessment adds paths from untrusted content toward a model, from a model toward a connector, and from a connector toward business action. The team can consider a low-privilege user, a malicious document in a knowledge base, a compromised third-party integration, an insider with access to prompt templates, or a system fault that changes an agent’s fallback behavior. These are not exotic fantasies; they are ways of checking whether trust decisions match actual operations.

The plan should separate testing of the model from testing of the surrounding system. A model may generate unreliable text without creating a security issue if that text has no authority and no sensitive data. The same model output becomes dangerous when an application treats it as a command, a database query, an approval, or an action under a privileged account. OWASP’s 2025 guidance on LLM and generative-AI risks includes prompt injection, insecure output handling, supply-chain vulnerabilities, sensitive-information disclosure, and excessive agency. Those categories point to different layers of a system and should lead to distinct test objectives.

A planned engagement should also identify safe test data. This is especially important where models have access to internal knowledge stores or production-like records. Teams can create controlled documents, accounts, and benign actions that reveal whether a boundary fails without exposing unrelated individuals or systems. They can pre-agree escalation procedures for any unexpected data exposure. Safe demonstrations are stronger than dramatic demonstrations that create new harm.

Finally, AI complicates sequencing. A tester may discover a suspicious model response early, but the next step should not be broader probing by default. The team needs to understand the context path, the authority available, and the risk of triggering actions. Sometimes the correct move is to pause and ask the owner for telemetry or to reproduce the behavior in a test environment. This is a mark of mature testing, not restraint for its own sake.

Model-assisted planning should produce sharper questions, clearer evidence requirements, and safer tests. When it produces only a longer checklist, it has added volume rather than understanding.

Plans should state which assumptions will be independently challenged. Examples include the assumption that a retrieved document is trustworthy, that a model cannot reach a write tool, or that a user approval is informed. Recording those assumptions turns a vague review into a focused engagement and gives the client a clear account of what assurance the test did and did not provide. An explicit assumption is easier to test than an implied one.

It also helps to define evidence before execution. Knowing which logs, configuration snapshots, and test records will support a conclusion reduces improvisation when a suspected issue appears. Planning evidence makes testing safer.

It also reduces pressure to improvise under an uncertain result.

The human tester now runs a different loop

The pentester’s working loop has changed. Earlier engagements often required hours of manual collection before a pattern appeared: reading documentation, sorting responses, identifying repeated implementation choices, and connecting a minor clue to a business process. AI can condense that first pass. The tester spends more time asking whether the pattern is real, whether it crosses a meaningful boundary, and whether a safer explanation exists. The loop is now hypothesis, evidence, challenge, and only then conclusion.

That change favors testers who are comfortable moving between technical detail and systems thinking. A model may surface a list of similar endpoints, but a person has to recognize that they all rely on a shared authorization middleware. It may summarize a repository, but a person has to notice that a legacy integration bypasses the new control. It may propose a remediation, but a person has to judge whether the fix will survive a migration or a future feature. These are not tasks that disappear because the first draft arrives sooner.

A productive engagement therefore treats the model as a junior research assistant with unusual breadth and uneven reliability. It is given bounded material, clear questions, and no independent authority to decide what is true. Its work is checked against tools and artifacts that have known provenance. The tester keeps notes that distinguish model suggestions from observed behavior. This simple separation protects the integrity of the report and the team’s own reasoning.

The approach also avoids overfitting the test to the model’s language. When a system uses an LLM, testers can become so focused on prompts that they ignore ordinary application defects. The real weakness might be a missing authorization check in the document service, a connector with excessive scope, an insecure secret store, or a logging sink that exposes sensitive context. The model layer may make the problem easier to reach, but it is not necessarily the layer that needs the fix. OWASP’s web and LLM guidance are best read together for this reason.

The working loop also needs deliberate pauses. AI speed encourages a tester to keep generating variants, summaries, and scenarios. At intervals, the team should ask: What have we actually demonstrated? Which assumptions remain untested? Are we collecting information we do not need? What would falsify our current theory? A scheduled challenge step is a practical defense against model-amplified confirmation bias. It is especially useful when a model’s early answer sounds technically sophisticated.

Peer review becomes more valuable in this environment. A second tester can inspect the evidence record without seeing the original model conversation, reproduce the observed behavior, and challenge the severity assessment. That reduces the risk that a persuasive narrative becomes self-reinforcing. It also teaches teams where models consistently help and where they cause errors, allowing the organization to refine its internal use rules.

The client relationship changes too. Faster first-pass work can free time for conversations with system owners, which often reveal the controls that are not visible externally: rate limits, data segregation, break-glass procedures, operational checks, and planned changes. Those conversations should not be skipped in the name of automated coverage. They are part of understanding whether an apparent vulnerability is exploitable and how a recommendation will work in practice.

The strongest testers will not be those who use the largest number of agents. They will be those who preserve a high signal-to-noise ratio while using AI to widen their view. Speed is only a security advantage when it shortens the route from observation to verified remediation. That is the human loop that remains irreplaceable.

The loop should produce durable artifacts, not just better conversations. A concise decision log can record the hypothesis, supporting evidence, disconfirming evidence, reviewer, and final disposition. That record helps a team compare future model performance, train new testers, and defend the report when engineers challenge an issue months later. Reasoning that is written down can be audited; reasoning left in a chat transcript cannot.

This discipline improves handover too. A colleague can continue the work without relying on undocumented intuition or a private chat history. Shared reasoning is more resilient than individual memory.

It gives later reviewers a reliable starting point.

It also creates a record of rejected paths, which prevents the same discarded theory from consuming time in later engagements and gives reviewers a clearer picture of the work already completed.

It makes assurance cumulative.

Reconnaissance at scale needs tighter rules

Reconnaissance is the phase where AI’s apparent advantage is easiest to see and easiest to misuse. A model can digest public documentation, organize an authorized asset list, extract terms from an API definition, compare error messages, and turn scattered notes into a candidate map of technologies and trust boundaries. Those tasks are real work, and reducing their cost lets a tester inspect more carefully. The discipline is to use AI for organization while keeping collection and verification under explicit control.

The control issue begins with data handling. Uploading reconnaissance notes, source fragments, configuration exports, screenshots, or ticket data into an external model can create a disclosure path of its own. Even an internal service needs a retention policy, access controls, and a decision about whether the material is appropriate for the selected model. Test teams should use only approved tools, minimize sensitive inputs, redact what is unnecessary, and retain a record of what was shared. The NCSC’s secure-AI guidance calls for security across design, development, deployment, and operation; those principles apply to the pentester’s assistant as well as the application under test.

A second issue is rate and reach. Models make it simple to produce a long queue of discovery actions, but authorization does not expand merely because an assistant proposed them. A team should preserve normal rules for permitted targets, request volume, credentialed access, third-party services, and data collection. This is particularly important when a target application depends on shared SaaS, content delivery, identity, or AI-provider services. The engagement may authorize testing of the customer’s configuration, not broad activity against every provider named in a response header.

The table below separates useful assistance from decisions that need direct human ownership. It is not a list of product features. It is a way to keep accountability visible during a fast-moving phase.

Where AI fits in reconnaissance

The table makes the handoff from model sorting to human accountability explicit.

A tester should be able to answer where every material assertion came from, when it was observed, and whether the asset was actually in scope. Traceability is the price of using automated breadth responsibly. That record makes later retesting and client review faster and reliable.

AI application discovery adds categories that conventional recon checklists often omit. A team should identify whether the service accepts uploaded files, fetches external content, retrieves internal documents, calls tools, stores memories, or delegates to other agents. It should identify the identities used for those operations, the controls around tool invocation, and the logs available to reconstruct behavior. CISA’s 2026 guidance on agentic AI warns that agentic systems introduce distinct security challenges and recommends security practices across design, deployment, and operation. That supports a practical rule: discover the paths of authority before testing the prompts that travel across them.

A well-run reconnaissance phase also defines a stopping point. AI makes it tempting to keep expanding the inventory because another pattern may be one prompt away. That is rarely the best use of an engagement. The tester should pivot when the mapped attack surface is sufficient to test the highest-risk paths, document known gaps, and return to those gaps only when new evidence justifies it. Coverage is not an endless list; it is a defensible argument that the important paths were examined.

This approach produces a better client outcome. The final report can explain not only which issue was found but which systems, identities, data sources, and operational assumptions were examined. It can show where the organization lacks inventory or observability. It avoids the false assurance that often follows a visually impressive map generated from partial information. AI makes mapping easier. A serious tester makes the map accountable.

Teams should also decide in advance whether their AI assistant may touch client data at all. A private deployment, an approved enterprise service, and a public consumer interface present different confidentiality, retention, and access questions. The answer belongs in engagement setup, together with rules for evidence storage and disposal. A fast workflow that quietly sends sensitive material to an unapproved destination is not a security improvement. The tester’s tooling is part of the threat model.

Retention should be considered after the engagement as well. Drafts, prompts, and source excerpts can remain sensitive even after a report is issued. Teams need a deletion or archival rule that matches their contractual and investigative needs. Convenience copies create lasting exposure.

That rule should cover temporary files, browser history, and collaboration spaces, not only the final report.

Vulnerability triage became faster but not more certain

Vulnerability triage is where AI can save substantial time and create substantial confusion. Security teams receive scanner alerts, dependency notices, bug reports, cloud findings, code warnings, user reports, and test observations. Many describe the same underlying condition in different language. A model can cluster duplicates, extract affected components, identify missing fields, and draft questions for the owner. That is useful triage work, but it is not a severity decision.

Severity depends on context that is often absent from a ticket. A vulnerable library may not be reachable. A risky endpoint may be protected by a strong control. A theoretical data path may contain only test information. A low-complexity flaw may have high business impact if it reaches a privileged workflow. Conversely, an issue with a dramatic label may be mitigated by segmentation, narrow permissions, monitoring, or a requirement an attacker cannot meet. The tester has to evaluate the actual system and describe the conditions clearly.

Models are especially prone to treating standardized labels as complete explanations. A CVE title, a CWE category, an OWASP label, or a scanner rule may provide a starting point, but it cannot establish exposure in a particular environment. NIST’s testing guidance emphasizes analyzing findings and developing mitigation strategies, which is a reminder that raw technical output needs interpretation. A finding becomes actionable when evidence connects it to the system’s real risk.

AI can assist that interpretation by creating a structured evidence sheet. For each candidate issue, the sheet might identify the asset, owner, observed version or behavior, entry condition, reachable data or action, compensating controls, validation status, and open questions. The model can populate a first draft from approved artifacts; the tester should mark every field as confirmed, inferred, or unknown. That small practice is powerful because it makes uncertainty visible before an issue reaches a client report.

AI systems create triage categories that deserve their own handling. A prompt-injection observation, for example, may be low impact in a chat interface that exposes no private data and cannot act. The same observation may be urgent if an agent can read confidential documents or invoke tools with broad permissions. OWASP describes prompt injection as a vulnerability in which user prompts alter an LLM’s behavior; its risk depends on the surrounding controls and authority, not on the mere existence of an adversarial string.

Triage must also distinguish model unreliability from security failure. A model that produces a wrong answer could be a quality concern, a safety concern, or an integrity concern. It becomes a security finding when the wrong answer bypasses a control, exposes protected information, enables an unauthorized action, or undermines a trust decision. This distinction helps clients avoid treating every surprising output as a critical vulnerability while still taking system-level failures seriously.

The same caution applies to generated remediation advice. Models often recommend generic fixes: add validation, use least privilege, sanitize inputs, apply a guardrail. Those phrases may be directionally correct but insufficient. The report should identify where validation belongs, which identity needs less scope, what output must be parsed or constrained, and how the team will verify the fix. A remediation is credible only when it closes the demonstrated path.

Triage quality is visible in the final backlog. A mature program produces fewer duplicated issues, clearer owners, explicit evidence levels, and a defensible order of work. AI can help create that structure at speed. It cannot decide which business risk the organization is willing to accept, and it cannot remove the tester’s responsibility to prove that a condition exists. The point of triage is not to make every alert sound urgent. It is to make the truly important work unmistakable.

A triage meeting should end with a decision, not a paragraph. Each candidate needs a status such as confirmed, disproven, accepted as a limitation, pending evidence, or queued for retest. That discipline stops AI-generated summaries from circulating indefinitely as unresolved risks. It also gives owners a precise way to challenge a claim without dismissing the entire assessment. Status is a control against ambiguity.

The decision record should retain the reason for closure and the evidence reviewed. That makes future retests faster and reveals recurring categories where the organization needs better telemetry or a clearer design standard. Triage is institutional memory when it is documented.

It also makes management reporting more honest because unresolved evidence remains visible.

Owners can then act on it without reconstructing the discussion.

Exploitation remains a judgment test

Exploitation remains the moment when a penetration test stops being a collection of suspicions and becomes a measured demonstration of risk. AI has changed the preparation for that moment: it can help a tester understand protocols, review code, compare error behavior, and design safe test cases. It has not changed the standard. A claimed impact still needs to be demonstrated within scope and with the least harmful method available.

That standard protects both the client and the testing team. A model may suggest a chain of events that sounds plausible, but a tester has to determine whether its prerequisites exist, whether it would cross an agreed boundary, and whether a less intrusive proof is sufficient. The right demonstration might be access to a controlled record rather than a broad export, a harmless action in a test account rather than a production change, or a log-confirmed tool-call attempt rather than an irreversible operation. Professional judgment is central because the safest proof is rarely generated automatically.

The arrival of AI features complicates what exploitation means. A risky model response is not necessarily an exploit. A security impact usually requires a link to a protected asset, unauthorized disclosure, integrity loss, availability impact, or an action the system should not allow. The test must show where the malicious or untrusted influence entered, which safeguards should have contained it, and what authority was actually reached. The exploit path is the chain from influence to consequence, not the cleverness of the input.

This is particularly relevant for agentic systems. An agent may interpret context, select tools, construct parameters, call external services, and write results to another system. Errors can occur in any layer. A pentest should test the control plane around those actions: identity scopes, allowlists, parameter validation, confirmation requirements, transaction limits, environment separation, and audit records. The model’s reasoning should never be the only authorization check for a sensitive action. CISA and partner agencies’ 2026 guidance frames agentic AI as a security issue requiring deliberate adoption and controls, rather than an ordinary chat feature.

AI can also make an exploitation narrative too elaborate. A model may combine a weak signal from one service with a theoretical weakness in another and present a long chain that no attacker could realistically execute. Testers should challenge every link: Is the system reachable? Does the identity have the stated permission? Is the token available? Is the configuration current? Does the downstream service accept the input? Is the result observable? A chain is only as strong as its least verified step.

Conventional caution about availability still applies. Some actions can cause noise, cost, data changes, account lockouts, or operational disruption even when the target is “only” a software service. AI systems can add variable cost, unpredictable external calls, and data-processing side effects. Engagement plans should define which operations require approval, which tools are off limits, and how the team will halt a test. A model’s ability to propose many variations does not make it appropriate to execute them.

A good exploitation record includes the business context. It explains the relevant attacker position, the asset affected, the evidence of access or action, the protective control that failed, and the conditions that would limit impact. It avoids unverifiable counterfactuals. It also states where the test did not proceed because of safety or scope. Those boundaries increase credibility; they do not weaken the finding.

AI has made it easier to imagine exploitation. The pentester’s job is still to prove it carefully. In practice, that means smaller demonstrations, stronger traces, explicit preconditions, and recommendations that fix the failed boundary rather than merely changing the model’s wording.

Testing teams should be cautious about using models to generate exploit steps or execute actions without review. The useful workflow keeps execution under an authorized operator who understands the environment and can stop safely. Models may accelerate preparation, but operational control must remain human-led. Automation must not outrun the engagement’s safety rules.

This is not a limitation on creativity. It is the professional distinction between preparing a test and taking an action that affects another party. Control of execution is a core safety property.

That boundary protects clients from avoidable operational risk.

It also makes it easier to distinguish a technical demonstration from an operational incident and to maintain the trust needed for future testing.

The record should make that difference unmistakable.

It supports safer retesting.

Reporting gained speed and lost a margin for error

Reporting is one of the areas where AI offers an obvious productivity gain. Models can turn raw notes into a coherent draft, adapt a technical explanation for different audiences, compare findings for duplication, and produce first-pass remediation language. That can free a tester to spend more time on evidence and review. The danger is that a fluent report can hide a weak test record. A polished sentence is not an additional fact.

The report’s role is larger than documentation. It becomes the bridge between a temporary test and long-lived engineering work. Developers need affected components, conditions, technical evidence, and fix guidance. Risk owners need to understand impact, likelihood, and residual exposure. Auditors may need traceability to tests, decisions, and remediation evidence. Incident responders may need to know whether the finding describes an active threat path. AI can help tailor the presentation, but each version must stay faithful to the same confirmed facts.

Language models introduce two reporting hazards. The first is invented specificity: a model can add a date, version, configuration detail, or claim of access that was not in the notes. The second is false precision: it can convert uncertainty into a neat severity rating or declare a control absent because it did not see evidence of it. Every material sentence should be checked against an artifact or clearly marked as analysis. This should be an explicit part of the quality review, not an informal hope.

AI-related findings require reporting that is more architectural than theatrical. A screen capture of a jailbreak-like response may attract attention, but it rarely tells engineering what to fix. The report should identify the content source, context construction, model behavior, tool or data route, identity, policy decision, and observable impact. It should distinguish direct prompt injection from retrieval-based or tool-description influence, and it should state whether an action was actually executed or only proposed. OWASP’s LLM risk guidance is useful for naming patterns, but the report must remain grounded in the client’s implementation.

Remediation writing also needs care. “Improve the prompt” is often a weak recommendation because prompts are not a hard authorization boundary. More durable recommendations may include narrowing a connector’s permissions, enforcing server-side parameter checks, separating read and write actions, requiring confirmation for high-impact operations, isolating untrusted content, recording tool calls, and creating regression tests. The appropriate choice depends on the demonstrated path. The fix belongs at the control that should have stopped the action.

Good reports include limits. They explain the time frame, scope, target versions, test accounts, assumptions, exclusions, and conditions that prevented a fuller demonstration. This is particularly important where AI systems change frequently through model updates, corpus changes, prompt revisions, or tool integrations. A result observed on one date may need revalidation after a material change. Clear limits let the client decide what to retest rather than treating the document as timeless truth.

AI can improve consistency too. A team can use it to compare terminology, ensure every finding has required fields, flag unexplained acronyms, and detect conflicting severities. Those are editorial checks. They should not be allowed to rewrite technical claims without review. The safest workflow keeps the source notes, evidence, and approved finding text under version control, with model output identified as drafting assistance.

A strong report turns a verified path into a clear decision. AI makes the prose cheaper; the team must use the saved effort to make the conclusion more exact. That is the only reporting efficiency that matters to a client trying to reduce risk.

There is also a client-facing benefit to restraint. Engineers are more likely to act quickly when a report shows exactly what failed and why the proposed fix will prevent recurrence. Inflated language or speculative impact damages trust and encourages defensive debate. A concise, evidenced finding makes room for the technical conversation that actually reduces exposure. Credibility is a remediation accelerant.

Reviewers should check that screenshots, logs, and text agree. Small mismatches often expose a copied assumption, a changed environment, or a misunderstood condition. Consistency checks catch errors before the client does.

It makes the final report easier to defend and faster to remediate.

Teams should preserve the final evidence package in a form that permits an independent check after personnel, tools, or environments change. That is especially valuable for high-severity findings and recurring control failures.

It also supports fair challenge from the engineers responsible for the system.

Adversarial AI changed the test target

AI has changed pentesting because it has changed the target. An application that once accepted requests, processed data, and returned deterministic responses may now retrieve documents, construct context, call a model, select tools, and take actions through APIs. The visible chat box is only one surface. The security question is whether untrusted influence can cross these layers and reach data or authority that the user should not control. Testing AI systems means testing an interaction system, not a single model.

The model itself can have security-relevant weaknesses: susceptibility to instruction manipulation, exposure of sensitive information, unstable output, or behavior affected by poisoned data. But the surrounding application often determines impact. A model that gives an incorrect response is not necessarily a security incident. A model whose response is treated as an executable instruction, an authorization decision, a database operation, or a trusted summary may become the entry point to a wider failure. NIST’s generative-AI profile separates information security, data privacy, information integrity, and component integration precisely because these risks do not collapse into one prompt problem.

MITRE ATLAS provides a threat-informed vocabulary for AI systems, with tactics and techniques based on real-world observations and realistic demonstrations from red teams and security groups. For pentesters, its value lies in expanding questions: could training or retrieval data be manipulated, could model outputs influence a tool, could a model or component be stolen or altered, could a system be evaded, and could the AI capability itself assist an attacker? It is a knowledge base, not an engagement script. Frameworks guide attention; they do not replace evidence about a specific deployment.

The attack surface begins with data ingress. Content can arrive from a user prompt, a file, an email, a web page, a repository, an enterprise knowledge base, a connector, a tool response, or a memory store. Each source needs classification and controls. The next surface is context construction: selection, ranking, truncation, prompt templates, system messages, and provenance. Then comes action: tools, APIs, credentials, transaction boundaries, output parsers, and human approval. Finally there is observability: logs, traces, model versions, tool calls, and incident response. A weakness at any point can change what the system does.

This makes the old separation between application security and machine-learning security less useful for practitioners. Application engineers may own authentication and APIs; data teams may own corpora; platform teams may own model access; product teams may define agent actions; security teams may own monitoring. The adversary sees one chain. Pentesting has to follow that chain across organizational boundaries. The most serious AI failures often occur in the handoff between teams.

A useful test plan lists the authority granted at every link. Can the model read only selected records, or an entire tenant? Can it invoke read-only tools, or make changes? Are destructive actions isolated? Does an action use the end user’s identity, a shared service account, or a highly privileged agent account? Is every tool call validated by the receiving service? These questions bring AI testing back to familiar security foundations: least privilege, separation of duties, secure defaults, explicit authorization, and auditability.

AI also changes incident thinking. An adversarial interaction may be transient, but a compromised document, connector, prompt template, or memory entry can persist and affect later users. Response plans therefore need ways to identify the affected inputs, pause risky capabilities, revoke credentials, update data or prompts, and retest the full path. The new target is a living system with state, dependencies, and authority. A pentest that only tests clever prompts will miss most of it.

This broader target also changes ownership of the final finding. A model team may need to adjust context handling, while an application team narrows tool schemas, an identity team reduces a service account, and an operations team improves logging. A report that names only “the AI system” will not reach the people who can close the path. Findings must follow the control owner, not the product label.

This also changes remediation workshops. The discussion should include the people who own data, tools, permissions, and operations, rather than being limited to the interface team. Cross-functional fixes match cross-functional risk.

That ownership map should be validated before closure, not assumed from an organizational chart.

It also improves retesting because each team knows what result proves its own corrective work and what evidence must be collected before the issue is closed.

That prevents ownership gaps from becoming residual risk.

Prompt injection is an application security problem

Prompt injection is often described as an AI-specific novelty, but its practical security meaning is familiar: untrusted input influences a system in a way that bypasses the developer’s intended control. The novelty lies in the medium. Instead of exploiting a parser or a query builder, the attacker may shape language that a model interprets inside a context window. The security failure occurs when that influence reaches protected data or a privileged action.

OWASP lists prompt injection as the first risk in its 2025 Top 10 for LLM and generative-AI applications. The category covers direct inputs from a user and indirect inputs embedded in content the system processes, such as retrieved documents or external web pages. The label is useful, but teams should avoid treating it as a magic property of a phrase. Whether an observation is serious depends on the application’s architecture, the authority available, the data involved, and the controls outside the model.

A secure test focuses on the route. First, identify which inputs can enter the model context. Second, identify whether the application distinguishes instructions from untrusted content. Third, inspect what the model can influence: answers only, retrieval queries, tool selection, tool parameters, workflow state, or user-visible approvals. Fourth, verify whether deterministic controls enforce permissions before any consequential action. Prompt defenses are helpful, but server-side authorization is the actual boundary.

This framing prevents two errors. The first is underreaction: a team sees a malicious instruction in a document and assumes it is harmless because the model has a system prompt. System prompts may help steer behavior, but they are not equivalent to a policy engine. The second is overreaction: a team demonstrates an unusual response and labels it critical without showing access or action. A mature finding states the observed influence, the reach of the affected path, and the controls that would contain it.

Testing needs controlled content and a safe endpoint. A team can use benign documents or messages that reveal whether untrusted text crosses into privileged reasoning or tool selection, without asking the system to expose genuine secrets or perform harmful actions. Logs should capture source provenance, the relevant context, policy decisions, and any attempted tool calls. This lets engineers reproduce the issue and decide whether the root cause lies in retrieval filtering, prompt construction, tool design, output validation, or identity scope.

Indirect injection raises a governance problem as well as a technical one. Business owners may regard a document store as a trusted source because it sits inside the company. Yet its content may have been authored by many people, imported from external systems, or copied from public sources. Trust should be based on provenance and authorization, not location. A document can be permitted for search and still be unsafe to treat as an instruction source.

Prompt injection also tests the quality of human oversight. Some systems require a user to approve a proposed action. That can reduce risk, but only if the approval is informed, specific, and difficult to spoof. A vague confirmation after the model has summarized an action may not expose the true parameter values, destination, or data scope. The receiving service should validate the request independently, and high-impact actions should be designed to be reversible where possible.

The strongest defenses use layers: isolation of untrusted content, clear tool schemas, restricted identities, server-side checks, action allowlists, parameter constraints, confirmation for sensitive actions, monitoring, and regression tests. No single prompt format eliminates the issue. Pentesting should therefore test the controls that survive a model misunderstanding. That is the right standard for an application in which language is an input, not a security boundary.

A strong regression test should continue after a first fix. Teams can check whether the same untrusted content reaches a different model, tool, language, or fallback path. They can verify that the action receiver denies unauthorized parameters even if the model still produces them. That is a more durable outcome than chasing a single phrase. The target is resilient behavior, not one blocked string.

Teams should record the rejected routes too. A control that blocked one path may reveal what it does well, while another route may remain untested. Negative evidence sharpens the next test cycle.

It also discourages fixes that merely change surface wording.

That distinction keeps the test focused on authorization and consequence rather than on a contest over phrasing.

It also supports better engineering reviews.

Review outcomes independently.

Tool-using agents change privilege risk

Tool-using agents alter the security model because they move from generating text to requesting actions. An agent may search an internal service, send a message, create a ticket, alter a record, start a workflow, or call an external API. Each tool expands useful capability and introduces a new path for confused, manipulated, or unauthorized behavior. The decisive question is not whether an agent is intelligent; it is what authority its tools place behind its words.

OWASP’s LLM and generative-AI guidance identifies excessive agency as a critical risk. The problem appears when an application gives an LLM more capability, permission, or autonomy than is necessary, then allows unexpected or manipulated output to trigger damaging actions. The category captures an old security lesson in a new setting: do not give a component broad privilege merely because it might need it someday.

A pentest should map each agent tool to an identity and an authorization decision. Does it operate as the requesting user, as a constrained service account, or as a shared administrator? Can it read a single object, a collection, or an entire environment? Is a write action available in production? Does the destination service enforce permission independently, or does it trust the agent’s request? This mapping often reveals risk before any adversarial prompt is tested. Tool access without an explicit privilege model is an architecture flaw.

Identity design deserves special attention. Using the end user’s delegated identity can preserve least privilege and auditability, though it may introduce consent and token-management complexity. A narrow service identity can work for tightly defined tasks, provided its scope is limited. A broad shared account makes implementation easier but produces a dangerous blast radius and weak attribution. The best choice depends on the workflow, but no choice should be invisible. Logs should associate every tool call with the initiating user, the agent, the credential, the requested action, the policy decision, and the result.

Agents also require action boundaries. Read and write capabilities should be separated. High-impact actions should have narrow schemas, server-side validation, rate or value limits, and confirmation from an authorized person where appropriate. The action receiver should reject requests that violate its own rules even when they come from a trusted agent service. The model may propose; the system of record must decide.

CISA and its partners issued guidance in 2026 on the careful adoption of agentic AI services, stressing the security challenges that arise when organizations introduce such systems into IT environments. That timing matters. Agents are increasingly deployed as operational components, yet their behavior can be variable and their tool ecosystems can be complex. The sensible response is not to ban automation. It is to begin with tasks whose failure is contained, measure behavior, and widen authority only after controls are tested.

Pentesters should also inspect fallback behavior. An agent may use another tool when the preferred one fails, retry an action with altered parameters, or seek information from a less trusted source. These routes are easy to miss in a nominal workflow. They need the same identity, validation, and audit protections as primary paths. A crisis often reveals the permission model more clearly than a demo does.

The right test outcome is rarely “the agent was tricked.” The useful outcome is “untrusted content influenced a tool request, and the receiving system accepted it because the agent held unnecessary authority or the action lacked independent validation.” That statement points to durable engineering work. Tool use turns model risk into system risk, so the controls must live in the system.

The testing scope should include human-facing tool descriptions and approval screens. An agent may select a technically safe API but present a misleading explanation of what it will do. Clear, structured displays of object, action, destination, and effect help users recognize an unexpected request. Human approval works only when the human sees the real action.

Operational safeguards matter after execution as well. Rate limits, anomaly detection, and a rapid disablement path reduce harm if an agent behaves unexpectedly despite pre-action controls. Containment is part of authorization.

Those operational controls should be tested under realistic load and failure conditions.

The goal is to ensure that a safe design remains safe when presentation, context, or timing is imperfect.

These checks should be documented as part of the tool contract.

They should describe the input, permitted effects, validation rules, and audit fields without relying on free-form interpretation.

Data pipelines became an attack surface

Data pipelines have become a direct part of the attack surface. AI applications often depend on collections of documents, records, code, messages, images, or tool responses that are ingested, transformed, indexed, retrieved, and inserted into model context. Those steps determine which information the system sees and how it interprets it. A secure AI feature needs data controls that are as deliberate as its access controls.

The first risk is unauthorized exposure. A retrieval system may return documents from the wrong tenant, the wrong role, or the wrong business context if filtering is weak or applied after retrieval. Embeddings do not erase access-control obligations. A tester should examine how the system attaches ownership and permissions to source material, how it enforces those permissions during retrieval, whether cached results respect changes in access, and whether a user can infer sensitive content through summaries or citations. The security objective is not simply to hide a source document; it is to prevent the system from revealing protected information in any form.

The second risk is integrity. A document, record, or connector response may be altered deliberately or accidentally, then used as a source for later model behavior. In a retrieval-augmented system, that can mean a false policy, a malicious instruction, outdated guidance, or misleading context influences many interactions. CISA’s 2025 information sheet on securing AI data emphasizes the role of data security in the accuracy, integrity, and trustworthiness of AI outcomes. Data provenance is a security control when models treat content as context.

The third risk is uncontrolled retention. Teams sometimes create debugging logs rich with prompts, retrieved passages, tool inputs, outputs, and user identifiers. Those records are useful for security investigations, yet they can become a concentrated store of sensitive information. Pentesters should assess who can access them, how long they are retained, whether secrets are redacted, how they are separated by tenant or environment, and whether operational staff can search them broadly. Logging is necessary for accountability, but it needs its own security design.

Data pipelines also complicate change management. A model can behave differently after a corpus refresh, an embedding-model change, a new ranking rule, a connector update, or a prompt-template revision. A secure delivery process should identify those changes, test critical permissions and action paths, and retain enough metadata to reconstruct the system’s behavior later. A model version alone is not enough to describe the deployed system. The relevant version may include data, index, prompt, tool schema, policy, and identity configuration.

Testing should use controlled data wherever possible. Teams can create tagged documents that test access filtering, provenance displays, instruction isolation, and stale-content handling without exposing production records unnecessarily. They can verify that deleting or changing a source is reflected appropriately downstream. They can test whether an agent receives raw text that should have been sanitized or a structured result that preserves trust metadata. These exercises are safer and more informative than attempting to elicit arbitrary confidential content.

The supply chain matters here too. Data may arrive through software vendors, internal business systems, SaaS platforms, open-source repositories, public web sources, or human uploads. Each source has different assurance, ownership, and update processes. The test plan should identify who can add material, who can approve it, whether tampering is detectable, and what happens when a source becomes untrusted. ENISA’s work on AI cybersecurity describes the AI ecosystem through a lifecycle and highlights supply-chain concerns, reinforcing the need to look beyond the model itself.

The secure question is not “does the model know this?” but “why is this content present, who may use it, and what can it influence?” Pentesting brings those questions into a testable form. It turns data governance from a policy statement into a set of observable controls at ingestion, storage, retrieval, context construction, and action.

Data tests should include deletion and correction as well as retrieval. If a source is removed, updated, reclassified, or found to contain harmful content, the organization needs to know how that change propagates through indexes, caches, evaluations, and agent memory. Otherwise a security response may fix the record while stale context continues to influence the model. Data lifecycle controls are containment controls.

Source quality needs routine review. Content that was safe and accurate at ingestion may become outdated, revoked, or compromised later. Trust expires unless it is maintained.

That review should include ownership, access rights, and the mechanism for withdrawal.

This prevents an apparently completed remediation from leaving a second, older copy of the risky material available to the system.

Model supply chains demand conventional security

AI systems have supply chains in the same way that conventional software does, but the chain is broader and harder to see. It can include foundation models, model-hosting platforms, SDKs, orchestration frameworks, embedding models, vector databases, retrieval libraries, tool servers, plugins, container images, data sources, evaluation sets, and monitoring services. A weakness or unexpected change in any of those components can affect the application. Pentesting now needs a component map that includes both software dependencies and AI dependencies.

The practical starting point is a bill of materials with ownership. Teams should know which model is in use, how it is accessed, who can change versions, what data leaves the environment, which connector services are trusted, and which libraries parse or execute outputs. This is not just procurement hygiene. It allows a tester to identify where a security assumption may originate. A third-party tool connector with broad access is a different risk from a local read-only integration. A hosted model with retention controls is a different risk from a public endpoint used without an approved data policy.

OWASP’s 2025 LLM guidance includes supply-chain vulnerabilities among the major risks for LLM applications. The category reflects the fact that compromised or poorly governed components, services, or data sets can undermine confidentiality, integrity, or availability. Security teams should apply familiar practices: inventory, vendor assessment, signed and pinned artifacts where appropriate, change control, patch processes, least privilege, monitoring, and incident response clauses. The model ecosystem does not excuse ordinary supplier-risk discipline.

AI components also introduce a distinct transparency problem. Teams may know the name of a model but lack visibility into its training history, update schedule, behavior changes, or service-side controls. That does not make it unusable; it changes what the organization must test and monitor in its own layer. A client should be clear about what it can verify, what it must accept as a provider claim, and which risks must be contained through architecture rather than assumed away.

Testing supply-chain exposure involves more than checking versions. A pentester can review whether connectors authenticate each side, validate data formats, constrain actions, log requests, and fail safely. They can assess whether a new plugin can be added without review, whether tool definitions are sourced from trusted locations, and whether dependency updates alter production behavior without a security gate. They can examine whether a model or agent receives instructions from a source the system owner has not classified. Trust should be explicit at each integration point.

The risk of component substitution deserves attention. A configuration change can send requests to a different model, a different region, a different embedding service, or a different connector. An attacker who can alter that configuration may redirect data or change the system’s behavior without touching application code. Configuration integrity, secret management, deployment controls, and audit logs are therefore central to AI security. The distinction between “model security” and “platform security” is often artificial in this scenario.

NIST’s generative-AI profile names value-chain and component integration as a risk area. That language helps teams widen the assessment beyond direct model behavior: components need clear ownership, suitability checks, monitoring, and contingency plans. A provider’s security documentation may be relevant, but it should not become a substitute for testing the client’s own configuration and use of the service.

The test outcome should produce operational questions. Can the organization identify every component that handles its AI data? Can it disable a connector quickly? Can it rotate credentials used by an agent? Can it roll back a prompt, corpus, model, or tool schema? Can it tell which customers were affected by a faulty source? Supply-chain security is recoverability as much as selection.

AI has expanded the dependency graph. Pentesting needs to follow it with the same patience previously reserved for cloud accounts, identity providers, libraries, and third-party APIs. The components may be novel; the need for accountable boundaries is not.

A dependency map should be reviewed after incidents and major releases, not left as a launch artifact. Teams need to know which changed component explains a new behavior and which owner can respond. An accurate map shortens both testing and recovery.

This is especially relevant for emergency changes. A fast rollback is useful only when teams know the exact component and dependency path involved. Recovery depends on visibility.

It also makes vendor transitions less disruptive.

Teams should rehearse that map rather than discover it during a crisis.

Red teaming became a broader assurance practice

Red teaming has become a popular label in AI discussions, sometimes used to mean any attempt to make a model say something unusual. That is too narrow for a security program. A serious red-team exercise simulates adversarial behavior against meaningful objectives under agreed rules, then uses the results to improve controls. For AI systems, the work often includes model behavior, data handling, tool use, identity, application logic, monitoring, and response. Red teaming is an assurance practice, not a collection of prompts.

The overlap with penetration testing is substantial. Both seek evidence of exploitable weakness, use authorization and safety boundaries, and translate technical observations into remediation. The difference is often emphasis. A conventional pentest may focus on known system classes and a defined technical scope. An AI red team may explore broader abuse pathways, emergent behavior, misuse scenarios, model evasion, content manipulation, and organizational response. The two should inform each other instead of competing for ownership.

MITRE ATLAS is useful here because it links adversarial-AI tactics and techniques to mitigations and case studies. Its value is not that every organization must test every entry. It is that teams can formulate scenarios beyond model prompts: data poisoning, model theft, evasion, supply-chain manipulation, or malicious use of AI capabilities. Those scenarios should be selected according to the system’s function and authority. A low-risk internal assistant does not require the same exercise as an agent that moves money or controls infrastructure.

A red-team program needs a clear test charter. It should define the assets to protect, plausible adversary positions, rules for data and disruption, observers, escalation paths, evidence standards, and measures of success. It should also specify what is not being claimed. A single successful manipulation does not prove that every user is exposed; a failed attempt does not prove immunity. The goal is to measure the effectiveness of controls under selected conditions and reveal where assumptions fail.

AI makes exercise design harder because of variability. The same interaction may produce different outputs across model versions or sampling settings. Teams should focus on measurable system outcomes: unauthorized data access, attempted tool invocation, policy violation, unlogged action, failure of isolation, or evasion of a detector. Capturing traces, configurations, inputs, and results makes the test repeatable enough to become a regression check. Measure the boundary, not just the conversation.

The red team also needs independence. The people who built an agent may be the best source of architectural knowledge, but they can overlook assumptions they designed into the system. Bringing in security testers, data owners, identity specialists, and operational staff can expose different paths. This is also why a finding must be reviewed by the team responsible for the action receiver, not only by the prompt engineer or model vendor.

NIST’s AI RMF proposes the functions Govern, Map, Measure, and Manage. Those functions offer a practical rhythm for red teaming: establish accountability, identify context and risks, test and assess them, then prioritize treatment and monitoring. The NIST playbook explicitly presents suggested actions rather than a single checklist, which matters because testing must be tailored to the actual system.

The best red-team result is a change in system behavior or control design that can be re-tested. A dramatic demonstration may earn attention, but an evidence-backed regression suite, a narrower identity, a safer tool contract, or a better incident runbook improves the organization long after the exercise ends.

Exercises should also feed a standing regression program. When a red team demonstrates a broken boundary, the organization should preserve a safe version of the test, identify the control owner, and rerun it after changes. This converts a one-time lesson into recurring evidence. A red-team result has lasting value only when it changes the baseline.

The exercises should include operational staff, not only builders. A response team needs to practice interpreting traces, revoking access, and communicating scope under realistic conditions. Assurance includes the ability to respond.

Exercises that omit response work leave a critical assurance gap.

A useful report also names which assumptions survived the exercise, which failed, and which are still unmeasured. That prevents a successful scenario from being treated as a complete security verdict.

It also gives leaders a better basis for allocating remediation resources.

It also helps teams compare exercises over time and recognize whether a control truly improved.

It makes the exercise a practical management tool rather than a memorable demonstration.

It improves resilience.

Automation changed the economics of testing

AI changes the economics of security testing by lowering the cost of some cognitive tasks. A tester can summarize large volumes of text, normalize inconsistent records, draft queries, compare code paths, prepare reports, and generate candidate test cases with less manual effort. That can make more frequent testing practical. It can also create a misleading expectation that a team should deliver the same level of assurance for a fraction of the effort. Cheap text is not cheap verification.

The cost structure has shifted rather than vanished. Time saved in routine review should be spent on architecture, validation, evidence capture, safe reproduction, remediation discussion, and retesting. AI systems add work of their own: mapping data sources, reviewing identities, inspecting tool contracts, testing context boundaries, evaluating changes, and building logs that support investigation. A client that buys “AI-powered pentesting” but funds only a shallow review may receive a large report with a low evidence density.

This matters for service providers as well. Models can help consultants scale research and drafting, but firms need quality controls that protect client data and prevent cross-engagement leakage. They need approved tools, clear policies on what may be uploaded, human review requirements, retention and deletion practices, and training on common model errors. Using AI inside a security practice creates a security program of its own.

Internal teams face similar choices. They can use AI to triage incoming work, identify weak signals in logs, produce first-draft communications, or generate regression tests. Yet each use case carries a decision about data sensitivity, access, oversight, and error tolerance. A team should start with tasks where an incorrect output is easy to detect and does not directly change production. It can then expand only after measuring accuracy and reviewing failure modes.

The economics also affect attackers. The UK NCSC has assessed that AI is changing cyber threat and noted that the field is moving quickly, with technical surprise likely. Its 2025 assessment focuses on AI’s impacts on cyber intrusion through 2027. The point for defenders is not to assume every attacker has autonomous capability. It is to recognize that language models can lower the friction of research, translation, social engineering, and scripting while still requiring human direction, access, and operational skill.

A realistic security program responds by investing in controls that remain useful regardless of attacker tooling: accurate asset inventory, strong identity, secure software delivery, logging, segmentation, tested recovery, incident response, and routine vulnerability management. AI-specific controls are necessary where AI is deployed, but they do not replace the basics. In many cases, a poorly managed credential or a weak authorization check remains more consequential than a sophisticated model manipulation.

Measuring productivity requires care. Counting reports, prompts, test cases, or tickets rewards volume. Better measures include time from confirmed finding to owner assignment, percentage of material claims supported by evidence, rate of duplicated findings, time to reproduce a defect, remediation validation rate, and coverage of high-risk authority paths. A faster process is only better if it produces fewer blind spots and more durable fixes.

There is a market risk in the language of automation. Vendors may describe broad, automated discovery as continuous pentesting or autonomous red teaming. Those products may be useful, but buyers should ask what they actually verify, where human review occurs, what environments they can safely test, how evidence is captured, and which AI risks they do not cover. Naming a product “pentesting” does not change the professional requirement for authorization, scope, validation, and a defensible report.

AI will continue to alter unit costs. The durable advantage belongs to teams that reinvest the savings in the judgment-heavy parts of security work. The economics are favorable only when speed buys proof, not merely more output.

Budget discussions should distinguish acceleration from autonomy. Paying for a tool that organizes evidence or generates draft text may reduce analyst effort. Paying for a system to make unsupervised security judgments creates a different risk and control burden. Procurement should price verification time, not only software licenses.

The strongest programs also preserve manual spot checks. They sample the model’s classifications and summaries, compare them with raw evidence, and revise the workflow when the error pattern changes. Automation needs calibration.

This prevents a productivity claim from becoming an unmeasured risk.

Leadership should ask where the saved time went: into deeper validation and remediation work, or simply into a higher count of generated artifacts.

That distinction should appear in security performance reviews.

Measuring outcomes matters more than counting findings

A security program can easily mistake activity for assurance. AI intensifies that problem because it produces visible artifacts at high speed: summaries, prompts, test cases, tickets, dashboards, and risk labels. Those outputs can be useful, but none tells a leader whether the organization is safer. The right measures describe verified exposure, control performance, and the pace of risk reduction.

For pentesting, a basic measure is evidence quality. Teams should know what portion of reported findings has a reproducible demonstration, clear affected assets, confirmed preconditions, and an owner-approved remediation path. They should track how many findings were later withdrawn, merged, or reclassified because the original evidence was incomplete. A rise in false positives after introducing AI assistance is not a minor tuning issue; it is a sign that speed is outpacing review.

AI systems need measures that follow the path of authority. A team may track which agents have tool access, which tool calls use delegated identities, which actions require confirmation, how many tool calls fail policy checks, and whether every consequential action has an auditable trace. It can monitor access-control tests across retrieval and tool layers, measure how quickly a risky data source can be quarantined, and test whether a change in model, prompt, corpus, or connector triggers appropriate regression checks. The measure should match the boundary that protects the business.

The table below offers a compact set of measures that serve different decisions. It is not a universal scorecard. A high-risk system needs more rigorous targets and independent review than a low-authority internal assistant.

Measures that reveal assurance rather than volume

The measures connect AI activity to controls that protect data, identity, and consequential actions.

These measures should be accompanied by qualitative review. A metric can be gamed if it becomes a target without context. For example, a team could increase the verified-finding rate simply by reporting fewer difficult issues, or reduce time to reproduce by writing shallow findings. Leadership needs to ask whether the measures reflect the highest-risk flows and whether their definitions are stable enough to compare over time.

NIST’s AI RMF is helpful because it frames risk management as a set of functions rather than a single maturity score: Govern, Map, Measure, and Manage. Measurement in that model is connected to context, accountability, and treatment. That is a better fit for AI security than a dashboard that collapses model behavior, data exposure, and operational authority into one number.

Pentest reporting can support this approach. Instead of ending with a static list of vulnerabilities, it can identify which controls were tested, which assumptions were not verified, what telemetry was missing, and which regression tests should be added. It can separate immediate remediation from program improvements. A finding about a broad agent identity, for example, may require a quick scope reduction and a longer project to redesign delegated authorization.

Measurement also matters for procurement. A vendor’s claim that an agent is safe or a model is guarded is less useful than answers about the system’s actual controls: can actions be constrained, can logs be exported, can data sources be isolated, can changes be tracked, and can incidents be investigated? Security buyers should measure what their own deployment can prove, not what a demonstration suggests.

The goal is not perfect certainty. It is a clear view of what has been tested, what is controlled, what remains uncertain, and how quickly the organization can respond when assumptions fail. AI increases the amount of activity. Good measurement preserves the connection between activity and safety.

Metrics also need owners. Someone must review a drop in authorization coverage, investigate a spike in blocked tool calls, and decide whether a regression is a real control failure or a measurement defect. Numbers do not manage security by themselves. A metric becomes useful only when it triggers accountable action.

Metric definitions should be versioned. When a new tool, policy, or environment changes what is counted, a trend line may otherwise imply improvement or decline that is only a measurement change. Comparable evidence needs stable definitions.

Teams should review those definitions alongside the reports they inform.

A small set of trusted measures is better than a large dashboard that nobody can interpret or challenge.

Review them after each material architecture change.

A useful scorecard also separates leading indicators from outcome indicators. Training completion or prompt-test volume may show preparation; verified boundary failures and remediation time show whether the program is reducing exposure.

The review should test whether the measures drive the right corrective work.

Re-test.

Security operations receive new inputs and new noise

Security operations are receiving new inputs from AI-assisted testing and AI-enabled products. A pentest may now generate traces of model context, retrieval decisions, tool calls, policy checks, identity delegation, and agent actions alongside the familiar logs of HTTP requests, authentication events, and infrastructure changes. Those records can materially improve detection and investigation. They can also create a flood of sensitive, ambiguous data. Observability is useful only when teams can distinguish signal, provenance, and consequence.

For a conventional application, an analyst often asks who authenticated, what request was made, which resource was reached, and whether the result was allowed. An agentic system adds questions: what user or workflow initiated the task, which model and configuration were used, which content entered context, which tools were considered, which policy checks fired, which identity called the receiving service, and what action occurred. The system should make those questions answerable without reconstructing the event from scattered vendor consoles.

Logging must be designed for privacy and security. Raw prompts and retrieved documents may contain sensitive business data, personal data, secrets, or adversarial content. A program needs redaction, access controls, retention limits, tenant separation, and a clear rule about when detailed traces are collected. It also needs a way to preserve evidence during an incident without retaining everything forever. The audit trail must be secure enough to investigate the system it records.

AI can assist operations by grouping alerts, summarizing incidents, identifying related events, or drafting investigation timelines. The same evidence discipline applies. Analysts should be able to inspect the underlying records, understand the model’s confidence or uncertainty, and avoid allowing a model summary to alter a live containment decision without human review. A false association can be costly; a missed association can be worse. The model should reduce analyst workload, not create an opaque decision layer.

Pentesting can reveal observability gaps before an incident does. If a tester cannot tell which source influenced a model, which tool was invoked, or which identity carried out an action, the incident-response team may face the same problem under pressure. If a test cannot distinguish a blocked request from a successful but unlogged action, the control cannot be measured reliably. Missing telemetry is itself a security finding when the system holds authority.

The operational model should include containment controls. Teams may need to disable a tool, revoke a token, remove a document from a retrieval index, roll back a prompt or connector configuration, restrict an agent to read-only behavior, or route traffic to a safe fallback. Those actions should be rehearsed. A response plan that requires manual discovery of every agent dependency during an incident is not a response plan; it is a hope.

NIST’s AI risk work and the joint guidance on agentic AI both reinforce lifecycle thinking: security is not a one-time model review but an ongoing activity through deployment, monitoring, change, and response. For security operations, that means treating models and agents as managed services with owners, inventories, logs, change records, and incident procedures.

The operational payoff is not just faster detection. Better telemetry helps developers fix the right issue, helps auditors understand control performance, and helps leaders make proportionate decisions. It shows whether an agent was merely exposed to malicious content or actually attempted a privileged action. It reveals whether a failure is isolated or systemic. Without that distinction, teams will either overreact to strange outputs or underreact to dangerous actions.

AI adds new noise to security operations, but it also creates a chance to instrument trust boundaries more precisely. The teams that benefit will be those that log for accountability, protect the logs, and practice using them when an automated system behaves in an unexpected way.

There is a cultural issue as well. Analysts need permission to say that the system does not yet provide enough evidence to determine what happened. An AI summary may sound decisive under pressure, but a responsible team preserves uncertainty until traces confirm the event. Fast incident communication should never turn ambiguity into fact.

Runbooks should specify which AI-specific artifacts to preserve and who can access them. That reduces hesitation when logs contain sensitive context and decisions must be made quickly. Prepared evidence beats improvised collection.

The same preparation supports a clearer post-incident review.

It should be clear which actions are reversible, which credentials can be revoked, and how the organization will notify affected owners.

This should be tested during ordinary operations, not only after a suspected compromise.

AI in attacker workflows changes realistic scenarios

A credible pentest models realistic attacker behavior without exaggerating what AI has changed. Language models can reduce friction in research, translation, content generation, scripting, and the synthesis of large amounts of information. They may help an attacker produce more convincing messages or work through unfamiliar technical material. They do not remove the need for access, reliable intelligence, operational discipline, or the ability to act on a target system. AI changes the speed and breadth of some attacker tasks, not the laws of security.

The UK NCSC’s assessments of AI and cyber threat make this distinction. Its 2024 assessment noted the importance of quality data and scaling barriers for automated reconnaissance, social engineering, and malware. Its 2025 assessment described a fast-changing picture through 2027 and warned that technical surprise is likely. Those are useful statements for planning because they avoid both extremes: dismissing AI as hype and assuming autonomous compromise is routine.

For testers, the implication is scenario design. An engagement can examine whether a credible attacker with AI-assisted research could identify exposed services, exploit weak identity practices, manipulate staff through personalized messages, or move through poorly segmented systems. It should not presume capabilities unsupported by the scope or evidence. The goal is to test the organization’s controls against plausible acceleration, not to perform a theatrical imitation of an imagined superintelligence.

Social engineering deserves careful treatment. AI can make language smoother, adapt tone, and translate content, which may increase the volume and quality of fraudulent communication. Yet success still depends on the target’s processes: identity verification, payment controls, help-desk procedures, reporting culture, email authentication, and the ability to pause a suspicious request. A security program that strengthens those controls reduces risk whether the message was written by a person, a model, or a template. Good process resists persuasive text regardless of its source.

AI can also assist defenders in the same areas. It can organize threat information, surface anomalies, draft awareness material, and help analysts move through evidence. That symmetry is a reason not to treat “AI use” as a severity multiplier by itself. The relevant question is which control is being pressured and whether the organization can observe and contain failure. A prompt-generated phishing email is dangerous because a payment workflow can be fooled, not because the prose is technologically novel.

Pentesting should adapt by testing decision points. Can staff verify identity through an independent channel? Can a service desk resist a request that sounds urgent but lacks required evidence? Can a system prevent a new payee from receiving a large payment without a second control? Can a developer recognize an unsafe generated code suggestion? Can an administrator detect an unusual agent token request? These are testable questions that translate a broad threat trend into local assurance.

The broader threat model includes AI as a target too. Attackers may seek access to model data, training material, prompts, credentials, tools, or agent workflows. They may use a compromised content source to influence later interactions. They may exploit a weak integration to move from an AI feature to a conventional system. MITRE ATLAS and NIST’s generative-AI profile both support this broader view of adversarial risk.

Security teams should plan for attacker acceleration, not attacker magic. That means investing in identity, asset inventory, secure development, resilient operations, and evidence-led testing. Those measures address the real ways AI changes pressure on a system while avoiding decisions based on fear or marketing.

Threat modeling should also account for the uneven distribution of capability. Attackers will use whatever tools fit their goals, and defenders should not predict a single universal AI-enabled playbook. Tests are strongest when they begin with local assets, incentives, and controls. Plausibility comes from the target environment, not a headline about AI.

The test plan can also include defender time. If an AI-assisted attacker produces many variants, the organization may need controls that absorb volume without exhausting reviewers. Resilience includes workload management.

That requirement should be reflected in exercises and incident planning.

That preparation makes the organization less dependent on perfect detection and better able to absorb a burst of suspicious activity.

It also gives defenders time to investigate without sacrificing critical service availability.

That approach also avoids wasting effort on speculative scenarios while still preparing for credible changes in attacker pace.

That keeps planning tied to control performance rather than speculation about tools.

Teams should document tested assumptions and their limits.

Controls that reduce risk without trusting the model

The safest AI security controls are often unglamorous. They do not depend on a model recognizing a malicious instruction or following a carefully worded prompt. They limit what a component can access, validate what it is allowed to request, and preserve evidence when something goes wrong. Security improves when the system remains safe even after the model misunderstands context.

Least privilege is the starting point. An AI feature should receive only the data and tool access required for its defined task. Read access should be limited by tenant, role, and purpose. Write capabilities should be separate from search or summarization. Sensitive operations should use narrow service identities or delegated user permissions, not a broad shared administrator. This is familiar identity engineering, but it becomes urgent when a model can be influenced by text it did not originate.

Deterministic authorization belongs at the receiving system. A model may propose an action and supply parameters, but the service that owns the data or workflow should authenticate the caller, verify the user’s rights, validate the parameters, enforce policy, and record the result. Tool schemas, allowlists, value limits, and state checks reduce the space in which a generated request can cause harm. The service of record should never accept “the model said so” as proof of permission.

Untrusted content needs separation. Retrieved documents, web pages, user uploads, external tool responses, and memory entries should carry provenance and be treated as data, not privileged instructions. Where possible, the system can pass structured fields rather than raw unbounded text to action-selection components. It can restrict which sources are eligible for sensitive workflows, display citations or origin data to users, and prevent untrusted content from modifying system configuration. These patterns do not eliminate influence attempts, but they narrow their reach.

Output validation is equally important. If a model produces a value that feeds an API, query, file path, message, or decision, application code should parse it against an expected schema and reject anything outside the permitted set. A user-facing confirmation should show the actual action, target, and critical parameters rather than a vague natural-language summary. High-impact operations should be reversible where feasible, with approval flows that are independent of the model. A well-designed agent has friction at the moments where error becomes harm.

Monitoring and containment complete the control set. Teams need logs of context sources, model and configuration versions, attempted and completed tool calls, identities, policy decisions, and errors. They need the ability to revoke tokens, disable tools, remove a data source, roll back a configuration, and investigate who was affected. CISA’s guidance says AI systems, like other software systems, need secure-by-design attention across their lifecycle. That principle is more concrete when translated into these operational capabilities.

The NCSC’s secure-AI guidelines similarly organize security through secure design, development, deployment, and operation. A pentest can examine each stage: whether risks were considered before a tool was added, whether code and configuration were reviewed, whether deployment identities are constrained, and whether operational teams can detect and respond to failure.

Controls should be layered because each can fail. A prompt may be bypassed, a retrieval filter may be misconfigured, a token may be over-scoped, or a monitoring rule may miss a new pattern. Layers make failure less catastrophic. They also make testing more productive because a team can identify which control stopped an unsafe path and which one needs improvement.

The best security design assumes the model will occasionally be wrong, manipulated, or unavailable. It places authority in systems that can validate it, limits blast radius, and gives people enough visibility to intervene. Pentesting then becomes a way to verify those claims under controlled conditions.

Controls also need routine negative testing. Teams should verify that a user cannot retrieve a record outside their role, that an agent cannot call an unapproved tool, that a malformed output is rejected, and that a revoked credential no longer works. These tests make security properties concrete and catch drift before a live incident. A control that is not tested will eventually become an assumption.

Recovery should be tested too. A team that can block an unsafe action but cannot identify or correct its side effects has only partial control. Secure design includes repair.

Repair paths deserve owners, runbooks, and tests.

Those tests should include partial failures, delayed logs, and configuration drift, since real incidents rarely follow a clean path.

Guardrails are not security boundaries

Guardrails are useful, but they are often described with more confidence than their technical role supports. A guardrail may be a system prompt, a content filter, a classifier, a policy model, a response template, or a check that blocks certain requests. These mechanisms can reduce unsafe outputs and improve user experience. They are not, by themselves, a reliable authorization boundary for data or actions.

The reason is structural. Guardrails operate in a system that processes ambiguous language, changing context, model updates, and sometimes adversarial inputs. They can be bypassed, misconfigured, overloaded, or simply misunderstood by the model. A classifier can make an error; a prompt can lose priority; a policy can cover one language or scenario better than another. Even a high-performing control may not provide the assurance required before a system changes a record, sends sensitive information, or initiates a financial transaction.

OWASP’s LLM guidance highlights several related risks, including prompt injection, insecure output handling, sensitive-information disclosure, and excessive agency. Read together, they point to the same engineering lesson: a model-facing control is not sufficient when an application gives model output real authority. The application must apply ordinary security controls after the model generates its response.

Pentesters should test guardrails honestly. They can assess whether a filter detects known unsafe patterns, whether a prompt template resists simple manipulation, whether a policy generates useful audit signals, and whether blocked interactions are handled safely. They should avoid presenting a single successful bypass as proof that every guardrail is useless. The valuable question is whether the guardrail is the only thing preventing a serious outcome. A guardrail failure becomes high risk when no independent control limits the consequence.

The system’s interface matters. A chatbot that displays a questionable answer to a human who must independently act on it is different from an agent that automatically calls a tool. The first may require content quality controls, training, and clear disclaimers. The second requires strict identity, authorization, parameter validation, transaction design, and logging. Treating both as “prompt security” obscures the difference in consequence.

Guardrails can also create a false sense of coverage in testing. Teams may run a suite of adversarial prompts, see high refusal rates, and conclude that the system is secure. Those tests are useful as behavioral measurements, but they do not evaluate the data pipeline, connector permissions, service-side checks, or incident response. A well-designed test suite includes negative tests for prompts and independent tests for every consequential integration. Behavioral resistance and system security are related but not interchangeable.

The same caution applies to vendor claims. A provider may have strong safety systems at the model layer, but the customer still owns deployment decisions: which data goes into prompts, which tools are exposed, which identities are used, what output is accepted, and what is logged. The provider’s controls may reduce residual risk, yet they cannot replace controls in the customer’s application and environment.

A practical design principle is to use guardrails as one layer of defense and as a source of telemetry. They can block obvious misuse, guide the model, and alert teams to suspicious interactions. Then place the hard boundaries in deterministic services and identity systems that can make enforceable decisions. The NIST AI RMF’s emphasis on managing risk across an AI system supports this broader approach rather than a narrow focus on model behavior.

A guardrail is a helpful brake. It is not a seat belt, a locked door, and a driver’s license at once. Security testing should ask which of those protections the surrounding system actually provides.

That distinction changes remediation priorities. A weak prompt deserves attention, but a broad service account, an unrestricted tool endpoint, or missing parameter validation usually creates the higher-consequence problem. Teams should fix the durable control first, then improve guardrails as a supporting layer. Put the strongest protection closest to the protected action.

A review should identify where guardrails fail open, fail closed, or silently degrade. Those behaviors affect availability, user trust, and the chance that staff will bypass the protection. Failure modes deserve explicit tests.

They should be documented in release criteria and incident procedures.

A mature team also tests whether a guardrail itself creates a bypass route, a denial of service, or an incentive for users to seek an unsafe workaround.

This applies to fallback models and emergency operating modes as well.

Review it after changes.

Identity is becoming the control plane for agents

Identity is becoming the control plane for agents. Once a model can interact with tools, APIs, repositories, messaging systems, or operational services, every meaningful security question leads back to who or what is allowed to act. The model may choose an action, but the identity attached to the request determines the blast radius. A secure agent architecture begins with an explicit, testable identity model.

There are several common patterns. An agent may act with the end user’s delegated identity, preserving the user’s existing permissions. It may use a dedicated service identity with tightly bounded scope for a specific workflow. It may require a human approver to use a separate identity for high-impact actions. Or it may rely on a shared account with broad access. The last option is often easiest to deploy and hardest to defend because it weakens attribution, expands access, and makes policy enforcement coarse.

Delegated identity is attractive because it allows the receiving service to make its usual authorization decision. If a user cannot access a record through the normal application, the agent should not bypass that rule. This pattern needs careful token handling, consent, and session design, but it aligns AI actions with the organization’s existing access model. Service identities are appropriate when an agent performs a narrow back-office task, provided the permissions are specific and periodically reviewed. No agent identity should be broader than the action it is designed to perform.

A pentest should examine token lifecycle and boundaries. Where are credentials stored? Can a prompt, retrieved document, tool response, or configuration change cause the agent to use a more powerful identity? Are tokens scoped to an environment, tenant, API, and duration? Is there a way to revoke them quickly? Does the tool receiver validate the token and the request independently? These questions are familiar to cloud and API security teams. AI makes them visible in workflows where people may otherwise focus only on model behavior.

Identity also improves auditability. A high-quality record ties an action to the initiating user or process, the agent, the model configuration, the credential used, the requested operation, the policy result, and the system response. This supports detection, dispute resolution, remediation, and compliance. A shared credential that masks all of those links may be acceptable only for very limited low-risk tasks. Without attribution, an autonomous action is difficult to govern.

The design should include separation of duties. An agent that can propose a change should not automatically approve and execute it under the same broad identity. A system can require an independent role for approval, enforce limits in the action service, and record the final decision. This is not a rejection of automation. It is a way to confine automation to a role that the organization can monitor and reverse.

The 2026 joint guidance on agentic AI services places emphasis on the cybersecurity challenges associated with introducing agentic systems into IT environments. Identity management is an obvious practical response because it converts a variable model interaction into an accountable request to a controlled service.

Identity-aware testing should also cover nonhuman actors. Connectors, tool servers, orchestration services, evaluation pipelines, and scheduled jobs may all hold credentials or invoke APIs. An attacker does not need to compromise the visible chat interface if a background integration offers a weaker route. The full agent identity graph belongs in the scope of a serious assessment.

AI applications often make autonomy look like the key design choice. In practice, the more important choice is which identity carries the action and which system can stop it. That is why identity is the control plane.

Identity reviews should be repeated when an agent gains a new tool, a new data source, or a new operating mode. Permission sprawl can emerge one useful integration at a time. A quarterly access review is helpful, but high-risk changes should trigger an immediate check. Autonomy grows through permissions, so permissions need change control.

Identity telemetry should distinguish a denied action from a successful action that later failed. That detail tells the team whether a policy worked or an operational error merely prevented impact. Controls need observable outcomes.

This is particularly important where actions affect more than one tenant or customer.

It allows teams to decide whether the policy, the receiving service, or the operational workflow needs adjustment.

It also makes incident investigations less dependent on informal knowledge.

It supports prompt containment.

Governance must be testable

Governance often fails when it is treated as a set of documents separate from engineering. An AI policy that says “use secure practices” does little unless it identifies owners, decision rights, data classifications, change controls, test requirements, logging expectations, and incident procedures. Governance becomes credible when a tester can observe it in the system.

A practical governance model starts with inventory. The organization should know which AI systems exist, their business purpose, owners, model providers, data sources, tool access, deployment environments, and risk classification. This includes experimental features and internal assistants, because shadow deployments frequently bypass the controls designed for official products. Inventory is not glamorous, but it is the prerequisite for deciding what needs a security review or a red-team exercise.

The next step is accountability. Product owners should know the business purpose and acceptable failure modes. Engineering teams should own implementation and remediation. Data owners should decide which sources may be used. Identity teams should govern credentials and access. Security teams should define testing expectations and review evidence. Legal, privacy, and compliance teams may have additional roles depending on the jurisdiction and use case. An AI system without a named risk owner is difficult to test and impossible to govern well.

Change management matters because AI behavior can change without a traditional code release. A new model version, a modified prompt, a refreshed knowledge base, a connector update, an added tool, or a revised moderation policy can alter security posture. Governance should define which changes need review, what regression tests apply, who approves higher-risk changes, and how a rollback occurs. The implementation should produce records that prove the process happened rather than relying on an informal memory of a discussion.

NIST’s AI RMF organizes organizational work around Govern, Map, Measure, and Manage. That structure gives governance a direct connection to testing. Govern assigns policies and accountability; Map identifies context and impacts; Measure assesses risks and controls; Manage prioritizes and responds. A pentest or red-team result feeds each function by testing whether the stated control environment exists in practice.

ISO/IEC 42001 provides an AI management-system standard intended to help organizations establish, implement, maintain, and continually improve management of AI-related risks. It does not replace technical testing, but it reinforces the principle that AI needs a repeatable management system rather than a one-off project review.

A tester can make governance concrete by asking simple questions. Who approved this data source? Which risk assessment covers the tool connector? Where is the evidence that the agent’s actions were tested? Which person can disable it? Where is the incident runbook? How are model or prompt changes recorded? Which dashboard shows unauthorized tool requests? These are not paperwork questions. They reveal whether the organization has the ability to recognize and control a failure.

Governance also requires exception handling. Teams will sometimes need to use a high-capability model, a broad data source, or a temporary tool integration. An exception should identify the owner, reason, compensating controls, expiration date, and review cadence. Otherwise, temporary risk becomes permanent architecture.

The testable form of governance is a working system of decisions, evidence, and accountability. AI may be new, but the discipline is familiar: define the asset, assign the owner, set the rule, enforce it technically, watch it operate, and revisit it after change.

A governance review should also confront incentives. Teams under delivery pressure may bypass review steps, reuse a broad credential, or import a convenient data source. Leaders need routes for escalating those tradeoffs and must treat security evidence as part of readiness, not a delay tactic. Governance fails when speed is rewarded and accountability is optional.

Regular review forums make this visible. They can compare planned controls with live configurations, open exceptions, test findings, and incidents. A policy without a feedback loop becomes a historical document.

That feedback should feed directly into engineering priorities.

The review should record disagreements and risk acceptance decisions, so future testers can see why a control exists and when it must be revisited.

That transparency helps prevent a known risk from becoming a forgotten exception.

Those records should connect directly to release gates, access reviews, and incident exercises. A governance program becomes durable when its decisions change what teams build, deploy, and monitor, rather than adding a separate administrative layer around the work.

That connection gives governance practical force in daily engineering decisions.

It turns policy into operating practice.

Legal and assurance requirements have broadened

AI security is increasingly tied to legal, regulatory, contractual, and assurance expectations. The exact obligations depend on jurisdiction, sector, system role, and the nature of the deployment, so a pentest cannot declare legal compliance on its own. It can, however, generate evidence about controls that legal and assurance frameworks expect organizations to understand: robustness, cybersecurity, data governance, traceability, human oversight, incident handling, and supplier management. Testing supplies facts; legal interpretation supplies the conclusion.

The European Union’s AI Act is one prominent example. Regulation (EU) 2024/1689 includes Article 15, which states that high-risk AI systems must be designed and developed to achieve an appropriate level of accuracy, robustness, and cybersecurity and to perform consistently in those respects throughout their lifecycle. The regulation also requires relevant documentation around tested and validated performance. The application of specific provisions depends on the system and timeline, so organizations should use qualified legal advice for their own situation.

For security teams, the practical implication is not to create a generic “AI compliance test.” It is to map technical evidence to the organization’s obligations and claims. A team may need to show that access controls were tested, that changes are traceable, that incidents can be investigated, that data sources are governed, that a model’s limitations are documented, or that human oversight works in the intended workflow. These are engineering and operational facts that support a broader assurance process.

The EU’s Cyber Resilience Act and sector-specific rules may also intersect with AI-enabled products, particularly where software components, vulnerability handling, or product security obligations apply. The exact legal analysis is context-dependent. A prudent program avoids assuming that an AI label removes ordinary software-security expectations. AI systems remain software and service systems with traditional security duties.

Standards help organizations structure that work. ISO/IEC 42001 sets out requirements and guidance for an AI management system, while ISO/IEC 23894 provides guidance for AI risk management. Neither standard substitutes for a threat model or a technical assessment. They provide common management structures that can make security evidence easier to organize, review, and improve over time.

Contractual assurance matters too. Customers may ask whether prompts and data are retained, where processing occurs, whether tools can reach other systems, how incidents are notified, which subprocessors are involved, and how the provider changes models or connectors. Pentesting can test the technical side of those statements. It should not repeat unsupported promises from a vendor questionnaire. A strong report distinguishes observed behavior, configuration evidence, provider documentation, and unverified assumptions.

The distinction between compliance and security is important. A system may meet a documented process requirement and still have a serious design flaw. A system may have strong technical controls but incomplete evidence for an audit. Both problems matter, but they need different remedies. Assurance is strongest when governance evidence and technical reality agree.

Testing should also be proportionate. A low-impact internal summarization tool may need inventory, approved data use, basic access controls, logging, and periodic review. A system that influences eligibility, medical decisions, financial actions, or critical operations needs deeper analysis, stronger testing, and more independent oversight. The test plan should reflect the potential harm and authority, not merely whether a model API is present.

Regulation is pushing AI security toward continuous assurance rather than a one-time review. Pentesting fits that direction when it produces traceable evidence, tests changes, and helps teams demonstrate that controls work throughout the system’s lifecycle.

Testing evidence can support contractual discussions without pretending to settle them. A report may show that an access-control test passed, that logs are incomplete, or that a change process is undocumented. Counsel and assurance leaders can then connect those facts to duties, representations, and risk acceptance. Technical findings are strongest when they preserve the boundary between proof and legal opinion.

Documentation should avoid ambiguous terms such as “secure” or “compliant” without a stated scope and basis. Those words can imply a conclusion that the assessment did not establish. Precision protects both the organization and its readers.

It also makes external review and internal decision-making more reliable.

It also prevents an audit-oriented checklist from hiding an unresolved engineering weakness or a business decision that has not been formally accepted.

The testing plan should identify which observations are technical evidence, which are operational claims, and which require further review by counsel or another assurance function. It should then record the resulting decision, owner, and review date.

Team skills are changing rather than disappearing

AI is changing security roles, but it is not making core security expertise obsolete. The most valuable skills remain the ability to understand a system, identify trust boundaries, validate evidence, reason about identity and authorization, communicate risk, and design tests that do not create unnecessary harm. AI changes where those skills are applied and how quickly a team can move through preliminary work. The profession is shifting from manual throughput toward evidence-led systems judgment.

Pentesters increasingly need fluency across application security, cloud security, identity, data engineering, and AI architecture. They should understand the difference between a model, an application, a retrieval pipeline, a connector, a tool, and an agent. They should be able to read traces of context construction and tool calls, ask useful questions about model and data versions, and recognize when a model behavior issue is actually a conventional authorization flaw. No one needs to become a machine-learning researcher to test an AI product, but no tester can ignore the architecture around the model.

Communication skills become more important because AI security spans teams with different vocabularies. A data scientist may focus on evaluation results. An application engineer may focus on tool schemas. An identity engineer may focus on tokens and scopes. A product owner may focus on user experience. A tester needs to connect those views into a single path of authority and evidence. The ability to explain a failure across disciplines is now a security control in itself.

Teams also need skills in model-assisted work. They should know how to write bounded questions, avoid uploading sensitive material to unapproved services, preserve sources, detect fabricated details, and challenge generated conclusions. This is closer to research discipline than to prompt artistry. A useful internal policy treats AI output as untrusted until it is verified, especially when the output contains a factual claim, a configuration detail, or a recommendation that will be implemented.

Training should include safe hands-on exercises. Practitioners can work with controlled agents, test identities, synthetic corpora, harmless tool actions, and trace data to learn how context, retrieval, and permissions interact. They can practice writing findings that describe a complete path from untrusted input to outcome. They can review examples where a prompt-based fix fails because the downstream tool still accepts an over-privileged request. Skills grow faster when the system makes cause and effect observable.

Organizations also need role clarity. A security team should not be expected to own every model decision or data classification. Conversely, product teams should not assume that a generic security review covers the special risks created by tools and autonomous actions. A shared responsibility model needs named owners for data, models, applications, identities, operations, and assurance. The tester can reveal gaps in that model, but cannot fill them indefinitely through one-off assessments.

The NCSC’s secure-AI guidelines are aimed at providers across the lifecycle and explicitly address designers, developers, managers, decision-makers, and risk owners. That breadth reflects reality: AI security is not a niche specialty performed after launch. It is a set of responsibilities distributed across the people who create and operate the system.

Career development should therefore reward careful work as well as speed. Analysts who find a model error before it enters a report, engineers who reduce a tool’s scope, and product owners who reject an unsafe integration are all contributing to security. Automated assistance can make those decisions more informed, but it cannot make them for the organization.

The future pentester is not replaced by AI. The future pentester is expected to test more complex systems with stronger evidence and broader architectural awareness. That is a demanding role, and it is where human judgment earns its value.

The team should retain conventional depth. A tester who understands network exposure, session handling, infrastructure-as-code, cloud policy, and software supply chains will recognize problems that a model-focused review misses. AI literacy expands the toolkit; it must not narrow the security foundation. The hardest failures still cross technical layers.

Mentoring should reinforce this balance. Senior practitioners can show newer staff how to test a model-connected feature while preserving the same skepticism applied to any complex distributed system. New interfaces do not repeal old security lessons.

The result is a wider, rather than shallower, professional practice.

It also helps teams avoid an artificial divide between AI testing and application security, since the most consequential failures often require knowledge of both.

That perspective makes training investments more practical and less driven by hype.

Procurement and vendor testing need new questions

Procurement has become a security control for AI-enabled systems. A product team may be able to connect a model, a retrieval service, a tool framework, and a set of external APIs in days. The organization still needs to know what it is buying, what data will move, what authority the service receives, and what evidence it can obtain when something fails. A vendor questionnaire is useful only when its answers lead to testable design decisions.

The first procurement question is purpose. What exact task will the product perform, and does that task require a model or an agent? The 2026 joint guidance on agentic AI services encourages careful adoption, a useful principle because autonomous tool use creates risks that a simpler workflow may avoid. A low-risk use case such as drafting internal text has different security needs from a service that reads sensitive records or executes transactions. The buyer should select capability proportionate to the task, not merely the most flexible tool available.

The second question is data. What information enters the service? Is it retained, used for model improvement, stored in a particular region, or shared with subprocessors? Can the customer prevent sensitive content from reaching the model? Can it export or delete records? What happens to prompts, retrieved context, tool logs, and evaluation data? Contractual answers should be confirmed against technical configuration and observed behavior where possible. Data-flow diagrams belong in the procurement file, not only in an engineering wiki.

The third question is authority. Which tools, connectors, or integrations are available? Can the product use customer identities or only a shared service account? Are tool calls subject to customer-controlled allowlists and server-side policy? Can the buyer separate read from write functions, require approval, set limits, and disable a connector promptly? A product that offers impressive autonomy but weak identity integration may be unsuitable for any sensitive workflow.

The fourth question is observability and response. Can the customer obtain logs that show model version, configuration, source provenance, tool calls, identities, policy results, and errors? Can it integrate those records with the organization’s monitoring? Can the vendor explain an incident, provide timely notification, and support investigation? Can the customer roll back a model change, a prompt, a tool definition, or a data connection? If a service cannot be observed or contained, it is hard to assure.

Security testing should begin before a full production contract. A buyer can ask for a controlled environment, documented interfaces, test accounts, and evidence of security practices. It can run a scoped assessment of identity flows, tenant isolation, tool permissions, output handling, logging, and recovery. It can test claims made in sales material against the actual deployment. This is not adversarial procurement for its own sake; it is a way to avoid building a critical workflow on assumptions.

The vendor relationship must remain active after purchase. Model providers may change behavior, pricing, regions, retention terms, features, tool ecosystems, or security documentation. Connectors may add capabilities. Frameworks may change their defaults. A mature program reviews material changes, requires notice where contractually appropriate, and retests important paths. The security posture of an AI product is a moving property, not a checkbox at signature.

NIST’s AI risk guidance and ISO’s management-system approach both support this lifecycle view: responsible adoption requires ongoing risk management, documented responsibilities, and continual improvement. A procurement process that records only a vendor’s marketing claims cannot meet that standard.

The most useful buying question may be the simplest: “Show us how our controls remain in force when the model is wrong.” A serious provider and an accountable internal team should be able to answer with identities, policies, logs, tests, and recovery procedures—not a promise that the model will behave.

Buyers should insist on exit options. They need a plan for exporting data, replacing a provider, disabling high-risk features, and retaining sufficient evidence for an investigation after a contract ends. Dependence without recoverability turns a service choice into a security exposure. A secure purchase includes a safe way out.

A pilot should include failure tests, not only a successful demonstration. Buyers learn more when they see what happens after a denied request, unavailable model, poisoned source, or revoked token. A demo proves capability; failure testing proves control.

Those tests should be documented before the purchasing decision is irreversible.

That evidence gives procurement, security, and product leaders a common basis for deciding whether the service is suitable for the planned authority.

A mature practice combines speed with proof

AI has changed pentesting in two connected ways. It has made portions of security work faster, especially research, organization, comparison, drafting, and the generation of candidate test cases. It has also made systems harder to test because models, retrieval pipelines, tools, agents, data sources, and identities can combine into new paths of influence and authority. The mature response is neither to reject AI nor to trust it blindly. It is to use speed in service of stronger proof.

The classic foundations still hold. A test needs authorization, defined scope, safe methods, reproducible evidence, clear impact, and remediation that addresses the actual weakness. NIST’s definition of penetration testing and its technical testing guidance remain relevant because they center resistance to active compromise and structured assessment rather than fashionable tools. OWASP’s web testing guidance remains relevant because AI features are embedded in ordinary applications with ordinary authentication, authorization, configuration, and dependency risks.

What changes is the map. A pentester now needs to trace not just a user request to a service, but an untrusted content source to a context builder, a model response to a tool selector, a tool request to an identity, and an identity to a protected action. The test must examine whether data is properly classified and filtered, whether instructions are isolated from untrusted content, whether agents hold only necessary permissions, whether services validate requests independently, and whether logs reveal what happened. The system needs to be safe when the model is mistaken.

This produces a clearer division of labor. AI is good at expanding the tester’s field of view, organizing material, drafting hypotheses, and translating evidence for different readers. Human professionals remain responsible for scope, safety, interpretation, validation, prioritization, and accountability. The model may accelerate the search for an answer; the tester decides whether the answer is supported and what risk it represents. That is not a temporary limitation. It is the core of credible security work.

The immediate practical priorities are straightforward. First, maintain an accurate inventory of AI applications, data sources, tools, connectors, and identities. Second, limit authority through least privilege, separation of read and write actions, server-side authorization, and reversible workflows. Third, preserve evidence through secure logs and traces. Fourth, treat model, prompt, data, connector, and policy changes as changes that may require security testing. Fifth, use red-team exercises and pentests to verify that controls work across the full path, not merely at the chat interface.

Governance turns those priorities into repeatable practice. NIST’s AI RMF gives organizations a structure to govern, map, measure, and manage risk. The NCSC’s secure-AI guidance organizes security across the system lifecycle. CISA and partner agencies’ recent guidance on agentic AI emphasizes careful adoption. Together, those sources point toward a practical standard: security must be designed into the system, tested in operation, and revisited as the system changes.

The broader lesson is about evidence. AI can generate a polished explanation of a system that does not exist, a vulnerability that is not reachable, or a remediation that does not close the path. It can also help a skilled team find a real issue faster and explain it more clearly. The difference lies in process: source control for facts, primary artifacts for claims, human review for conclusions, and tests that measure system behavior instead of verbal confidence.

Pentesting after AI is still pentesting. It is simply more demanding about boundaries. The tester must understand the traditional system and the AI layer; the organization must secure both the model-adjacent behavior and the identities, data, applications, and operations around it. Teams that make that shift will not be chasing novelty. They will be building a security practice capable of testing the systems that now run real work.

The lasting measure of success is not how many AI tools a security team adopts. It is whether the organization can explain and control the authority those tools exercise. That standard keeps the work grounded as technology changes. Proof, not novelty, remains the standard.

That focus also keeps reporting useful. Executives need to know which decisions reduce exposure, and engineers need to know which boundary to change. Both audiences benefit from verified clarity.

It also makes retesting after changes more practical.

It keeps the work connected to actual architecture, actual people, and actual decisions instead of abstract claims about the technology.

It is a standard that remains useful after the next model release.

Questions security leaders ask about AI pentesting

Author:
Jan Bielik
CEO & Founder of Webiano Digital & Marketing Agency

This article is an original analysis supported by the sources cited below

NIST SP 800-115 Technical Guide to Information Security Testing and Assessment
NIST guidance on planning, conducting, analyzing, and mitigating technical security tests.

NIST penetration testing glossary entry
NIST definition of penetration testing as active attempts to compromise a system, device, or process.

OWASP Web Security Testing Guide
OWASP framework for web application and web-service security testing across the lifecycle.

OWASP Top 10 for LLM Applications 2025
OWASP guidance on major risks and mitigations for LLM and generative-AI applications.

MITRE ATLAS
MITRE knowledge base of adversarial tactics and techniques affecting AI-enabled systems.

SAFE-AI A Framework for Securing AI-Enabled Systems
MITRE framework connecting adversarial-AI threats, defensive thinking, and secure AI system design.

NIST AI Risk Management Framework 1.0
NIST framework for managing risks associated with AI systems.

NIST Generative AI Profile
NIST companion profile addressing generative-AI risks, including information security and component integration.

NIST AI RMF Playbook
Suggested actions supporting the Govern, Map, Measure, and Manage functions of the NIST AI RMF.

NCSC Guidelines for secure AI system development
UK National Cyber Security Centre guidance covering secure AI design, development, deployment, and operation.

NCSC Principles for the security of machine learning
Pragmatic principles for addressing machine-learning vulnerabilities across generic workflows.

NCSC Impact of AI on cyber threat from now to 2027
UK assessment of expected AI-related changes to cyber intrusion through 2027.

NCSC The near-term impact of AI on the cyber threat
Earlier NCSC assessment of AI-enabled cyber threat, including data and scaling constraints.

NCSC secure development and deployment guidance
NCSC material on secure delivery practices such as version control, peer review, and automated testing.

CISA Artificial Intelligence
CISA overview stating that AI systems require secure-by-design security across their lifecycle.

CISA guide on securing AI data
CISA notice describing the role of data security in AI accuracy, integrity, and trustworthiness.

CISA Careful Adoption of Agentic AI Services
Joint guidance on cybersecurity challenges and practices for the secure adoption of agentic AI services.

ENISA Artificial Intelligence Cybersecurity Challenges
European Union Agency for Cybersecurity analysis of the AI cybersecurity ecosystem, lifecycle, and supply-chain risks.

ISO/IEC 42001 AI management systems
International Standard for establishing and improving an AI management system.

ISO/IEC 23894 AI risk management guidance
Guidance for managing risks related to AI products, systems, and services.

Regulation (EU) 2024/1689 Artificial Intelligence Act
Official EU regulation text, including requirements on accuracy, robustness, cybersecurity, documentation, and oversight for high-risk AI systems.

CISA Secure by Design
CISA framework encouraging manufacturers to take ownership of security outcomes throughout product development.