The AI security question leaders should be asking instead
Security leaders are asking the wrong first question about artificial intelligence. The most common question is what data a model was trained on. That concern is legitimate: organizations want to protect intellectual property and prevent confidential information from becoming part of a model's general knowledge. But training data provenance is only one layer of a much larger security problem. The better question is how an organization maintains authority and integrity over the data an AI system can access, use, and act upon.
That distinction matters because AI systems are no longer just generating text. They are being connected to enterprise systems, given credentials, and trusted to take actions. The risk is not only that a model may have seen sensitive data during training. The risk is that an AI agent may have permission to retrieve, modify, or move data that it should never have been able to touch. Security leaders who focus only on training data leave a gap in access control, identity, and authorization.
Key facts
- The common question about AI training data addresses confidentiality but not authority or integrity.
- Least privilege, authentication, access controls, session-based role-based access control, and scoped permissions remain foundational for AI agents.
- AI is narrowing the skill gap between attackers and defenders, increasing vulnerability volume and reducing the advantage of specialized expertise.
- One security team handled roughly nine times more vulnerabilities with the same staffing and reduced time to remediation by just over 5% using AI tools and human oversight.
- Hiring is shifting toward experienced operators who can recognize confident but wrong AI output and correct it.
- AI-generated inputs raise concerns about model collapse and the need for human-curated data diversity.
- Application security teams are using AI earlier in the software development lifecycle to evaluate policies and surface issues in real time.
The question everyone asks is incomplete
When security leaders compare notes, the training data question dominates the conversation. It is easy to understand why. If a company's source code, customer records, or internal strategy documents become part of a model's training corpus, the confidentiality damage may be difficult to reverse. Legal, compliance, and procurement teams have therefore made training data a central issue in AI reviews. Vendors are asked whether customer data is used for model improvement, whether prompts are retained, and whether a model can reproduce sensitive information.
Those are valid questions, but they do not answer the operational security question. A model can be trained on clean, public data and still be dangerous if it is given broad access to internal systems. Conversely, a model trained on sensitive data can be deployed in a tightly controlled environment with strict permissions and human review. The difference between those scenarios is not the training corpus. It is the authority and integrity of the data at the point of use.
Authority means that an AI agent should only be able to perform actions that a human or system has explicitly authorized. Integrity means that the data the agent reads and writes should remain accurate, complete, and protected from unauthorized modification. Both depend on identity, authentication, authorization, and auditing. Those controls have been the bedrock of enterprise security for decades, and they are just as relevant for AI agents as they are for human users and traditional applications.
Least privilege still anchors AI security
Least privilege is the principle that a user, service, or agent should have only the minimum permissions needed to complete a task. It is not a new idea, but AI agents make it more important. An AI agent that can read every document in a repository, send email on behalf of an executive, or call any internal API is a high-value target. If the agent is compromised, misconfigured, or manipulated through a prompt injection, the blast radius can be enormous.
Session-based role-based access control delivered through standards such as OIDC and on-behalf-of flows can limit what an AI agent can do in a given context. Appropriate permission scoping ensures that an agent acting for a user does not exceed the user's own rights. If a user cannot approve a payment above a certain threshold, the agent acting for that user should not be able to approve it either. If a user can only view records for one region, the agent should not be able to query records for another region.
These controls also support human-in-the-loop architectures. As agent-to-agent communication becomes more common, the risk of a chain of automated actions producing an unintended outcome increases. Strict authorization boundaries create checkpoints where a human operator can review an action before it proceeds. That does not mean every action needs manual approval. It means high-impact actions should have clear gates, logging, and the ability to stop an automated workflow before damage occurs.
The training data question is about what a model might know. The least privilege question is about what a model can do. For security leaders, the second question is often more actionable because it can be enforced through existing identity and access management systems. It can be tested, monitored, and audited. It can be tied to business roles and data classifications. It can be improved without retraining the model.
The attacker toolkit is faster and less specialized
The attacker toolkit has changed in two ways. The first is speed. Vulnerability discovery, exploit development, and deployment are happening faster than before. Even well-resourced adversaries are racing against clocks to find and exploit flaws before defenders can patch them. The volume of vulnerability reports has risen sharply, and major vulnerability databases have struggled to classify every submission. In some cases, reported vulnerabilities remain unclassified because the volume exceeds the capacity of the organizations that maintain those databases.
The second change is more significant: AI is eroding the asymmetry of ability that the security industry has historically relied upon. Finding the conditions that allow an exploit, designing a working exploit, and deploying it typically required deep expertise. That expertise was a barrier. AI has moved that barrier closer to the ground. People who lack traditional exploit development skills can now find vulnerable contexts more quickly, devise methods to leverage them, and implement attacks that would previously have required a specialist.
This does not mean every attacker is now an expert. It means the minimum skill level required to cause harm has fallen. At the same time, the speed of attack has increased. The combination is dangerous because it compresses the defender's response window. A vulnerability that might once have remained obscure for weeks can now be identified and exploited in a much shorter period. The defender's advantage is no longer just having more skill. It is having better visibility, faster triage, and stronger controls that limit what a successful attack can achieve.
As AI systems receive more contextual data and become better at recognizing patterns, the attacker's ability to find exploitable conditions will likely improve further. External factors such as government controls may slow some actors, but the underlying trend is toward greater symmetry between attackers and defenders. In that environment, the size and readiness of the defensive team matters. So does the architecture that determines how much damage a single compromised account or agent can cause.
Understaffed teams and the throughput paradox
Security teams are famously understaffed, and AI has not simply solved that problem. It has changed the shape of the gap. One security team reported processing just over nine times the number of vulnerabilities in a year compared with the previous year while maintaining the same staffing levels. That increase would have been impossible to handle through manual effort alone. The team used large language models and supporting tooling to triage, prioritize, and respond to findings. It also reduced time to remediation by just over 5%.
That result is impressive, but it is not automatic. Achieving that throughput required experienced engineers to design the system, integrate the tools, and keep the process current as the models evolved. The team had to react quickly to a large increase in volume. It had to decide which tasks could be delegated to AI, which required human review, and how to measure whether the output was reliable. The tools did not replace judgment. They amplified it.
This is the throughput paradox of AI in security. The same technology that helps defenders process more work also helps attackers generate more attacks. The volume of findings increases, the speed of attacks increases, and the need for experienced operators increases. Headcount alone is not the answer, but neither is unchecked automation. The answer is a system that combines AI-assisted triage with clear escalation paths, strong access controls, and human review for high-impact decisions.
That system must also be monitored. AI tools can produce confident but wrong output. They can misclassify a vulnerability, miss context, or recommend an action that looks reasonable but is unsafe. The security team needs to know where the model is being used, what data it is seeing, and how its output is validated. Without that oversight, the throughput gains can create new risks.
Hiring for judgment, not just tool use
AI is changing what security leaders look for when they hire. Current AI systems are capable of producing work that is on par with many entry-level analysts and engineers. That raises the bar for all talent. The most valuable skill is no longer the ability to produce a first draft or complete a routine task. It is the ability to recognize when AI output is confidently wrong and to correct it.
That requires a broad range of knowledge and experience across diverse technical systems, as well as an understanding of the compliance and governance requirements that drive many security programs. An experienced operator can look at an AI-generated finding and ask whether it makes sense in context. They can spot missing assumptions, unsafe recommendations, or gaps in evidence. They can also design the guardrails and review processes that make AI useful without making it reckless.
This creates a gap for new talent. If entry-level work is increasingly handled by AI, how do junior analysts and engineers gain the experience they need to operate AI systems effectively? They need supervised practice, exposure to real incidents, and mentorship from senior staff. They need to see how decisions are made when the answer is not clear. Without that pipeline, organizations may save time in the short term but lose the experienced operators they will need in the future.
The data diversity problem behind model collapse
There is another concern that sits underneath the hiring debate: the quality and diversity of data entering AI systems. If the majority of inputs into AI systems are themselves generated by AI systems, there is a risk of model collapse. The models may become less diverse, less accurate, and less able to handle novel situations. The analogy to genetic diversity is useful. A population needs variation to remain healthy. Large language models need input diversity that, today, still depends on human curation.
Organizations often treat the labor problem and the data problem as separate issues. One is about staffing. The other is about training data. But historically, labor was how fresh inputs were provided to systems. People wrote documents, created code, recorded incidents, and documented decisions. If those human inputs are reduced at the same time that AI-generated content increases, the data ecosystem may become less rich. That is a risk for model performance and for security, because security depends on understanding unusual and emerging patterns.
For security leaders, this means that human review is not only a safety control. It is also a source of data diversity. Experienced operators can identify edge cases, correct errors, and add context that a model may not have seen. Their judgment helps keep the system grounded in reality. That is one reason why hiring for judgment matters even as AI tools become more capable.
Where AI is genuinely helping
The most promising impact is at the application level. AI is allowing security concerns to move earlier in the software development lifecycle. Instead of waiting for a bottlenecked review period, teams can have AI evaluate code and design against a curated list of criteria and policies. The AI can surface concerns and provide guidance in real time as developers work. That shortens the feedback loop and reduces the need for architecture and design requirements to filter back down the pipeline after development is complete.
This does not remove the need for specialists. It changes when specialists are involved. Developers can propose and implement solutions with review from security experts, often avoiding rework later. The security team can focus on policy, high-risk decisions, and complex architecture rather than routine checks. Basic tasks that once depended on external development teams can become internal side projects with explicit review, changing the cadence of security work.
Source:Help Net Security News