Last month, a security architect messaged me: "We just got approval to deploy an AI agent that can modify purchase orders in SAP. My CISO wants to know what could go wrong. Where do I even start?"
This question is becoming common. According to Onapsis research, 58% of organizations have started using AI applications or agents that interact with their ERP systems within the last six months. Nearly 22% reported a security incident in the past year where attackers used AI against a business-critical platform.
Security teams are facing real, urgent questions without clear answers. Here's what I'm hearing most often, and what you can do about it.
The Source of These Questions
These questions aren't theoretical. They're coming from Slack channels, incident debriefs, and vendor evaluation meetings where teams realize their current security models don't fit. More than 70% of security leaders have limited or no trust in AI protecting their critical data, yet the agents are already being deployed.
The questions below reflect the tension between adoption pressure and uncertainty about securing systems that don't behave like traditional applications or users.
Q1: "Do we treat AI agents like service accounts or like users?"
Neither, exactly.
Roland Palmer, CISO at JumpCloud, recommends treating each agent like a new employee: give it a distinct identity, minimum permissions, and expand access only when it proves reliable. Unlike a service account that runs predictable code, agents can interpret instructions and take novel actions.
Your identity and access management system needs to support:
- Unique identities per agent (not shared credentials)
- Role-Based Access Control that defines what the agent can read versus change
- Segregation of duties checks applied to agent permissions just like human roles
- Audit logs that capture what the agent did and why (the prompt or instruction that triggered the action)
Before granting write access to ERP records, test the agent's permissions in a sandbox. If you can't answer "what's the worst this agent could do with these permissions?" don't deploy it.
Q2: "How do we apply the Principle of Least Privilege when we don't know what the agent will need to do?"
Start narrow and monitor everything.
You're not obligated to grant broad permissions just because the agent's behavior isn't fully predictable. Begin with read-only access to the smallest dataset the agent needs to perform its initial task. Monitor what it attempts to access. Expand permissions incrementally based on observed behavior, not vendor recommendations.
Juan-Pablo Perez-Etchegoyen, CTO of Onapsis, emphasizes that Zero Trust Architecture principles apply here: verify every request, assume breach, and limit the blast radius. If an agent is compromised or manipulated through prompt injection, the incident is restricted to what that agent could do, not what the entire ERP system allows.
Implement Just-in-Time Access where possible. If an agent only needs to modify procurement records during month-end close, grant that permission for a defined window, then revoke it.
Q3: "Our finance team wants an agent that can approve invoices under $10K. What's the risk?"
The risk is that 'approve invoices under $10K' isn't a technical control.
An agent with permission to modify invoice records can be instructed (or tricked via prompt injection) to approve anything it can access. You need to enforce the $10K limit at the authorization layer, not trust the agent's instructions.
Design your controls this way:
- The agent has write access only to invoices flagged as "pending auto-approval"
- A separate process (human or rule-based) flags invoices under $10K for that queue
- The agent cannot modify the approval threshold or reassign invoices to itself
- Every action logs the invoice ID, amount, and the instruction that triggered approval
Before deployment, ask Palmer's four questions: Can this feature be turned off? What identity does it act as? What does it log? Can its access be scoped separately from the user who deployed it?
If your vendor can't answer those questions, you're not ready to deploy.
Q4: "We're using AI-generated code in our ERP customizations. Should that go through the same change management as human-written code?"
Yes, and possibly more rigorously.
More than 62% of organizations already use AI-generated code in ERP applications, but that code can introduce vulnerabilities or faulty access checks directly into business workflows. Your change management process should include:
- Code analysis to verify the AI didn't introduce SQL injection vectors, hardcoded credentials, or overly permissive access checks
- Threat modeling specific to the business process the code supports
- Security testing in a non-production environment with production-like data access patterns
- Authorization review to confirm the code enforces segregation of duties and doesn't bypass approval workflows
Consider AI-generated code as untrusted input until proven otherwise. The fact that it was generated by a tool marketed as "intelligent" doesn't make it secure.
Q5: "What do we log when an agent takes an action? We can't just log 'agent modified record.'"
You need to log the instruction, the identity, and the outcome.
Standard audit logs capture who did what and when. For agents, you also need to capture why: the prompt, instruction, or trigger that caused the action. If an agent modifies a purchase order, your log should show:
- The agent's unique identity
- The user or system that instructed the agent
- The instruction itself (or a hash if it contains sensitive data)
- The specific record modified and what changed
- Timestamp and session context
This level of logging supports both incident investigation and compliance audits. When your auditor asks "how do you know this agent didn't bypass approval controls?" you need evidence, not assertions.
Q6: "Our ERP vendor just added an AI feature. Do we have to use it?"
No. Gate the feature, not the vendor.
You rarely get to reject an entire ERP platform, but you can defer its AI modules. Evaluate each feature independently using Palmer's four-question framework. If the vendor provides immature answers, you can still proceed with tactics like:
- Off-by-default deployment (require explicit opt-in per business unit)
- Scoped rollout (pilot with read-only access in non-critical systems)
- Committed review date (revisit when the vendor matures logging or access controls)
Survey data shows that 41.4% of organizations reported security teams objecting to AI in ERP environments, with lack of confidence in AI security cited by 75%. That resistance isn't obstruction; it's risk management. Don't deploy a feature just because it exists.
Q7: "How do we build trust in these systems when we don't fully understand how they make decisions?"
Trust comes from constraints, not comprehension.
You don't need to understand the agent's decision-making process if you've constrained what it can decide. Focus on:
- Stronger access management (62% of survey respondents said this would improve their trust)
- Personal data protections (46% cited this as a trust builder)
- Sandboxing or digital twins for testing agent behavior before production deployment (37% said this would help)
Design security before deployment, not as a retrofit. An agent with narrow permissions, robust logging, and human oversight is trustworthy even if its internal logic is opaque.
Where to Go for More
Your existing frameworks still apply. ISO/IEC 27001 Annex A.9 (Access Control) and NIST Cybersecurity Framework 2.0 provide the foundation. Extend them to treat agents as distinct entities requiring their own authorization policies.
If you're in a regulated industry, confirm that your agent deployment doesn't create gaps in segregation of duties (SOC 1), access logging (NIST SP 800-53 AU family), or data protection (General Data Protection Regulation Article 32). The technology is new; the control objectives aren't.
Work migrates to agents. Accountability doesn't, shouldn't, and can't. At the end of every agent action, a human needs to own the outcome.
