Securing AI agents in operation: rules, permissions, logs
How companies keep hidden instructions from redirecting an agent, with an example from incoming invoices.
Home Research
Research for secure AI
We know how AI stays secure in operation.
Talk to Andreas Kamenz, our Chief Scientist, about joint research projects and calls for proposals.
Our research areas
Quick answers
How do we secure AI across its entire lifecycle?
We secure AI in every phase of the software lifecycle: in development, in operation and wherever systems keep learning while in operation. AI agents get clear rules, only the permissions they need and a log of every action. We test learning systems before every new model version. Code that AI helps write goes through the same checks as any other code.
Discuss a research partnership
Andreas Kamenz leads our research. Write to him about what you have in mind: a joint research project, a request for proposals or a call for funding proposals.
What happens next
- Andreas will get in touch and suggest a meeting.
- In the first conversation, we clarify the topic, the partners and the path to funding.
AI agents in operation
What security risks do AI agents create?
AI agents complete tasks in several steps and use your systems along the way, such as email, ERP or document storage. Their model stays the same in operation, but their behavior depends on the texts they are reading at that moment. This is where prompt injection comes in: attackers hide instructions in an email, a document or a web page, and the agent then takes them for an assignment. The security organization OWASP ranks prompt injection first in its 2025 and 2026 lists of risks for applications built on large language models.
Source: OWASP, LLM01:2025 Prompt InjectionSource: OWASP, Top 10 for LLM Applications 2026
Our approach
- Define rules and responsibilities (governance) for each agent: which task, which data, which limits.
- Limit permissions to what is necessary; your experts only sign off where their judgment matters.
- Log every action and test the agent with targeted attacks before launch and regularly afterward.
Learning AI systems
How do we stop learning AI systems from learning the wrong things?
Some AI systems adjust their parameters in operation, that is, the numerical values the model has learned. Experts speak of online or continual learning when a system keeps learning from new data. If it adapts its behavior to feedback such as ratings or clicks, this is called online RL (reinforcement learning in operation). Such systems can learn the wrong things from manipulated data or amplify their own errors. For high-risk systems that keep learning, the EU AI Act requires measures against such feedback loops (Article 15(4)).
Our approach
- Check where data comes from and how good it is before the system learns from it.
- Define in advance what may change in operation. Test every new model version against fixed test cases.
- Keep earlier versions, monitor deviations and roll back to the previous version if anything looks wrong.
AI in software development
How secure is code that AI helps write?
Code suggested by AI tools regularly contains security flaws. In a test by the code security vendor Veracode with more than 100 language models and 80 programming tasks, the code the models produced contained a vulnerability from the OWASP Top 10 for web applications in 45 percent of cases (2025). Germany's Federal Office for Information Security (BSI) and the French cybersecurity agency ANSSI therefore recommend that developers always review and understand AI-written code (2024).
Source: Veracode, 2025 GenAI Code Security Report (press release of July 30, 2025)Source: ANSSI and BSI, AI Coding Assistants (October 4, 2024)
Our approach
- Have every change reviewed by a person who understands the code. Run automated tests and security scans on top.
- Check new libraries against a list of approved packages, because AI tools also suggest packages that do not exist.
- Maintain a software bill of materials (SBOM), so that when a new vulnerability appears, it is immediately clear which components are affected.
How our research feeds into client projects
When AKARA integrates AI into your workflows and systems, we apply the results of this research in every project. Every AI agent we connect to ERP, merchandise management or line-of-business systems gets only the permissions its task requires. Your experts only sign off where their judgment matters. Every action the agent takes can be traced in the log. Before an agent goes live, we test it with the attacks we study in our research. Experienced developers and automated security tests review the code we deliver, including where AI helped write it.
Learn more: Insights: AI agents: what they can do and where their limits areSeminar (in German): AI compliance and security (EU AI Act)Service: Compliance and security
Research partner
We conduct research together with HITeC e.V., the technology transfer institute at the Department of Informatics of the University of Hamburg.
Publications
Securing learning AI systems
What changes when a model keeps learning in operation: data poisoning, faulty learning and the right controls.
AI in software development: how code that AI helps write becomes secure
What studies show about the quality of AI-written code and which review methods development teams use.
How secure are your AI agents?
We answer your questions about securing your AI systems in a personal call.