Last week, Amazon used an agentic LLM tool to successfully troubleshoot an issue in a production environment faster than a team of human engineers. Just kidding, it actually deleted the whole environment and caused a major outage in China. To be fair to the LLM, it probably did do this faster than a team of human engineers could have.
On a similar note, Facebook’s “AI alignment director” (which is ostensibly a role involving making LLMs safer) had her agentic LLM try to delete her email inbox; according to the article:
Yue had previously tried OpenClaw on her “toy inbox,” where she wrote that the bot worked well and gained her trust. Testing it on her “real inbox,” the bot had to compact a much larger set of emails. She instructed it not to take action without approval, but OpenClaw lost the prompt during compaction, she wrote.
I’m not usually a pessimist, but here I feel justified in saying that it’s only going to get worse. Both inside and outside of the tech industry, people who should know better are encouraging the expansion of LLMs into production, mission-critical systems without nearly enough compensating controls. This is likely due to two factors:
- The flaws in LLM outputs are not always immediately apparent. This creates a short-term bias when LLMs appear to output miraculous amounts of high-quality, functional code, on-par with human outputs. This results in a false confidence in LLM coding abilities that leads decision-makers to approve the use of unmitigated, unsupervised agentic LLM operations (such as writing entire codebases from scratch or managing a production cloud environment).
It takes time - sometimes months - for the glaring issues with these outputs to make themselves apparent (either through outages, security vulnerabilities, or bugs that would have been obvious to/avoided by a human programmer). Decision-makers are less likely to truly connect the actual cause (bad LLM outputs) to the subsequent, time-delayed effect, resulting in other factors being blamed instead. Even when the time between cause-and-effect is minimal, the profound awe experienced by decision-makers seeing apparently-perfect code built by LLMs can create a bias toward LLM outputs - leading to these decision-makers blaming external factors for errors in LLM outputs.
- LLMs are making humans dumber and more reliant on LLMs. There are no shortages of studies demonstrating this effect in various ways - like this one from MIT researchers showing both cognitive and neurological deficiencies in LLM users, or this one from the Swiss Business School asserting that younger, LLM-dependent participants displayed reduced critical thinking compared to other groups, or this one from Anthropic, the company behind the Claude series of LLMs, showing that programmers using LLMs not only did not experience efficency gains over non-LLM users, but also that they had impaired skill formation compared to non-LLM users.
Most people experiencing this cognitive decline will take the easy way out: increasing their dependence on LLMs as opposed to curtailing their use. Human-vetted outputs will steadily decline in prominence as they are supplanted by unvetted “AI slop” which is understood neither by the LLM (which cannot think or reason in any capacity) nor by the human being who commissioned it.
How Will This Impact Security Operations?
Negatively. Across multiple domains, security responders will see the degradation of norms which have prevented some security incidents and an increase in obvious security issues within their organizations. For example:
Coding
Your organization has probably only started taking secure coding practices seriously within the last 5-10 years. You may have only recently added secure coding training to engineers’ yearly regimen, or just onboarded a code-scanning tool to catch the obvious mistakes before they cascade into incidents.
This steady improvement is most likely going to nosedive if your organization does not properly restrict the use of LLMs for production codebases. You will see an increase in bad code generated by LLMs in these codebases, leading to a resurgence of security vulnerabilities which secure coding practices had previously reduced.
Decision-makers can try to fight fire with fire by adding LLM code security review to the stack, but given everything we know about trusting LLM outputs in production, this is not only unlikely to solve the problem (given it was caused by LLM usage to begin with) but will actively make the problem worse by giving decision-makers a false sense of confidence in codebases cleared-for-production by LLMs.
Inevitably, either through a bug bounty (you hope) or through active exploitation by threat actors (more realistic), LLM-induced vulnerabilities will be exploited and you will be called upon to perform incident response. You are likely used to this from human-introduced vulnerabilities, but the addition of LLM-introduced vulnerabilities will make this a more common occurrence. If the human who commissioned this vulnerable code is suffering from “AI”-assisted cognitive decline (as discussed previously), then you may be asked to identify the vulnerable code. In healthy, mature organizations, this would not be your responsibility - it would instead be the job of the person that wrote the code, maybe in conjunction with an Application Security team. But in the advanced stages of LLM-driven cognitive decline, the person who commissioned the code might actually have no idea what it does, how it works, or what part of it could have created the vulnerability. This will make incident response slower, which (in turn) will increase the cost of these security incidents.
Systems Engineering
Decision-makers and engineers will increasingly design, implement, and maintain systems by having LLMs do it for them. This will increase the number of security flaws that are baked into systems from the start, increasing the odds that full architectural revamps will be necessary to truly remediate security vulnerabilities.
These same LLM users may believe the best way to remediate the security vulnerabilites that their LLMs introduced is by using more LLMs to fix the problem. That will, in turn, exacerbate the issue and eventually necessitate human intervention to close the loop. The humans in question will, more likely than not, be you and your team. This is both good and bad - good because you will be able to directly influence the design of these systems toward security, and bad because (in all likelihood) this is not meant to be your job.
Novel Attack Vectors
LLMs don’t just exacerbate existing security issues. They also create brand new ones that are specific to LLMs! Research regarding LLM poisoning is new and ongoing, but right now it appears that these attacks are fairly trivial; as new LLMs are brought to market, the likelihood that it will have been trained on poisoned data increases with time (as threat actors become increasingly aware of the proofs-of-concept regarding these attacks).
Poisoning attacks are, as best as I can tell, purely theoretical for now. Not-so-theoretical novel attack vectors include:
- End users uploading confidential info to public LLMs.
- Organization-deployed public LLM support chatbots making legally-binding statements.
- Prompt injection attacks against chatbots that lead to the disclosure of internal/confidential information.
- Prompt injection and RCE attacks against agentic LLM tools such as OpenClaw.
And Much More
This isn’t an exhaustive list because if guardrails for LLM tooling aren’t enforced, there will be no limit to their impact - they can (and will) touch every part of your organization.
What Can Security Responders Do About It?
- Increase your headcount. You’re going to need more people. What we are seeing now is the beginning of an avalance. As LLM usage continues to be encouraged and guardrails continue to be removed, the number of incidents you experience will increase - steadily if you’re lucky, or astronomically if you’re not. Get the ball rolling on hiring more responders for your team. The sooner you start, the more well-prepared you will be when the true scope of this issue is revealed within your organization.
- Apply internal pressure against unmitigated LLM usage. You probably already have specific examples of where LLM usage has (either actually or almost) caused a major security incident. Put together a case for restrictions and bring it to your decision-makers. If that falls on deaf ears, all the better for your case - you’ll get plenty more examples to include and redeliver to the deciders in your organization. The more noise you make about this, the more likely it is that you’ll get some restrictions approved or implemented - and subsequently, an increase in your organization’s security posture. It’s unfortunately true in some organizations that security issues are not considered until a major incident happens, and even then, only for a period of time. If that’s true for yours, build out a living document now. That way, you can make sure that everything you include is actually considered once people are ready to listen.
- Proactively detect and respond to insecure LLM usage. Deploy detections for unrestricted agentic LLM usage, regardless of whether or not it’s used to perform menial labor on inboxes or if it’s used to write entire codebases from scratch. Find and root out unauthorized LLM tooling that doesn’t have enterprise controls in place. Audit the inputs your end users are passing into public models. Then, add your findings to the aforementioned living document to keep the pressure on your decision-makers to choose long-term security over short-term convenience.
- Add LLM awareness to your security training. It sounds silly, but you need to explicitly tell people that they cannot blindly trust LLM outputs. Ensure that your training makes it clear that all LLM outputs must be fully audited, validated, and understood by a human being before going anywhere near production. Integrate this with your secure coding training so people understand that the code that both they and their LLMs produce needs to follow best practices. And of course, ensure that your end users know that they cannot upload confidential information to a public LLM.
- Enforce personal accountability for code, regardless of origin. If someone commissions insecure code from an LLM and deploys it to production, or uses an LLM-generated script to take down a production server, or otherwise causes actual harm to your organization with the help of their favorite model, then they should be held to the same standard as someone who does it without an LLM. LLMs are not people and are not employees; they cannot be held liable for causing catastrophic security incidents. That means you need to hold the prompters responsible instead. Mandate retraining, require additional (human) code review, and take any appropriate remediative action to push your organizational culture toward responsible LLM usage and away from unrestricted slop generation.
Markets may not punish outages and security incidents as heavily as they used to, but there are still severe regulatory and business consequences for this behavior. Our role as security responders is not only to mitigate incidents but to do everything we can to prevent them. You should explore ways of responding to incidents that will reduce future issues; make recommendations to decision-makers grounded in hard evidence regarding the security risks posed by LLM usage; and leverage the security risks posed by this problem to improve your response capabilities.