CXC Insight: An Unprecedented Cyber Incident and What CI Operators Need to Know
OpenAI and Hugging Face have launched a joint investigation into an incident they are referring to as unprecedented and novel. The details of the incident can quicky become overwhelming to the reader, but at a basic level two AI models operating in a sandbox testing environment were tasked to pursue advanced exploitation objectives. For the purposes of the test the two models operated without significant guardrails. The models did as requested, with one of them breaking out of the sandbox, moving laterally, escalating privileges to gain internet access, and then targeting Hugging Face – as a likely holder of critical data required for it to meet its test objectives. Importantly, when Hugging Face attempted to respond using their own AI defense systems, the systems refused, analyzing the request to be nefarious – and showing that safeguards do not predictably differentiate between offensive actions and defender requests for information. For additional details and considerations on the attack, check out Anna Corsaro’s Homeland Security Today article.
Critical infrastructure owners and operators need to be watching and paying attention to this investigation. This unprecedented incident highlights:
1. How autonomous AI changes the threat model – this incident shows us just how capable LLMs can be, chaining together multiple vulnerabilities, escaping a testing environment, moving laterally, escalating privileges, and targeting an external organization.
2. The velocity of AI enabled attack – in this case, the AI model executed over 17,000 actions over a weekend, elevating the case for critical infrastructure to maintain continuous monitoring, automated detection, identity protections, behavioral analytics, and isolation capabilities.
3. OT Systems continue to be the most vulnerable – often operating on legacy systems and designed to be on flat networks, organizations need to stop asking whether there is an exploitable vulnerability and start to think through the sequence of vulnerabilities that an AI agent could discover and chain together to exploit a targeted system.
4. Critical infrastructure and asset owners need to consider using defensive AI to oppose adversarial actions - at the same time we must ensure a fuller understanding of LLM capabilities for risk characterization and response.
5. AI is part of the supply chain – critical infrastructure owners and operators should be developing AI vendor risk management programs and posing questions to service providers and business partners: for example, asking what AI models are embedded in their projects, how they are trained, what testing is done, how updates are governed, and what happens when AI behaves unexpectedly.
6. Containment matters – in this case, the flawed containment of testing sandboxes made all the difference – understanding which and how environments are isolated, how networks are segmented, and closely monitoring access control can significantly reduce risk.
7. The importance of transparency and collaboration – Hugging Face has been very transparent about this situation, supporting a greater understanding of the threat landscape, best practices, and failures that led to the incident. Collaboration among stakeholders and the sharing of information is critical in moving forward with AI frameworks, security programs, and response plans.