When an AI Cyber Test Reached Real Companies

When the Test Reached Real Targets

A cyber-capable Gemini model was meant to attack fictional targets. With unintended internet access, it reached three real companies instead.

That is the development worth sitting with, because the model did not need a science-fiction escape plan. During a third-party cybersecurity evaluation in May, it apparently used access that should not have been available, found public information and guessed credentials for websites it believed were inside the test. Google confirmed the incident on September eighteenth.

Google security executive Heather Adkins said systems belonging to three companies were accessed. The organisations were informed, and Google says the evaluator changed its testing processes. We do not know which Gemini model was involved, which systems it reached, how long the access lasted or whether it caused damage. Google has not published a full incident report. Those gaps matter, so this needs to be described accurately: it was an evaluation-containment failure, not a malicious AI deployment.

Even with that qualification, the failure is serious. A simulated target only remains simulated when the surrounding environment enforces the boundary. If an agent can reach the public internet, find a real service, try credentials and continue acting, a mistaken belief about scope can become an intrusion. The model does not need malicious intent. It only needs enough capability, an ambiguous target and a harness that lets its actions travel too far.

Public information and guessed credentials also make this a useful warning about ordinary attack paths. Nothing in Google’s limited account suggests the model discovered an exotic new vulnerability. It appears to have combined information it could find with access attempts it was capable of making. That is precisely why containment cannot depend on whether an evaluator expects the model to succeed. A test environment has to remain safe when the model finds an approach the people running it did not anticipate.

Security evaluators already work with tools that can scan systems, run commands and test credentials. Adding a model gives those tools more freedom to decide what to try next. That makes the controls outside the model decisive: network isolation, explicit destination allowlists, credentials created only for the test, and machine-enforced limits that do not depend on the agent correctly interpreting a written instruction. A prompt saying a target is fictional is context. A blocked network route is a control.

There is also an uncomfortable lesson here for ordinary enterprise pilots. An organisation may spend a great deal of time evaluating whether a model refuses a dangerous request, then give the resulting agent broad network access and reusable credentials. The safety case cannot stop at model behaviour. It has to cover the complete path from an instruction, through tools and identity, to the system that receives the action.

My assessment is that cyber-agent evaluations now need to be treated much more like live offensive-security exercises. Assume the agent can misunderstand the boundary, make the boundary technical, and log enough detail to reconstruct what happened. Until Google publishes more, we cannot judge the impact of this particular incident. We can judge the control failure: three real organisations ended up inside a test that was supposed to stay fictional.

Qwen Builds Across Every Medium

Keep that gap between capability and control in mind as we shift to a much broader kind of agent.

Alibaba’s Qwen team has released Qwen three point eight Omni Flash through its hosted platform and APIs. It accepts text, images, audio and video, and it is designed to use tools as part of agent-style workflows. The headline technical number is a context window of up to one million tokens. That creates room for very large inputs, including long audiovisual material, without a developer first reducing everything to a short text transcript.

The attraction is architectural. A media workflow often chains together speech recognition, a vision model, a language model and a separate tool-running layer. Every handoff adds code, latency and another place where meaning can be lost. A single model that can interpret the media and decide which tool to call could make that stack smaller. It could also keep relationships between speech, images and events that disappear when each stream is processed in isolation.

Qwen says the model improved by more than twenty-six per cent on average across thirty evaluations compared with Qwen three point five Omni Plus. That is a company result, not independent proof of real-world performance. An average across many tests can also hide the one weakness that breaks a particular product. A video-support agent may care more about precise timing and tool reliability than a broad benchmark score. A research workflow may find that the full context window is possible but too slow or expensive for routine use.

The useful developer takeaway is that unified multimodal agents are becoming a practical hosted option, not merely a collection of research demos. They may remove several integration steps for products that work with meetings, recordings or mixed-media archives. I would still test the complete job on representative material: what the model misses, how reliably it invokes tools, how latency changes with long inputs, and what the hosted bill looks like. The simpler architecture is the benefit. The vendor’s aggregate benchmark is not the deployment decision.

Claude’s Vetted Biology Access

A different access question appears in science, where useful work can resemble the requests a general model is trained to block.

Anthropic opened applications on September seventeenth for a beta Life Sciences Verification Program. Vetted teams can use Mythos, Opus and Sonnet models with biology safeguards adjusted for professional research. Standard grants cover verified organisations. Higher-risk grants are approved for a particular project, remove blocks on life-science requests and come up for renewal every six months.

The program changes where the control sits. Anthropic is replacing some real-time biology blocking with offline monitoring for approved users, and it requires thirty-day retention of program traffic so that monitoring can happen. Access depends on verified identity, the institution and, for higher-risk work, the stated project. That can let a legitimate laboratory ask questions that a public chatbot would refuse, while giving Anthropic a way to review how the access is used.

The trade is quite direct. Researchers gain capability, but they accept vetting, monitoring and a new data-governance obligation. The beta is not available on individual plans or through third-party platforms. Organisations using a business associate agreement for protected health information are also excluded, so this is not a route for every biomedical workload. Any laboratory considering it has to separate eligible research data from material that cannot enter the program under those terms.

Anthropic has not yet shown how effective this combination will be at scale, or how far access may expand after the beta. Offline monitoring can identify misuse after a request has run; it is not the same as stopping that request in real time. Institutional accountability therefore carries more weight.

My read is that trusted-access programs are a sensible attempt to stop broad safeguards from blocking serious science, but they do not make the risk vanish. They move part of it into identity checks, audit data and the institution’s own oversight. For qualified laboratories, more model capability becomes available. The price is a much clearer record of who is using it, for what purpose, and under which data rules.

Plugin4Shell Breaks the Pin

The next weakness sits closer to a developer’s desk: the plugin that looks pinned may not be the code that runs.

Air Security disclosed a supply-chain flaw it calls Plugin4Shell across Claude Code, Codex, GitHub Copilot and the now-deprecated Gemini CLI. The affected clients could accept a pinned Git reference without verifying the final commit checked out on disk. If an attacker controlled, or later took over, the repository behind an installed plugin, that gap could let attacker-controlled code replace the version the user thought they had selected.

The zero-click part comes from background updates. A developer may have approved the plugin when it was first installed, yet a later replacement could execute without another approval. Pinning normally gives people confidence that an update cannot silently move them to different code. Here, the label could stay stable while resolution landed somewhere else. Provenance only helps when the client checks the exact resolved commit before trusting what arrived.

There are important limits. One demonstrated route relies on a Git host accepting branch names that resemble commit identifiers; GitHub rejects those names. The researchers reported no exploitation in the wild and no CVE. They also report fixes in Claude Code version two point one point one seventy-nine and Codex version zero point one forty-six point zero. They did not identify corresponding fixes for GitHub Copilot or the deprecated Gemini CLI, and some patch information comes from the researchers because not every vendor had published its own advisory.

For developers and engineering organisations, the immediate judgement is practical. Check the client version, then review whether automatic plugin updates are acceptable on any agent that remains unpatched. Removing an unused third-party plugin is a smaller cost than relying on a pin the client does not truly enforce. Repository ownership, signing and a familiar vendor name still do not answer the final question: did the exact commit approved by the user become the code executed by the agent?

Coding agents are unusually sensitive to this class of failure because plugins often inherit access to source code, terminals and developer credentials. The repair belongs in the client. It has to verify the resolved commit, fail closed when it differs, and require a fresh decision before new code crosses that trust boundary.

A Credential Leaks After Decryption

Even a sound identity vault has a vulnerable moment: the agent eventually needs the usable credential in memory.

Palo Alto Networks’ Unit forty-two demonstrated an indirect prompt-injection attack against an AWS AgentCore Harness agent. The agent processed untrusted content containing the injected instructions. With the default-enabled shell tool available, it read a plaintext credential from its own process memory and sent that credential out. The secret had been resolved through AgentCore Identity, but the harness and the usable token occupied the same runtime space.

The researchers then replayed the captured token from a separate laptop. They accessed data through the downstream MCP server without needing AWS credentials of their own. That sequence is the key detail. The identity store did not have to be cryptographically broken. Once the authorised agent decrypted the credential for use, an overpowered tool could inspect the process holding it and turn a temporary runtime secret into a portable bearer token.

AWS treated the disclosure as a customer-configuration issue. It did not assign a CVE or announce a platform patch. There is no reported production exploitation, and this was a controlled researcher demonstration. Even so, the mechanism is relevant to any agent that reads untrusted text while holding credentials and offering a shell, debugger or similarly broad local tool. Prompt injection supplies the instruction; excessive tool access and token privilege supply the consequence.

For AgentCore operators, encryption at rest is only one layer. The stronger design gives the agent the narrowest tool set it needs, scopes the downstream credential to the smallest useful permission set, keeps its life short and restricts outbound connections. Separating high-risk content processing from the process that holds a valuable token can reduce what one successful injection exposes.

The organisational takeaway is that a managed identity service does not automatically make an agent’s use of that identity safe. Security reviewers need to follow the credential all the way into runtime memory and ask what the model’s tools can inspect or transmit at that moment. In this demonstration, the vault continued doing its job. The surrounding agent made the decrypted secret reachable, and that was enough.

NIST’s Token Security Baseline

That runtime attack has a useful policy counterpart, because NIST has now finalised guidance for the tokens agents increasingly depend on.

The US National Institute of Standards and Technology published the final version of IR eight five eight seven on September fifteenth. It covers identity tokens, access tokens and assertions used across cloud systems, with recommendations for key management, token verification, lifecycle controls, continuous monitoring, single sign-on, federation and API access. The intended readers are US federal agencies and cloud service providers.

For Australian organisations, this is implementation guidance, not a new local legal requirement. Its value is as a current technical reference. Agent systems increasingly act through delegated identity: a service issues a token, an agent presents it to an API, and the receiving system trusts the permissions encoded or associated with that token. If the token is stolen or accepted without proper verification, a well-behaved model does not save the downstream service. The attacker can use the delegated access directly.

A review based on the guidance can trace how a token is issued, which audience can accept it, how narrowly it is scoped, where it can travel, how quickly it expires, what monitoring detects unusual use and how revocation takes effect. That connects neatly to the AgentCore demonstration. Short life and narrow scope do not prevent memory extraction, but they limit how useful the captured credential remains. Strong verification also reduces the chance that a forged or misdirected token is accepted somewhere it was never meant to work.

My practical assessment is that delegated identities now belong in the core architecture review for any agent with tools. Model safeguards and prompt filters can reduce unsafe decisions. Token controls limit the damage when those safeguards fail, or when the model is bypassed completely. NIST has given security and platform teams a shared baseline for that less glamorous part of the system, and it is often the part carrying the actual authority.

Memory Moves Into the AI Bottleneck

Capability also runs into a physical limit, and for long-context agents that limit is increasingly the movement of memory.

SK hynix used the AI Infra Summit on September seventeenth to demonstrate three approaches: high-bandwidth flash, processing-in-memory hardware and software called SALT-KV that tiers a model’s key-value cache. The key-value cache stores intermediate attention data so a model does not have to recompute the whole conversation for every new token. As context grows, that cache can become large, expensive and difficult to keep in the fastest memory.

The company describes high-bandwidth flash as a capacity layer between high-bandwidth memory and ordinary solid-state storage. It uses vertically connected NAND to increase bandwidth. Processing-in-memory takes a different route by doing some work closer to where data sits. SALT-KV handles the problem in software: it assigns parts of the cache to high-bandwidth memory, conventional DRAM or SSD storage according to estimated reuse value and storage cost. Frequently useful data can stay close to the accelerator, while colder data moves to a cheaper tier.

That is a plausible direction for long-running assistants and agents. Faster accelerators do not solve every inference cost when the system spends time moving a huge cache or keeping it in scarce premium memory. Smarter tiers could support longer sessions or more simultaneous users without putting every byte in the most expensive place.

There is no shipping date, standard production specification or independent performance measurement for these demonstrations. The engineering trade-offs remain open: moving data between tiers can add latency, reuse estimates can be wrong, and another memory layer adds operational complexity.

For infrastructure buyers, the useful conclusion is to watch memory capacity and cache movement as closely as accelerator throughput. These designs may eventually lower the cost of persistent agents, but a summit prototype is not yet evidence for a purchasing decision. The economics become convincing only when shipping systems show that the cheaper capacity saves more than the extra tiering costs.

What Changes for You

For a change you can use immediately, ChatGPT now sits inside the document many people already have open all day.

OpenAI released ChatGPT for Word as a Microsoft add-in on September seventeenth, and it is available across every ChatGPT plan, including Free. The sidebar can work with the open document to draft, summarise, revise selected text and make basic changes to structure or formatting. That removes the familiar copy-and-paste loop between Word and a browser.

For an individual, the practical difference is speed. You can select an awkward paragraph, ask for a tighter version and compare the result without losing your place. For working AI builders and organisations, it also shows how quickly the model is moving into the application where the final artefact lives, rather than waiting in a separate chat window.

There are boundaries. The add-in cannot directly pull context from other local files. Complex layouts may need manual repair, usage remains subject to the limits of the user’s ChatGPT plan, and organisational access depends on both ChatGPT and Microsoft three sixty-five administrators allowing the add-in. Important facts and edits still need checking, especially when the document is going to customers or the public.

My take is that this is genuinely useful because it shortens a common editing loop, not because it changes what the model can reason about. The same convenience can make an unchecked rewrite feel finished simply because it already sits in the publishable file. The gain is less friction. The limitation is that review has to survive that friction disappearing.