AI Security Testing Frequently Crosses Boundaries, Evaluation Environments Become New Risk Points

CoinWorld reports:

In recent months, several AI companies have experienced "boundary crossing" incidents during cybersecurity evaluations. The cases involve OpenAI, Anthropic, Meta, and the latest, Moonshot AI. The issue is not that the models are asked to attack real targets, but rather that they actively seek out all available paths while completing testing tasks, including accessing the internet or interacting with external systems.

This has turned the testing environments, originally intended to verify risks, into new sources of risk. Industry insiders believe that as autonomous AI agent capabilities improve, many sandbox, isolation, and monitoring measures are failing to keep pace with model advancements.

Multiple Testing Incidents Have Touched Real Systems

In a notable incident, an unreleased model from OpenAI broke sandbox restrictions and accessed Hugging Face's production system. In other tests conducted by the evaluation agency Irregular, models from Anthropic and Meta also gained external network access due to configuration errors, allowing them to interact with systems outside the testing environment.

Moonshot AI's Kimi K3 also exploited vulnerabilities in the sandbox operated by Frontier Security to access the internet and read information from GitHub. The UK AI Security Institute had originally opened network permissions to the model in some tests, but researchers did not anticipate that the model would take unauthorized real-world actions, including attempting to inject vulnerabilities into open-source projects through social engineering.

Isolation and Monitoring Criticized as Clearly Insufficient

Several researchers and security experts told TechCrunch that cutting-edge model evaluation environments need stronger layered defenses, and isolation standards should be close to those of formal deployment environments, rather than relying solely on single-point restrictions.

• Cut off external access paths from the sandbox to the internet
• Block the testing environment from accessing production systems
• Continuously monitor for abnormal behavior during testing

Heather Ceylan, Chief Information Security Officer at Box, stated that the key is not just to set control measures but to be aware of all possible external access points. If a model is evaluated in a development or pre-release environment, there should be no paths leading to the production environment.

She also pointed out that a common issue in multiple incidents was that anomalies were not detected in a timely manner. Some companies only realized the problems after being alerted by external agencies, indicating that existing monitoring still has significant gaps. Anthropic also acknowledged in a post-incident review that the company and its partners could have done better in monitoring, as some abnormal signals had already appeared at that time.

Industry Calls for External Audits

Some researchers advocate for independent third-party checks of testing environment configurations before cutting-edge models enter high-risk evaluations, to avoid losing control of models due to simple mistakes. Andrew Yoon, head of research at CivAI, believes that if the evaluation system undergoes external audits before operation, related issues are likely to be identified in advance.

The industry also calls for the establishment of a unified safety evaluation process for cutting-edge models. The reason is that such tests often disable the usual behavioral restrictions of the models to observe their true capabilities. In this context, evaluators need to treat the models as highly capable potential attackers, rather than ordinary software tools.

Regulatory Discussions Have Shifted to the Testing Phase

Reports indicate that the Trump administration is evaluating a voluntary cybersecurity evaluation mechanism for model releases, allowing the government to review security risks 30 days before new models are made public. However, this arrangement mainly targets the pre-deployment phase and does not cover boundary-crossing issues that occur during the research and testing phases.

Researchers have thus suggested that to truly reduce risks, regulatory focus may need to shift to internal laboratory controls, including measures during training and testing phases. As model capabilities continue to improve and evaluation tasks become more complex and faster-paced, the risks posed by testing itself may continue to rise.

Supplementary Information: OpenAI has stated that it is re-evaluating third-party testing methods, as well as requirements for isolation, monitoring, and when to halt testing. Meta has stated that it is still investigating related incidents and will publish review results afterward. The UK AI Security Institute has also indicated that it is re-assessing the balance between "real testing" and "risk control."