Amazon (AWS) has announced the general availability of its new security AI service known as 'Continuum', which aims to scan source code repositories, detect and verify vulnerabilities, match them with the Amazon cloud environment, and then open Pull Requests to fix them automatically. This launch comes after the tool achieved notable results in benchmark security tests.
Details of the 'CyberGym-E2E' Benchmark Results
The tool achieved a success rate of 89.0% by passing 819 out of 920 tasks in the benchmark test known as CyberGym-E2E, surpassing the previous record of 65.9%. This test relies on giving the AI-based cyber agent 90 minutes to discover a security vulnerability in one of 139 open-source projects, without providing a detailed description of the vulnerability, a crash log, or a previous patch.
It is worth noting that all tasks in this test focused exclusively on memory-related security vulnerabilities in the C and C++ programming languages, ignoring other programming languages and types of vulnerabilities. The evaluation process operated in four cumulative stages that included detecting the vulnerability, stopping the crash, and maintaining the integrity of the project's existing tests.
What does this achievement mean in practice?
Although the tool excelled at finding and fixing real software defects without breaking existing tests, its success rate in fixing the specific historical vulnerability set by the benchmark for testing dropped to 37.8%. Amazon explains this discrepancy by noting that software repositories may contain multiple vulnerabilities; therefore, the tool fixing another real defect does not diminish the value of the work performed, especially since the agent did not know about the targeted historical vulnerability in advance.
Limitations and Open Questions
Despite the impressive figures published by Amazon, which confirm the tool's adherence to internet access limits and test conditions without searching for previously published solutions, questions remain about the tool's actual performance when software teams ask it to fix a specific, previously known vulnerability rather than performing free discovery. Furthermore, the results to date rely exclusively on tests conducted by the company itself.