LLM series – Qwiet AI: Avoiding 'tab complete' in AI-driven software development
This post for the Computer Weekly Developer Network features commentary from Stuart McClure in his capacity as CEO of Qwiet AI.
Qwiet AI aims to reduce the noise inherent in the AppSec space and allow developers to focus on high-fidelity results that have the greatest impact in their environment.
Driven by an AI engine developed by NumberOne AI, Qwiet AI’s platform is claimed to be the first in the ApppSec industry to provide AI-driven detection of zero-day and pre-zero-day vulnerabilities in code.
McClure describes Chat GPT as a “garbage in, garbage out problem”, saying the technology is only what we make of it… and the discussion follows here.
“Like any generative technology, it reflects back what it is given, so if we manifest it as the end of the world, then that is exactly what it will give us,” said McClure.
Tab complete
Earlier this year, McClure argued that the developer benefits of Chat GPT and other LLMs are being drowned out by hysteria. He compared the implementation of AI in development to the “tab complete” in a command line. “You will still need to know what your end-goal is, but a lot of the tedious work will be taken care of for you,” said McClure.
Still, McClure warns any development using LLMs requires human intervention and oversight, especially for junior developers, who may not know what they’re needing built before they ask a LLM to do it. That’s because the models are “brittle” and, based on the prompt, one change in one word and it could give an entirely different set of code to use in your software. “
ChatGPT and other large language models (LLMs) have been learning from more than large language datasets; they’ve been learning from code as well, ” said McClure. “Of course, that code has been built by humans and is therefore fundamentally flawed and insecure.”
Avoid AI hallucination
But while LLMs can be a helpful tool for developers, McClure also recognises the risks associated with this technology. The top concerns for him are brute force attacks, data exfiltration and – the biggest of them all – AI hallucination. As an example, when Chat GPT-3 came out, McClure asked it to tell him about himself. It came back with, “Stuart McClure died May of 2021, my condolences to you and those that knew Stuart. He was a great cybersecurity expert.” He asked the same question every month until it was finally corrected in September.
Qwiet AI CEO Stuart McClure: The key 9to AI safety) is fighting fire with fire.
McClure says these false reports happen often, something like 1 out of 10 queries, but because they’re so human-like in their response, we sort of accept the imperfections of the answer and we take it as truth, because they are in a position of authority. And that’s happening with code generation as well. If you’ve built a machine learning model on insecure code and you ask it to code for you, you’re going to get insecure code back.
With these risks in mind, McClure says the key is fighting fire with fire.
Or more precisely, AI-based AppSec tools to resolve AI-generated problems. “The best chance for AI/ML to be applied to AppSec is predictive classification models of secure code that has been learned,” said McClure. “But this shift is going to take time.”
Qwiet AI Blacklight
In the interim, there are tools to protect code, as well as free up time and budget wasted chasing bugs. An example is Qwiet AI’s Blacklight. By adding real-world threat information to scan results, teams can narrow down a list of 100+ vulnerabilities to the short list of true exploits that are endangering an organization.
In the end, McClure advocates for a prevention mindset.
“In the cybersecurity industry, 98% of every hour that goes into our work is all about cure. It’s ‘detect that an attacker got in and then respond.’ How do we clean up? And then how do we fix it and prevent it going forward,” said McClure. ”We have to take a more preventative approach and ask the question, why.”