AI system poisoning is a growing threat—is your security system ready?

WhatsApp Group Join Now
Telegram Group Join Now
Instagram Group Join Now

Consulting firm Protiviti recently worked with a client company experiencing an unusual attack: a hacker attempting to manipulate data being fed into one of the company's AI systems. Company leaders are still unraveling the details of the attack, but John Stevenson, managing director of Protuity, says the company suspects the hacker was trying to tamper with the output of the AI ​​system.

Such attacks are not new, yet they are at the forefront of the cybersecurity world. As software company Splunk's “2024 State of Security” report states: “The potential for AI poisoning remains, [but] It is yet to become public.”

However, this external status is expected to change, with leading security experts predicting that hackers will increasingly target AI systems and, in particular, attempt to poison them by corrupting data or models.

CISOs everywhere need to be prepared, as organizations of all sizes and types can be targets.

“Every company has to face it one way or another. [in-house developed] through the AI ​​models or third-party AI tools they use,” says Stevenson.

NIST issues warnings about 4 types of poisoning attacks.

The National Institute of Standards and Technology (NIST), a US agency, warned of what's to come in a January 2024 paper. “Poison attacks are very powerful and can cause availability violations or integrity violations,” the researchers wrote.

“Specifically, availability poisoning attacks lead to indiscriminate degradation of machine learning models on all samples, while targeted and backdoor poisoning attacks are more covert and cause integrity violations on a small set of target samples. lead to,” NIST wrote.

This paper highlights four types of poisoning attacks.

  1. Availability Poison“which indiscriminately affects the entire model of machine learning and, in essence, causes a denial of service to users of the AI ​​system.”
  2. Targeted poisoningin which hackers “change the [machine learning] Model prediction on a small number of target samples.
  3. Backdoor poisoningwhere “image classifiers can be poisoned at training time by adding a small patch trigger to a subset of images and changing their label to the target class,” NIST further notes that although “back While the majority of door poisoning attacks are designed for computer vision applications, this attack vector has been effective in other application domains with various data modalities, such as audio, NLP, and cybersecurity settings.”
  4. And Model poisonattacks that “attempt to directly modify a trained ML model to inject malicious functionality into the model.”

NIST and security leaders note that poisoning is in addition to several other attack types targeting AI, such as privacy compromise as well as direct and indirect instant injection.

“Deploying AI in your enterprise introduces a new attack surface that is very different,” says Apostol Vassilev, NIST research team supervisor and co-author of the NIST paper. “We've seen achievements by academics and other researchers trying to identify potential problems, but the more this technology is used, the more important it will be for hackers to attack, and That's why we're going to have more consequences.”

He adds: “Already we're starting to see it accelerate.”

AI poisoning attacks can come from inside or outside an organization.

Security experts say that poisoning attacks can be launched by both internal and external hackers – as is the case with more traditional types of cyber security attacks.

And in another parallel with conventional attack types, “nation-states are probably one of the biggest threats here because they have the capacity and the resources to invest in it. [type of attack]”, says David Yusuf, a managing director at FTI Consulting and leader of the North American incident response efforts for the firm's cybersecurity practice.

Bad actors' motivations for malware attacks are also familiar, according to security experts, who say hackers can target AI systems for the same reasons they launch other types of cyberattacks, such as a Disrupt or harm the organization. Some say that hackers can also launch poisoning attacks to gain access to proprietary data or get money.

“Could someone use it for extortion? Absolutely,” says Eric Avakian, technical advisor at InfoTech Research Group and former state CISO for the Commonwealth of Pennsylvania. If he can, he can use it. They might say, 'We poisoned the model, now you have to pay us. [to get information on what we did].'

The primary targets will likely be tech companies that build AI systems.

While such motivations mean any organization using AI could fall victim to it, Ken McGladrey, a senior fellow at the Institute of Electrical and Electronics Engineers (IEEE), a nonprofit professional association. , and the field CISO at Hyperproof says they expect more hackers. Tech companies that build and train AI systems are likely to be targeted.

But CISOs shouldn't breathe a sigh of relief, McGladrey says, as their organizations could be vulnerable to these attacks if they're using vendor-supplied corrupted AI systems.

A recent case illustrates the potential for far-reaching damage in such scenarios. Researchers at tech company JFrog discovered that nearly 100 malicious machine learning models had been uploaded to Hugging Face, a public AI model repository. The researchers said in a February 2024 blog that malicious ML models could enable malicious actors to inject malicious code into users' machines once the model is loaded, a scenario that could lead to the vulnerability of the user's environment. Counts can quickly become compromised.

Experts say more such incidents are on the horizon.

“I'm thinking this is an emerging risk, and once AI technology scales, the risk of poisoning is greater,” says Mary Carmichael, managing director of risk advisory at Momentum Technology and a member of the Emerging Trends Working Group. It will become clear.” Risk Advisory Committee at Governance Association ISACA.

Preparing to respond to AI poisoning now will help protect against what's to come.

Security experts and CISOs themselves say that many organizations are ill-prepared to detect and respond to toxic attacks.

“We're a long way from having really strong security around AI because it's evolving so quickly,” says Stevenson.


He points to a Proteity client that suffered a suspected poisoning attack, noting that workers at that company identified a potential attack because its “data wasn't syncing up, and when they When he dived into it, he identified the problem. [The company did not find it because] The bells and whistles of a security tool were ringing.”

He adds: “I don't think many companies are set up to detect and respond to these types of attacks.”

A February 2024 report from ISC2, a nonprofit organization offering training and certification for cybersecurity professionals, sheds light on whether CISOs feel prepared for what's ahead.

The report found that 75% of more than 1,100 respondents said they were moderately to highly concerned that AI would be used for cyberattacks or other malicious activities, including deepfakes, Misinformation, and social engineering are the top three concerns for cyber professionals.

Despite this high level of concern, only 60% said they felt confident in their ability to safely lead their organization's adoption of AI. Moreover, 41% said they have little or no expertise in securing AI and ML technology. Meanwhile, just 27 percent said their organization has formal policies in place regarding the safe and ethical use of AI.

“The average CISO is not skilled in AI development and does not have AI expertise as a core competency,” says Jon France, CISO with ISC2.

Even if they were AI experts, they would likely face challenges in determining whether a hacker had successfully executed a poison attack.

As Vassilev explains, owners and users of AI systems will struggle to detect hackers, who can turn behaviors on and off without detection. And they won't be able to look at the source code and find a trigger once the model is poisoned.

The unpredictable nature of creative AI further challenges detection and response, he added.

Defense against threats to AI systems

As has long been the case in security, no single tool is immune to poisoning attacks.


Similarly, long-standing security practices can reduce risk, detect anomalies and speed up recovery, experts said. They recommend a multi-pronged defense strategy that includes a robust access and identity management program, a security information and event management (SIEM) system, and anomaly detection tools. “So, you know if someone has accessed your system,” says Avakian.

Avakian added that robust data governance practices, as well as oversight and monitoring of AI tools, are also essential.

Therefore, Carmichael says, it's also a good idea for vendors to ensure that vendors providing AI tools are doing what they should to protect their products from being vulnerable to malicious attacks.

Vassilev says CISOs should work with other executives to identify and understand the risks associated with AI tools (including poisoning attacks), devising strategies to mitigate those risks, which are very high. are high, and clarify the residual risk they want. to accept.

Vasiliev says CISOs and their organizations should also be aware of the models they use and the lineage of their data.


The NIST Adversarial Machine Learning paper provides more detailed mitigation strategies as well as more details on poisoning and other types of attacks.

Some security leaders advise CISOs to also add talent specifically trained in AI security to their teams.

“This work requires advanced data scientists, teams that know how to evaluate training sets and models. It's not going to be done by your average SOC teams,” Yusuf says, adding that Chief AI officers and CISOs should work together on governance and security projects. “The general protections we have today may not be enough, but the right approach is not to avoid AI but to understand the risk, work with the right people, assess it properly and take steps to minimize it. Is.”

WhatsApp Group Join Now
Telegram Group Join Now
Instagram Group Join Now

Leave a Comment