Are AI-based attacks great for security awareness training?

By Tom Tovar, CEO and co-creator, Appdome

In an era where artificial intelligence (AI) is advancing at an astonishing pace, traditional security awareness training is being challenged like never before. The rise of sophisticated AI-powered threats such as phishing, vishing, deepfakes, and AI chatbot-based attacks can quickly render this traditional human-centric approach to defense ineffective.

Today, humans have a slight advantage

Currently, security awareness training teaches individuals to recognize the signs and tactics used in social engineering attacks. Customers and employees are taught to recognize suspicious emails (phishing), suspicious text messages (smiling) and manipulative phone calls (vishing). Training programs help individuals identify red flags and detect subtle inconsistencies—such as slight variations in language, unexpected requests, or minor errors in communication—to provide an important line of defense.

A well-trained employee can see that an e-mail purported to be from a colleague contains unusual phrases or that a voice message requesting sensitive information comes from an executive who already has access to that information. Must be accessible. Users can also be trained to avoid mass-produced smiles and deception with some effect.

However, even the best trained are fallible. Stress, fatigue, and cognitive overload can affect judgment, making it easier for AI-attacks to succeed.

Tomorrow, AI has an advantage.

Fast forward two to three years and AI-powered attacks will have access to more data and bigger and better large-scale language models (LLMs). They will create more reliable, context-aware interactions that mimic human behavior with alarming accuracy.

Today, AI-assisted attack tools can craft emails and messages that are virtually indistinguishable from legitimate communications. Voice cloning can also virtually mimic someone's speech. Tomorrow, these techniques, combined with advanced deep learning models, will integrate real-time data, spyware, speech patterns and more into the nearest deep fax, making AI-powered attacks indistinguishable from human touch. can go

Already, AI-based attacks have advantages including:

At Appdome, we're starting to see exploits using AI chatbots, superimposed on a mobile application via an overlay attack, engaging a user or employee in a seemingly innocuous conversation. Some brands are starting to prepare for a similar attack via an AI-powered keyboard that an infected mobile device installs. In either case, the overlay or keyboard can collect information about the victim, persuade the victim, offer harmful choices, or act on the victim's behalf to compromise security, accounts or transactions. can do. Unlike today, where anomalies can be detected and actions controlled by an individual, the future of AI-powered attacks will involve autonomously generated interactions within applications and AI agents. Insects that act on behalf of the victim remove humans entirely from the attack lifecycle.

The future of security awareness training

As AI technology continues to evolve, traditional security awareness training faces an existential threat, and the margin for human error is rapidly diminishing. The future of security awareness training requires a multi-pronged approach that leverages human training and intuition as well as real-time automated intervention, improved cyber transparency, and AI detection.

Technical attack interference

Security awareness training should be expanded to teach individuals to recognize a true technical intrusion by a brand or enterprise, not just an attack. Even if the individual cannot distinguish a real from a fake interaction by an attacker, it should be easy to recognize a system-level intrusion designed to protect the user. Brands and enterprises can detect when malware, espionage techniques, controls, and account takeovers are in use, and they can use this information to intervene before any real damage occurs.

Better cyber transparency

To promote security awareness training, organizations need to adopt better cyber transparency so that users can understand the expected defensive responses in applications or systems. Of course, this requires strong defensive technology initiatives across applications and systems to begin with. Still, enterprise policies and consumer-facing product release notes should include “what to expect” when a threat is detected through brand or enterprise defenses.

Identify AI and AI agents interacting with apps

Brands and enterprises must deploy defenses that detect the unique ways machines interact with applications and systems. This includes typing, tapping, recording, in-app or on-device movements, and even the systems used for these interactions. Non-human patterns can be used to trigger end-user alerts, enhance due diligence workflows within applications, or perform additional authorization steps to complete transactions.

Prepare for an AI-powered future.

The rise of AI-powered social engineering attacks marks a significant shift in the cybersecurity landscape. If security awareness training is to become a valuable tool in cyber defense, it must include the ability to recognize application and system-level intrusions, improved cyber transparency, and automated interactions with applications and systems. By doing so, we can protect our brands and enterprises from the inevitable rise of AI-powered fraud and help ensure a more secure future.

About the author

Tom Tovar is the CEO and co-creator of Appdome, the only fully automated unified mobile app defense platform. Today, he is a coder, hacker, and business leader. He began his career as a Stanford-educated, tech-focused, corporate and securities lawyer. He brings practical advice as a board member and serving in C-level leadership roles at several cyber and technology companies.