Cybersecurity leaders scramble to educate employees on generative AI threats

Brainstorm AI U.S.

Sage Lazzaro

Happy new year, and welcome to Eye on AI. In this edition: How cybersecurity training is—and isn&#8217;t—keeping up with generative AI&#8217;s new threats; OpenAI beefs up its political lobbying; Nvidia to open-source Run:AI software following acquisition; and AI leadership roles triple in two years. <div class="paywall-selector"></div><div class="paywall">



2024 saw brand new types of generative AI-enabled digital fraud make headlines, from a <a href="https://fortune.com/2024/02/08/generative-ai-fraud-identity-theft-cybersecurity-risk/">deepfaked video call that cost a company $25 million</a> to new research on how AI copilots being built into enterprise software can <a href="https://fortune.com/2024/08/13/microsoft-ai-copilot-hacking-prompt-injectoin-attack-black-hat/">be weaponized as “automatic phishing machines”</a>. Even classic phishing attacks are getting worse and getting more personal, the <a href="https://www.ft.com/content/d60fb4fb-cb85-4df7-b246-ec3d08260e6f">Financial Times</a> reported today, thanks to AI bots’ ability to easily ingest large amounts of data about a company or person’s style and tone and then easily replicate it. They can also scrape data from a person’s online activity to make phishing emails more personal, and thus more convincing.  



As generative AI swiftly upends the cybersecurity threat landscape, companies need to ensure employees are aware of the technology, its capabilities, and its risks. To educate employees on how not to fall victim to various types of attacks, most companies turn to cybersecurity training, which typically consists of an informational video or a series of modules and quizzes that employees must complete. So how is this training keeping up with the new threats being posed by generative AI? I checked in with top providers including Huntress, Ninjio, and KnowBe4 and watched their trainings to find out.



<h2 class="wp-block-heading">Gen AI cybersecurity trainings cover some, but not all, new threats </h2>



Across all the training courses I tested, some of the new threats posed by generative AI were covered thoroughly, in particular how the technology can be used to create more convincing phishing emails and the risks involved with inputting sensitive company information into commercial chatbots. Ninjio’s AI-related training was the only one I reviewed that didn’t cover phishing, though the company says it plans to release a new video on this early this year. 



Most of the trainings also covered how generative AI can be used to make convincing deepfakes of someone’s voice. Some addressed video deepfakes, but they focused on videos that would be distributed to employees or shared online. The trainings from Huntress, notably, did not discuss deepfakes of any sort. None of the trainings, however, discussed the emergence of deepfakes being used in live video calls—like the aforementioned instance from earlier this year when an employee at an multinational company thought he was on a call with his executive team, but was actually talking to AI-generated deepfakes created by malicious actors, who deceived him into transfering $25 million of the company’s funds. 



None of the trainings addressed prompt injection, a new type of attack that could be used against companies deploying AI assistants in enterprise software like Microsoft 365. These attacks exploit the AI assistant’s ability to retrieve documents and can open up companies to data theft and new types of social engineering attacks. For example, a malicious actor could send an email to a victim at a target company that presents the bad actor’s bank information as that of some trusted entity. If the employee uses Copilot to search for that entity’s banking information, Copilot could surface the malicious email, misleading the victim into sending the money to the malicious actor instead. In the same vein, a hacker could send a malicious email to direct someone to a phishing site—all without having to gain access to the employee’s email. 



<h2 class="wp-block-heading">Companies must be proactive about employee education</h2>



I began looking into how cybersecurity training is addressing the new threats posed by generative AI after watching a training video assigned to a relative. When they told me they had just been assigned a new video specifically talking about generative AI as part of their annual cybersecurity training, I was excited to see how the increasingly important topic was being covered. After watching the video, however, I was shocked and disappointed. The video was just a few minutes long, barely touched on the new types of threats I had reported on this past year, and depicted generative AI as if it were some far-off sci-fi technology. It wasn’t until I started diving deeper into the offerings from the cybersecurity industry that I realized that video wasn’t the whole story.&nbsp;



That video was from KnowBe4, and on its own, I don’t think it would be sufficient for informing employees about the risks and threats. I soon discovered that it’s just one of many AI-focused cybersecurity videos offered by KnowBe4, which turned out to have the largest catalog of AI-focused videos and some of the most informative content out of everything I viewed. KnowBe4 told me company admins are able to preview and assign trainings based on their company’s needs. Clearly whoever chose the videos to assign for my relative’s company wasn’t as thorough as they should’ve been. There were additional videos on deepfakes, CEO fraud, phishing, and the dangers of AI chatbots that together would’ve been much more comprehensive.&nbsp;



This made clear that cybersecurity and IT leaders inside companies need to take an active role, familiarizing themselves with the new threats and the training content that exists to inform employees about them. More than ever, cybersecurity education needs to be continuous; a brief video once a year isn’t enough.



Cybersecurity is an endless cat-and-mouse cycle, with security professionals and IT teams often playing catch-up to whatever innovations the fraudsters and hackers decide to adopt. Huntress said it&#8217;s in the late stages of developing a training on AI hallucinations and plans to create one on deepfakes this year. Ninjio is still developing trainings on generative AI’s impact on phishing and how malicious actors can use AI to automate attacks. KnowBe4 said it’s working to incorporate information on prompt injection attacks into its trainings. (It was the only provider to directly address my questions about this type of emerging threat.) But the training courses won’t do anything unless corporate IT leaders do their job in making sure employees engage in the trainings thoroughly and regularly. 



And with that, here’s more AI news.&nbsp;



Sage Lazzaro sage.lazzaro@consultant.fortune.com <a href="http://sagelazzaro.com">sagelazzaro.com</a><h3>AI IN THE NEWS</h3>OpenAI expands its D.C. policy team to beef up lobbying efforts. The company recently tripled the size of its policy team to 12 members, scooping up D.C. insiders from across the political spectrum. While still smaller than many big tech lobbying shops, it signals how OpenAI is increasingly looking to influence tech policy in Washington, particularly as the Trump administration (which has appointed AI figures including Elon Musk and venture capitalist David Sacks to help run the government) prepares to reenter office. Specifically, OpenAI hopes to convince government leaders that the AI industry is a vital part of the economic and security race against China, as well as drum up support for building infrastructure to support AI development. You can read more from <a href="https://www.politico.com/news/2024/12/27/openai-washington-presence-00196056">Politico</a>.
Nvidia completes its acquisition of Run:AI and says it will open-source the software. The company’s software so far has only supported Nvidia’s GPUs, enabling customers to schedule GPU resources for AI workloads in the cloud. Open-sourcing it was potentially key to avoiding antitrust scrutiny of the deal, which Nvidia is increasingly subjected to due to its enormous growth and dominant role in AI hardware. The final cost of the deal was not disclosed, but prior reports estimated it at $700 million. You can read more in <a href="https://venturebeat.com/ai/nvidia-acquires-software-maker-runai-to-orchestrate-gpu-clouds-for-ai/">VentureBeat</a>. 
State legislatures keep their eyes on AI for 2025. Anticipating more gridlock in Congress, state lawmakers are planning to take on big issues themselves, including tech topics such as AI, <a href="https://www.pbs.org/newshour/politics/ai-housing-and-other-top-issues-for-state-lawmakers-in-2025">PBS reported</a>. 2024 saw state lawmakers struggling with whether to pass legislation on AI, with some state-level efforts stalling after key legislators or state governors argued that regulation of a technology as impactful and sweeping as AI should be set at the federal level. 
<h3>FORTUNE ON AI</h3><a href="https://fortune.com/2025/01/02/kobold-mining-jeff-bezos-bill-gates-investment-round-copper-lithium/">AI-powered mining firm backed by Bill Gates and Jeff Bezos is now worth $2.96 billion as it takes on Chinese rivals</a> —by Eleanor Pringle
<a href="https://fortune.com/2024/12/20/openai-o3-sam-altman-agi-chollet/">Sam Altman says OpenAI’s new o3 ‘reasoning’ models begin the ‘next phase’ of AI. Is this AGI?</a> —by Sharon Goldman
<a href="https://fortune.com/europe/2025/01/01/2025-predictions-european-business-landscape-ai-consumer-spending-fashion-luxury-auto/">From AI reasoning to green fatigue: Fortune’s bold business predictions for Europe in 2025</a> —by Fortune Editors
<h3>AI CALENDAR</h3>Jan. 7-10: CES, Las Vegas
Jan. 16-18: DLD Conference, Munich
Jan. 20-25: World Economic Forum, Davos, Switzerland
Feb. 10-11: AI Action Summit, Paris, France
March 3-6: MWC, Barcelona
March 7-15: SXSW, Austin
March 10-13: Human [X] conference, Las Vegas
March 17-20: Nvidia GTC, San Jose
April 9-11: Google Cloud Next, Las Vegas
<h3>EYE ON AI NUMBERS</h3><h2>$1 billion</h2>
That’s how much Nvidia invested in AI startups in 2024, compared to $872 million in 2023, the <a href="http://ft.com/content/f8acce90-9c4d-4433-b189-e79cad29f74e">Financial Times reported</a>. The investment took place across 50 startup funding rounds and several corporate deals. Much of the money has gone towards backing its own customers and other companies supporting its position in the AI race. The mass of spending makes Nvidia one of the most crucial investors of the AI, as well as being the most crucial hardware provider of the generative AI boom.</div>

Cybersecurity training is starting to cover deepfakes and improved phishing campaigns, but emerging risks, such as prompt injection, remain unaddressed.

Cybersecurity leaders scramble to update training to cover new generative AI threats

A infographic depicting which companies are most impersonated in online phishing scams.

This is the online version of Eye on AI, Fortune's weekly newsletter on how AI is shaping the future of business. <a href="https://www.fortune.com/newsletters/eye-on-ai?&itm_source=fortune&itm_medium=nl_article_tout&itm_campaign=eye_on_ai">Sign up for free</a>.

P-Newsletter

Eye on AI

Newsletters

Tech

Employee training courses cover deepfakes and phishing, but prompt injections remains unaddressed.

Cybersecurity leaders scramble to keep up with gen AI's new threats

Brainstorm A.I.

Brainstorm Tech

Cybersecurity leaders scramble to update training to cover new generative AI threats | Fortune

Sara Braun

A young business man sits on steps and holds his head in his hands.

American worker confidence just hit a record low and is even worse than it was during the darkest days of the pandemic  

/2025/03/28/american-worker-confidence-record-low-linkedin-survey-worse-than-pandemic-levels/

Alicia Adamczyk

Nina Ajemian

Doja Cat and Charli XCX’s agent on her groundbreaking career in the music industry

/2025/03/28/doja-cat-and-charli-xcxs-agent-on-her-ground-breaking-career-in-the-music-industry/

Allie Garfinkle

Sam Altman holds a microphone and speaks amid a bright multicolor backdrop.

There are 372 AI unicorns right now, according to PitchBook. Here’s what that means

/2025/03/28/there-are-372-ai-unicorns-right-now-according-to-pitchbook-heres-what-that-means/

Sheryl Estrada

Toast’s CFO and president Elena Gomez got her start in finance at age 5, with the help of her grandfather

/2025/03/28/toast-cfo-president-elena-gomez-start-finance-age-5-grandfather/

Andrew Nusca

Ubisoft’s next chapter begins, with Tencent’s help

/2025/03/28/ubisofts-next-chapter-begins-with-tencents-help/

Diane Brady

‘We’re just hanging on for dear life’: How CEOs are navigating increasing geopolitical uncertainty.

/2025/03/28/were-just-hanging-on-for-dear-life-how-ceos-are-navigating-increasing-geopolitical-uncertainty/

The Associated Press

The European Union just issued a dire warning to its 450 million citizens: Stockpile supplies and prepare for disaster

https://fortune.com/2025/03/26/eu-citizens-stockpile-food-water-essentials-72-hours-hadja-lahbib/?itm_source=parsely-api

Chris Morris

Tesla is suing to open dealerships in Wisconsin, the state where Elon Musk is spending heavily to influence judicial elections

https://fortune.com/2025/03/27/tesla-elon-musk-wisconsin-dealerships-political-influence/?itm_source=parsely-api

Sasha Rogelberg

Oracle is reportedly among a wave of tech giants hit by DOGE cuts, as defense secretary balks at 'throwing more good taxpayer money after bad'

https://fortune.com/2025/03/28/oracle-contract-cut-defense-department-pete-hegseth-doge/?itm_source=parsely-api

Didi Tang

Matt O'Brien

American robot firms, including Tesla and Boston Dynamics, warn that without a national strategy ‘the U.S. will not only lose the robotics race but also the AI race’

https://fortune.com/2025/03/27/association-advanced-automation-tesla-boston-dynamics-national-strategy/?itm_source=parsely-api

Sydney Lake

Walmart CEO Doug McMillon says customers are exhibiting 'stressed behaviors'—and it's already tanked the company's valuation by $22 billion

https://fortune.com/2025/03/26/walmart-ceo-doug-mcmillon-customers-stressed-valuation-stock-drops/?itm_source=parsely-api

Irina Ivanova

Airbnb cofounder Joe Gebbia says DOGE is pushing to create an ‘Apple Store–like experience’ in government

https://fortune.com/2025/03/28/joe-gebbia-doge-federal-employee-retirement-system-apple-store/?itm_source=parsely-api