If you give Copilot the reins, don't be surprised when it spills your secrets
One hopes widely used enterprise software is secure enough. Get ready for those hopes to be dashed again, as Zenity CTO Michael Bargury today revealed his Microsoft Copilot exploits at Black Hat.
"It's actually very difficult to create a [Copilot Studio] bot that is safe," Bargury told
The Register in an interview ahead of conference talks, "because all of the defaults are insecure."
Bargury is speaking twice about security failings with Microsoft Copilot at Black Hat in Las Vegas this week. His
first talk focused on the aforementioned Copilot Studio, Microsoft's no-code tool for building custom enterprise Copilot bots, its defaults, and other aspects. The
second covered all the nasty things an attacker can do with Copilot itself if they manage to break into the IT environment of an organization that uses the tech, and how Copilot can help someone gain entry to that environment.
Zenity, for what it's worth, offers among other things security controls for Copilot and similar enterprise-level assistants. Bear that in mind. It warns of the risks of using Microsoft's AI services
here.
Your Copilot bots are quite chatty
If you don't have much exposure to
Copilot Studio, it's a tool for non-technical people to create simple conversational bots, using Microsoft's Copilot AI, that can answer people's questions using internal business documents and data. This is made possible by what's called retrieval-augmented generation, or
RAG.
It's Microsoft's way "to extend [Copilot's] tentacles into other business areas, such as CRM and ERP," as we wrote
here. Companies can create customer and/or employee-facing bots that provide a natural-language interface to internal information.
Unfortunately for all the Copilot Studio customers out there, we're told the default settings in the platform at least were entirely insufficient. Combine those with what Zenity marketing chief Andrew Silberman told us is nearly 3,000 Copilot Studio bots in the average large enterprise – we're talking Fortune 500-level companies here – along with research indicating that 63 percent of those are discoverable online, and you have a recipe for potential data exfiltration.
If these bots are accessible to the public, and we're told a good number of them are, they can be potentially tricked into handing over, or simply hand over by design, information to people that should not have been volunteered during conversations, it's claimed.
As Copilot bots frequently have access to internal company data and sensitive documents, it's a matter of figuring out how to fool or prompt them into disclosing that data, we're told. Bargury said he was able to do that by configuring ChatGPT to fuzz Copilot bots with automated, malformed prompts.
"We scanned the internet and found tens of thousands of these bots," Bargury said. He blamed the high online availability of these agents on default Copilot Studio settings that published them to the web without any need to authenticate to access them - an oversight Microsoft has since fixed after the Zenity team brought it to their attention.
Unfortunately, the new settings that keep Copilot Studio bots off the public internet by default currently only apply to new installations, Bargury said, so users of the suite who deployed bots before now should check their settings to be sure.
Bargury and his team have
released a tool to "scan for publicly accessible Copilot Studio bots and extract information from them," which folks are welcome to use to test their environments' security. Dubbed CopilotHunter, it's available as a module in PowerPwn, a
software toolkit Zenity released at Black Hat last year for
probing Microsoft 365 guest accounts.
In short, if you've deployed Copilot Studio bots, check to see how available they are to the world.
While Bargury told The Reg he may have overextended himself by planning two Black Hat talks this year, his second shows no less effort than the first.
Copilot, Bargury demonstrated this week, is susceptible to indirect prompt injection attacks, which he argues is on a par with remote code execution (RCE) in terms of severity.
"An RCE is simply, from a remote location, being able to execute code that does something on your machine," Bargury said. "Indirect prompt injection that makes an AI do something on your behalf is the exact same thing with the same impact."
With access to a compromised environment, Bargury said he can instruct that Microsoft customer's Copilot "to automate spear phishing for all of your victim’s collaborators," use the technology to lure internal users to phishing pages, access "sensitive content without leaving a trace," and more.
To top it all off, it's claimed Copilot can be tricked into granting someone initial access to that environment, and conduct other malicious activities, with nothing but an email, direct message, calendar invite, or other common phishing tactic, and that this can work without the user needing to interact with it or click a link because of how Copilot scans messages. The key thing here is the indirect prompt injection: Crafting messages that the AI assistant picks up and parses to be helpful, and as a result does things it shouldn't.
"Microsoft Copilot is built on the enterprise graph," Bargury explained. Once a message, email or invite is sent it hits the graph, Copilot scans it, "and that's a path for me to start with prompt injection."
In one example, Bargury demonstrated how he was able to change banking information to intercept a bank transfer between a company and client "just by sending an email to the person."
An AI bot feature
Bargury explained to us that he sees these discoveries as indicative of the industry still being in the very early days of artificial intelligence in the enterprise, and having to face the fact that AI is changing our relationship with data.
"There's a fundamental issue here," he said. "When you give AI access to data, that data is now an attack surface for prompt injection."
If that's true, Copilot bots are by their very nature insecure since many are publicly accessible, they're tied closely to enterprise data, and are ready to spill secrets with a bit of hidden HTML or a ChatGPT-powered fuzzing bot.
"It's kind of funny in a way - if you have a bot that's useful, then it's vulnerable. If it's not vulnerable, it's not useful," Bargury said.
The Zenity CTO noted that Microsoft has been incredibly responsive to his reports, and said several of the faults he found have been addressed, albeit within limits.
"[AI] apps are basically changing in production because AI chooses to do what it wants, so you can't expect to have a platform that's just secure and that's it," Bargury said. "That's not going to happen because these platforms have to be flexible, otherwise they're not useful."
Bargury believes that securing AI software like Copilot requires real-time monitoring, inspecting conversations with the bots, and tracking potential prompt-injection RCEs. The bottom line is that businesses are the guinea pigs testing an experimental drug called artificial intelligence, and we're not at a point where we know how to make it safe yet.
Finally, Bargury and team released another testing kit called LOLCopilot for organizations that want to test their setups for vulnerability to his exploits.
"Copilot has great skills. It can search, it can enable your employees to find data they have access to but didn't know they did … those things are important," Bargury told us. "But that's not as important as preventing remote code execution."
finance.yahoo.com
