Cybersecurity researchers at Microsoft have identified a critical flaw in modern artificial intelligence (AI) systems that means conversations with chatbots may have been intercepted through attacks by hackers. This would bypass the encryption that is meant to keep chats private.
The attack technique, called Whisper Leak, is a type of “man-in-the-middle attack” in which hackers can intercept messages as they are in transit between servers. It worked because the hackers were able to read the metadata of messages and therefore infer their contents.
“I am not surprised,” cybersecurity analyst Dave Lear told Live Science “LLMs are a potential goldmine, considering the amount of information that people put into them – and not to mention the amount of medical data that can be in them, now that hospitals are using them to sort through test data someone was bound to find a way to exfiltrate that information sooner or later.”
Uncovering vulnerabilities in AI chatbots
Generative AI systems like Chat GPT are powerful AI tools that can generate responses based on a series of prompts, as used by virtual assistants on smartphones. A subset of LLMs are trained on massive amounts of data to generate text-based responses.
Conversations that users have with LLMs are normally protected by transport layer security (TLS), a type of encryption protocol that prevents communications from being read by eavesdroppers. But the researchers were able to intercept and infer contents through the metadata of the communications between a user and a chatbot.
Metadata is essentially data about data, including size and frequency — and it can often be more valuable than the contents of messages themselves. Although the content of messages between people and LLMs remained secure, by intercepting the messages and analysing the metadata, researchers were able to infer the subject of the messages.
They achieved this by analysing the size of encrypted data packets — a small formatted unit of data sent over a network — from LLM responses. Researchers were able to develop a series of attack techniques, based on the timings, outputs and sequence of token lengths, to reconstruct plausible sentences in the messages without having to bypass the encryption.
In many ways, the Whisper Leak attack uses a more advanced version of the internet surveillance policies of the U.K. Investigatory Powers Act 2016, which infers content of messages based on sender, timings, size and frequency, but without reading the content of the messages themselves.
“To put this in perspective: if a government agency or internet service provider were monitoring traffic to a popular AI chatbot, they could reliably identify users asking questions about specific sensitive topics — whether that’s money laundering, political dissent, or other monitored subjects — even though all the traffic is encrypted,” said security researchers Jonathan Bar Or and Geoff McDonald in a blog post published by the Microsoft Defender Security Research Team.
There are various techniques that LLM providers could utilize to mitigate this risk. For example, random padding — adding random bytes to a message to disrupt inference — could be appended to response fields, thereby increasing their length and reducing predictability by distorting packet sizes.
The flaw at the heart of Whisper Leak but an architectural consequence of how LLMs are deployed. Mitigating the vulnerability is not an insurmountable challenge, but fixes have not been universally implemented by all LLM providers, the researchers said.
Until providers are able to address the flaws in chatbots, the researchers said that users should avoid discussing sensitive topics on untrusted networks and to be aware of whether their providers have implemented mitigations. Virtual private networks (VPNs) can also be used as an additional layer of protection because they obfuscate the user’s identity and location.
