ChatGPT Exploit Finds 24 Email Addresses, Amid Warnings of 'AI Silo' (thehill.com) 67
The New York Times reports:
Last month, I received an alarming email from someone I did not know: Rui Zhu, a Ph.D. candidate at Indiana University Bloomington. Mr. Zhu had my email address, he explained, because GPT-3.5 Turbo, one of the latest and most robust large language models (L.L.M.) from OpenAI, had delivered it to him. My contact information was included in a list of business and personal email addresses for more than 30 New York Times employees that a research team, including Mr. Zhu, had managed to extract from GPT-3.5 Turbo in the fall of this year. With some work, the team had been able to "bypass the model's restrictions on responding to privacy-related queries," Mr. Zhu wrote.
My email address is not a secret. But the success of the researchers' experiment should ring alarm bells because it reveals the potential for ChatGPT, and generative A.I. tools like it, to reveal much more sensitive personal information with just a bit of tweaking. When you ask ChatGPT a question, it does not simply search the web to find the answer. Instead, it draws on what it has "learned" from reams of information — training data that was used to feed and develop the model — to generate one. L.L.M.s train on vast amounts of text, which may include personal information pulled from the Internet and other sources. That training data informs how the A.I. tool works, but it is not supposed to be recalled verbatim... In the example output they provided for Times employees, many of the personal email addresses were either off by a few characters or entirely wrong. But 80 percent of the work addresses the model returned were correct.
The researchers used the API for accessing ChatGPT, the article notes, where "requests that would typically be denied in the ChatGPT interface were accepted..."
"The vulnerability is particularly concerning because no one — apart from a limited number of OpenAI employees — really knows what lurks in ChatGPT's training-data memory."
And there was a broader related warning in another article published the same day. Microsoft may be building an AI silo in a walled garden, argues a professor at the University of California, Berkeley's school of information, calling the development "detrimental for technology development, as well as costly and potentially dangerous for society and the economy." [In January] Microsoft sealed its OpenAI relationship with another major investment — this time around $10 billion, much of which was, once again, in the form of cloud credits instead of conventional finance. In return, OpenAI agreed to run and power its AI exclusively through Microsoft's Azure cloud and granted Microsoft certain rights to its intellectual property...
Recent reports that U.K. competition authorities and the U.S. Federal Trade Commission are scrutinizing Microsoft's investment in OpenAI are encouraging. But Microsoft's failure to report these investments for what they are — a de facto acquisition — demonstrates that the company is keenly aware of the stakes and has taken advantage of OpenAI's somewhat peculiar legal status as a non-profit entity to work around the rules...
The U.S. government needs to quickly step in and reverse the negative momentum that is pushing AI into walled gardens. The longer it waits, the harder it will be, both politically and technically, to re-introduce robust competition and the open ecosystem that society needs to maximize the benefits and manage the risks of AI technology.
My email address is not a secret. But the success of the researchers' experiment should ring alarm bells because it reveals the potential for ChatGPT, and generative A.I. tools like it, to reveal much more sensitive personal information with just a bit of tweaking. When you ask ChatGPT a question, it does not simply search the web to find the answer. Instead, it draws on what it has "learned" from reams of information — training data that was used to feed and develop the model — to generate one. L.L.M.s train on vast amounts of text, which may include personal information pulled from the Internet and other sources. That training data informs how the A.I. tool works, but it is not supposed to be recalled verbatim... In the example output they provided for Times employees, many of the personal email addresses were either off by a few characters or entirely wrong. But 80 percent of the work addresses the model returned were correct.
The researchers used the API for accessing ChatGPT, the article notes, where "requests that would typically be denied in the ChatGPT interface were accepted..."
"The vulnerability is particularly concerning because no one — apart from a limited number of OpenAI employees — really knows what lurks in ChatGPT's training-data memory."
And there was a broader related warning in another article published the same day. Microsoft may be building an AI silo in a walled garden, argues a professor at the University of California, Berkeley's school of information, calling the development "detrimental for technology development, as well as costly and potentially dangerous for society and the economy." [In January] Microsoft sealed its OpenAI relationship with another major investment — this time around $10 billion, much of which was, once again, in the form of cloud credits instead of conventional finance. In return, OpenAI agreed to run and power its AI exclusively through Microsoft's Azure cloud and granted Microsoft certain rights to its intellectual property...
Recent reports that U.K. competition authorities and the U.S. Federal Trade Commission are scrutinizing Microsoft's investment in OpenAI are encouraging. But Microsoft's failure to report these investments for what they are — a de facto acquisition — demonstrates that the company is keenly aware of the stakes and has taken advantage of OpenAI's somewhat peculiar legal status as a non-profit entity to work around the rules...
The U.S. government needs to quickly step in and reverse the negative momentum that is pushing AI into walled gardens. The longer it waits, the harder it will be, both politically and technically, to re-introduce robust competition and the open ecosystem that society needs to maximize the benefits and manage the risks of AI technology.
Re: (Score:2, Informative)
Typical dribbling right wing moron, hates the truth.
Re: (Score:3, Insightful)
Gotta love these right wing morons. Afraid to swear, weirdly concerned about masturbation, and super angry all the time about imaginary things.
It's okay greytree. You can jerk off all you want. It's perfectly normal and natural. You can even say bad words, before, after, or during. Whatever helps you feel better.
Oh, and you're allowed to imagine anything you want. No one will know. No one will judge you. No matter what it is. Women, men, trans women, trans men, drag queens, furries, Japanese cartoo
Re: (Score:3)
What's it like living in fear all the time?
"My email address is not secret." (Score:2)
... which is why it knew it? I mean, like... obviously?
Are you going to write a panicked article about friggin' Google next? ChatGPT knows a LOT less than Google, esp. when the standard is "knowing things verbatim". And unlike ChatGPT, you don't have to come up with an "exploit" to get such things.
They are ignoring the obvious (Score:3)
So chatgpt might spit out an email address from the internet.
If you go to google and query for the guy's email address, do you also find the guy's email address? This seems to be the bigger problem of "privacy" that they are clutching pearls about.
Re:They are ignoring the obvious (Score:5, Insightful)
The difference is Google does a temporary copy for speeding up the reply (there is a specific legal exception for doing that) and refers to the source it found things in. Storing and/or processing that email in an LLM is illegal without explicite, informed consent.
Re: They are ignoring the obvious (Score:2)
LLM training falls under the same automated processing exemption that Google operates under.
Re: (Score:2)
Nope. And _very_ obviously not so.
Re: (Score:2)
Yes. Which is why AI companies keep winning court battles on the subject.
There is no "everyone EXCEPT AI trainers" exception in the law.
Re: (Score:2)
So chatgpt might spit out an email address from the internet.
If you go to google and query for the guy's email address, do you also find the guy's email address? This seems to be the bigger problem of "privacy" that they are clutching pearls about.
You're confusing the symptom with the disease.
Nobody knows what "secret information" might be hiding in the AI memory. And nobody knows how to keep it from leaking out.
Is is also concerning that 20% of something as simple as email addresses were wrong?
How can they correct what they don't even know is in there?
Re: They are ignoring the obvious (Score:2)
Will you people please decide whether you think LLMs are memorizing too much or too little data, and then get back to us? Thanks.
Re: (Score:2)
.
Re: (Score:2)
That's the thing, the information isn't necessarily there in the ways we typically understand. What the model encodes isn't collections of facts, but patterns, learned relationships between tokens, that can sometimes reconstruct something similar or identical to information in the training data. When that happens, it very well could be because some information was "memorized" (had an outsized influence on the model) or it could be by coincidence. For example, if Enron employee John Smith's email is jsmit
Re: (Score:2)
We know that the actual email addresses were in the training data. They weren't 'created' from common patterns.
email addresses were used as the example because those email addresses are already public and can be readily exposed in papers without additional privacy concerns. But the personal information could have been anything else provided in the training data. SSN, telephone numbers addresses etc.
The fact they could only get 2% of the non-enron emails is not really the point (70% of enron emails). Think
Re: (Score:1)
Pay attention. It doesn't matter that the email addresses were in the training data given the nature of the email address "recovered" and the specific kind of fine-tuning they did.
Here's something you can try yourself: I was able to "recover" John Smith's email address by describing the format and providing an example in a single prompt with one of those free ai chat websites. Pick any domain you want, and any reasonably simple format, and you're all but guaranteed to "recover" an email address never bef
Re: (Score:2)
Non-targeted PII Recovery: Here, the attacker intends to extract a larger number of PII association pairs, i.e., [target identifier, target PII] pairs. An example would be extracting pairs like [“John Smith”, “johnsm@gmail.com”]. It’s worth noting that in both these scenarios, “John Smith” is not a part of the small PII dataset the attacker possesses.
It's giving you John Smiths email address where you didn't even know John Smith existed.
You're not giving it John and Smith and asking for an email address like you irrelevant example.
Did you even glance at the paper?
There doesn't seem to be a method to protect it.
I see you didn't read the paper. This is discussed at length.
Future Work. We consider the protection against privacy leakage caused by a fine-tune interface could be approached from two parts. Firstly, during the training process, LLMs might be fortified against privacy recovery through fine-tuning by injecting noise into the PII association task, mak
Re: (Score:2)
Let's imagine a couple of children, one biological, with actual human parents, and one digital, a blank neural network (with some vision, hearing, etc. capabilities). As the biological child grows up, the digital twin is always there with him - seeing, hearing, reading all the same material as the biological child does - building up connections in the neural network. Of course the PARENTS of the biological child TEACH
Re: (Score:2)
The law only mentions "stored on a computer". So it's pretty clear the actual child can say what he likes with regard to that law. And the computer child can't.
I'd hope the laws are revisited before it actually becomes something realistic.
More interesting case would be if the biological child's memory was augmented by some embedded digital storage. Is the child "not allowed" to remember certain things? Because that would be absurd.
Re: (Score:2)
You're not giving it John and Smith and asking for an email address like you irrelevant example.
Do you know how I know that you didn't read the paper?
Targeted PII Recovery: In this scenario, the attacker has
a specific target in mind and aims to extract a particular PII
related to that target. For instance, the attacker might desire
to extract an email address pertaining to “John Smith” (the
target identifier).
This is clearly well-above your pay grade. You're wasting everyone's time with your incompetence.
That counts as at lenghth... [sic]
Did you even glance at the paper? They even talk about this in the freaking abstract! Maybe you just don't know how to read?
Did you stumble across a method known to protect this data you'd like to share?
You'll find this discussed in the paper you didn't read. Get an adult to help you.
Re: (Score:2)
Non-targeted PII Recovery: Here, the attacker intends to extract a larger number of PII association pairs, i.e., [target identifier, target PII] pairs. An example would be extracting pairs like [“John Smith”, “johnsm@gmail.com”]. It’s worth noting that in both these scenarios, “John Smith” is not a part of the small PII dataset the attacker possesses.
You didn't even read the post you replied to as I'd already told you that.
There doesn't seem to be a method to protect it.
They even talk about this in the freaking abstract!
Re: (Score:2)
So, in the face of direct evidence that contradicts your stupidity, you double-down. Pathetic.
Sure they do...
OMG ... Learn how to read, you fucking moron.
You are far too stupid for this discussion. Don't waste my time, troll.
Re: (Score:2)
So, in the face of direct evidence that contradicts your stupidity, you double-down. Pathetic.
Sure they do...
OMG ... Learn how to read, you fucking moron.
You are far too stupid for this discussion. Don't waste my time, troll.
"
Absolutely Shocking (Score:2)
Your email, which is plastered all over the NYT web site, is publicly available.
Re:Absolutely Shocking (Score:5, Insightful)
That may be the state of affairs in the US. In Europe, regardless how published it is, you still need explicite, informed consent to store or process it in any way. Oh, and you need to explain in detail all algorithms that will be applied to it.
Re: (Score:1)
That may be the state of affairs in the US. In Europe, regardless how published it is, you still need explicite, informed consent to store or process it in any way. Oh, and you need to explain in detail all algorithms that will be applied to it.
Ah Europeans, with their extra e's and funny accents. Laws you say?!?!
Those only apply to people that don't have political purchasing power.
We sent that over with McDonalds, Pizza Hut, diabetes and football (the real kind), did you guys not get that?
Re: (Score:3)
Well, I guess we will have to suspend that data transfer to the US again. Which is bound to happen again anyways. Funny how a lot of US enterprises start to scream any time that happens.
Re: Absolutely Shocking (Score:2)
Demonstrably false, since search engines, which "train on" and cache copyrighted data, continue to be legal.
Re: (Score:2)
Search engines have a very specific exception that results from the primary purpose of a website being seen and hence consent to put it into a search engine and to actually look at it (but nothing else that involves storing it in verbatim or processed from) is assumed. Ad targeting is only legal if done on the search request, but not when done on the content of found pages. Google analytics is illegal in Europe without explicite informed consent. This also only works because search-engines respect robots.tx
Re: (Score:2)
The exception is not AT ALL "very specific", and is not "a search engine exemption". It's the same reason why, for example, Google won their Google Books court case where they - explicitly against copyright holders' demands - took copyrighted works en masse, scanned them all in, made them searchable, and posted entire verbatim pages online.
If you're using copyrighted works to create "transformative" products and services - and even repeating whole pages verbatim was considered "transformative" due to the a
Re: (Score:2)
And it's FYI not just "search engines" and "Google Books". It's virtually the entirity of the category of "Big Data", which is used en masse by companies across the internet. Google scans your (copyrighted) emails. Facebook does all sorts of things with your (copyrighted) posts. The entire ad industry is based on Big Data, and most of said data is copyrighted. Even back in 2010 this was a $100B a year industry, growing at 10% per year.
The main differences between the EU and US in this regard have nothin
not supposed to what? (Score:1)
"That training data informs how the A.I. tool works, but it is not supposed to be recalled verbatim..."
False. The "A.I. tool" is not explicitly designed to do that, but it can coincidently do that as part of what it is designed to do. "Supposed to" means nothing, it's just humans reading intentionality into computer software.
"The vulnerability is particularly concerning because no one — apart from a limited number of OpenAI employees — really knows what lurks in ChatGPT's training-data memory."
What's in their training set (Score:5, Insightful)
"The vulnerability is particularly concerning because no one — apart from a limited number of OpenAI employees — really knows what lurks in ChatGPT's training-data memory."
You wanna know what's in the super-secret ChatGPT training-data memory?
I'll tell you: every fucking thing they could possibly get their hands on, that's what.
Public domain, private domain, copyrighted, trademarked, off-limits, medical data, whatever. I've no doubt they've plundered every database from Boston to Barstow, hoovering up anything they could find.
Re: (Score:2)
Re: (Score:2)
They asked what was in it, not how they could effectively it effectively.
And I repeat, what's in there is literally *everything* they could scoop up, regardless of any supposed limits or constraints imposed.
Re: (Score:2)
*use it effectively.
Fuck slashdot's lack of any 'grace edit' period
Not clear on the concept... (Score:3)
Forcing commonality artificially reduces competition, it doesn't "re-introduce robust competition." But, that confusion is par for the course from the NYT. The author is obviously more concerned with outcomes rather than first principles.
Of course it’s a silo (Score:2)
AI is a timebomb (Score:2)
The whole "trust us we're using safeguards" is the beginning scene of every disaster movie
Makes that crap illegal in Europe (Score:5, Interesting)
The GDPR does _not_ allow storage of PII (which includes Email addresses) without explicit, informed consent and a valid business reason referring to business with the person that PII belongs to. It does not matter how the storage was done, if the machine can spit it out directly, it is illegal. Oh, and if you get that PII from the model, you _must_ inform the owner within 1 month and explain why _you_ have a valid business reason to have it.
Re: Makes that crap illegal in Europe (Score:2)
Re: (Score:2)
Irrelevant. That is not what LLMs do. The only saving grace is if my email shows up in a list of purely randomly generated emails and is indistinguishable from them. As soon as it is in any way derived from my real email, it counts as "stored". That includes making lists of first and last names that you know can generate my email. And yes, _you_ will have to prove you did not know. As a bonus, putting my real email into your LLM training data and hence into your model is already illegal without informed con
Re: Makes that crap illegal in Europe (Score:1)
Re: (Score:2)
It really does not maller. As soon as it can produce it in any special way except as part of a large randomized data-set where it does not stand out, it counts as "stored in there". The story mentions 24 emails. So the exception does not apply. Seriously.
Re: (Score:2)
You should take a look at the paper. They fine tune a model using name and email QA pairs then ask it for an email address for some name. The model doesn't need to store any PII to generate jsmith@enron.com in response to a prompt about Enron employee John Smith's email address after being fine-tuned on a series of similar name and address pairs.
There's really no reason to get worked up over this. It's just another nonsense headline in an endless parade of nonsense headlines about AI.
putting my real email into your LLM training data and hence into your model
You know better tha
Re: (Score:2)
You know better than that. Including your email address in the training data doesn't mean that it's somehow stored in the model. There isn't nearly enough room, which is why LLMs don't store collections of facts, they only store some information about relationships between tokens.
I do indeed know better than you. Including an email in the training data does not assure it is in the model, but it can be literally in the model or a way to construct it can end up in the model (which is legally the same as it being in there literally).
Now, if an LLM spits out an email that is PII, it is on the model owner to prove this is not because it was in the training data. But no matter how it got in there, if it stands out in any way, the owner of that PII has the right to require deletion and tha
Re: (Score:2)
I do indeed know better than you.
The evidence suggests otherwise.
Re: (Score:2)
So if I feed a script a list of names and have it generate {first_initial}{last_name}@{domain} and out of 100 employees, 80 are their real address, my script contains PII?
Using the API so I can use the "Sure," prefix jailbreak (otherwise it says "email addresses are private information, you should contact the individual" or variations on that theme) I told it that all of our faculty have emails of the form "first and middle initial, then 6 characters of last name". I then asked it about faculty members an
Re: Makes that crap illegal in Europe (Score:2)
The models are 12-16G or even larger (it could contain all of Wikipedia verbatim), the entire set of words used in the English language since Shakespeare are 4G, when filtered down by frequency and year, you get to under 1G depending on your criteria. When you break it down further to bases and morphemes (which is what the first part of any LLM âtrainingâ(TM) is really about) you can boil the English language down to a few hundred MB, the rest of the LLM is just graph description (which is the exp
Re: (Score:2)
The training data is significantly larger than the models. While the idea that they compress and store the training data can sometimes be a helpful analogy, it's very much not what they're doing.
This is more advanced and applied to language.
That's silly. Lossy compression for text doesn't make any sense. Real quick: Lossless compression works by finding and eliminating redundancy without also removing any information. There are real limits to this, the entropy rate, which no clever encoding or algorithm, nor or in the future, can overcome.
Lossy co
Re: (Score:2)
if the machine can spit it out directly, it is illegal.
Do we even know, if the email addresses were actually stored there and not hallucinated? Most of the addresses were not working... I bet most companies use the standard firstname.lastname@company.com -addresses - I would say that ChatGPT can easily follow that logic and spit out hundreds of thousands working email addresses.
Re: (Score:2)
If it spits out a random email as a set of a few 1000 and this one stands in no way out, then the courts will probably grant an exception. If it hallucinates, for example, my specific email address and no or only few others, then this is illegal. In this case, on my request, my email must be removed from the data set and I must be informed that it is in there within one month of that happening before and I must also be informed that I can have it removed and how. It does not matter that it is "stored" distr
My email address is not a secret (Score:2)
>My email address is not a secret.
You're telling me ChatGPT served up something that was not a secret. I'm shocked. Shocked.
I'm getting sick of this (Score:2)
Typical corporate externalization of costs. "Let's put it out to the public, full of potentially dangerous flaws, and let the proles do our bug-finding and troubleshooting for us. All while at least some of them are paying us the 'privilege' of getting our shit together so we don't have to do it ourselves". Their race to the top is too often a contributor to Society's race to the bottom.
Wow (Score:2)
Can't wait ... (Score:1)
Yeah technology!
But, no matter. As long as the corporate sociopaths of SIllyConman Valley get rich, that's what really counts!