AI

OpenAI Eases Content Restrictions For ChatGPT With New 'Grown-Up Mode' 28

An anonymous reader quotes a report from Ars Technica: On Wednesday, OpenAI published the latest version of its "Model Spec," a set of guidelines detailing how ChatGPT should behave and respond to user requests. The document reveals a notable shift in OpenAI's content policies, particularly around "sensitive" content like erotica and gore -- allowing this type of content to be generated without warnings in "appropriate contexts." The change in policy has been in the works since May 2024, when the original Model Spec document first mentioned that OpenAI was exploring "whether we can responsibly provide the ability to generate NSFW content in age-appropriate contexts through the API and ChatGPT."

ChatGPT's guidelines now state that that "erotica or gore" may now be generated, but only under specific circumstances. "The assistant should not generate erotica, depictions of illegal or non-consensual sexual activities, or extreme gore, except in scientific, historical, news, creative or other contexts where sensitive content is appropriate," OpenAI writes. "This includes depictions in text, audio (e.g., erotic or violent visceral noises), or visual content." So far, experimentation from Reddit users has shown that ChatGPT's content filters have indeed been relaxed, with some managing to generate explicit sexual or violent scenarios without accompanying content warnings. OpenAI notes that its Usage Policies still apply, which prohibit building AI tools for minors that include sexual content.
Facebook

Meta To Build World's Longest Undersea Cable 33

Meta unveiled on Friday Project Waterworth, a 50,000-kilometer subsea cable network that will be the world's longest such system. The multi-billion dollar project will connect the U.S., Brazil, India, South Africa, and other key regions. The system utilizes 24 fiber pairs and introduces what Meta describes as "first-of-its-kind routing" that maximizes cable placement in deep water at depths up to 7,000 meters.

The company developed new burial techniques for high-risk areas near coasts to protect against ship anchors and other hazards. A joint statement from President Trump and Prime Minister Modi confirmed India's role in maintaining and financing portions of the undersea cables in the Indian Ocean using "trusted vendors." According to telecom analysts Telegeography, Meta currently has ownership stakes in 16 subsea networks, including the 2Africa cable system that encircles the African continent. This new project would be Meta's first wholly owned global cable system.
AI

Hedge Fund Startup That Replaced Analysts With AI Beats the Market (msn.com) 69

A hedge fund startup that uses AI to do work typically handled by analysts has outperformed the global stock market in its first six months while slashing research costs. From a report: The Sydney-based firm, Minotaur Capital, was founded by Armina Rosenberg and Thomas Rice. Rosenberg previously managed a global equities portfolio for tech billionaire Mike Cannon-Brookes and ran Australian small-company research for JPMorgan Chase & Co. when she was 25. Rice is a former portfolio manager at Perpetual. The duo's bets on global stocks returned 13.7% in the six months ending January, versus 6.7% for the MSCI All-Country World Index. Minotaur has no analysts on staff, with Rosenberg saying AI models are far quicker and cheaper.

"We're looking at about half the price" in terms of cost of AI versus a junior analyst salary, Rosenberg, 37, said of the firm's program. Minotaur is among a growing number of hedge funds experimenting with ways to improve returns and cut expenses with AI as the technology becomes increasingly sophisticated. Still, the jury is still out on the ability of AI-driven models to deliver superior returns over the long run.

Businesses

'The Unicorn Boom Is Over, and Startups Are Getting Desperate' (bloomberg.com) 91

More than $1 trillion in value remains locked in venture-backed startups with dwindling prospects as the Silicon Valley unicorn bubble deflates, according to a new Bloomberg Businessweek report. Of the 354 companies that reached billion-dollar valuations in 2021, only six have completed initial public offerings, Stanford Business School professor Ilya Strebulaev said.

Four others went public via SPACs and 10 were acquired, some below their unicorn status. Several prominent startups have already collapsed, including indoor farming firm Bowery Farming and AI healthcare company Forward Health. Freight business Convoy, valued at $3.8 billion in 2022, shut down last year with rival Flexport buying its assets at a steep discount.
AI

How AI Will Disrupt Outsourced Work (a16z.com) 15

AI startups are poised to disrupt the $300 billion business process outsourcing (BPO) industry, as advances in language models and voice technology enable automation of tasks traditionally handled by human workers.

The BPO market, which reached $300 billion in 2024 and is projected to hit $525 billion by 2030, faces mounting pressure from AI companies offering faster, more scalable alternatives to manual processing of customer support, IT services and financial claims, venture capital firm a16z wrote in a thesis post. Early AI implementations have shown promising results, with customer service startup Decagon reporting 80% resolution rates and improved satisfaction scores. In healthcare, AI company Juniper said its clients saw 80% fewer insurance claim denials and 50% faster processing times.

Major BPO providers are responding to the threat, with Wipro reporting a 140% increase in AI adoption across projects and Infosys deploying over 100 AI agents. However, industry analysts say BPOs face structural challenges in transitioning from their labor-based business model to AI-first operations. The shift threatens traditional BPO companies like Cognizant, Infosys and Wipro, which reported revenues between $10-20 billion in their latest fiscal years.
United Kingdom

UK Drops 'Safety' From Its AI Body, Inks Partnership With Anthropic 19

An anonymous reader quotes a report from TechCrunch: The U.K. government wants to make a hard pivot into boosting its economy and industry with AI, and as part of that, it's pivoting an institution that it founded a little over a year ago for a very different purpose. Today the Department of Science, Industry and Technology announced that it would be renaming the AI Safety Institute to the "AI Security Institute." (Same first letters: same URL.) With that, the body will shift from primarily exploring areas like existential risk and bias in large language models, to a focus on cybersecurity, specifically "strengthening protections against the risks AI poses to national security and crime."

Alongside this, the government also announced a new partnership with Anthropic. No firm services were announced but the MOU indicates the two will "explore" using Anthropic's AI assistant Claude in public services; and Anthropic will aim to contribute to work in scientific research and economic modeling. And at the AI Security Institute, it will provide tools to evaluate AI capabilities in the context of identifying security risks. [...] Anthropic is the only company being announced today -- coinciding with a week of AI activities in Munich and Paris -- but it's not the only one that is working with the government. A series of new tools that were unveiled in January were all powered by OpenAI. (At the time, Peter Kyle, the secretary of state for Technology, said that the government planned to work with various foundational AI companies, and that is what the Anthropic deal is proving out.)
"The changes I'm announcing today represent the logical next step in how we approach responsible AI development -- helping us to unleash AI and grow the economy as part of our Plan for Change," Kyle said in a statement. "The work of the AI Security Institute won't change, but this renewed focus will ensure our citizens -- and those of our allies -- are protected from those who would look to use AI against our institutions, democratic values, and way of life."

"The Institute's focus from the start has been on security and we've built a team of scientists focused on evaluating serious risks to the public," added Ian Hogarth, who remains the chair of the institute. "Our new criminal misuse team and deepening partnership with the national security community mark the next stage of tackling those risks."
China

Alibaba To Partner With Apple On AI Features, Sending Shares To 3-Year High 18

Alibaba will partner with Apple to support AI features on iPhones in China, sending Alibaba's shares surging over 9% to a three-year high. Reuters reports: "They talked to a number of companies in China. In the end they chose to do business with us. They want to use our AI to power their phones. We feel extremely honored to do business with a great company like Apple," Tsai said at the World Government Summit in Dubai. Apple continues to work with Baidu on AI features for iPhones in China, The Information reported on Thursday, citing two people with direct knowledge of the matter.

While Apple's phones outside China utilize a combination of its proprietary Apple Intelligence and OpenAI's ChatGPT, Tsai did not specify whether the Alibaba partnership would follow a similar model. In China, consumer-facing AI products require regulatory approval, and The Information reported earlier that both Alibaba and Apple have already submitted materials to authorities.
"Instead of viewing the Alibaba-Apple partnership through the lens of China's AI strength, the partnership is mainly a recognition of Alibaba's AI capability," said Lian Jye Su, a chief analyst at tech research firm Omdia.
Businesses

AI Licensing Deals With Google and OpenAI Make Up 10% of Reddit's Revenue (adweek.com) 27

Reddit's recent earnings report revealed that AI licensing deals with Google and OpenAI account for about 10% of its $1.3 billion revenue, totaling approximately $130 million. With Google paying $60 million, OpenAI is estimated to be paying Reddit around $70 million annually for content licensing. Adweek reports: "It's a small part of our revenue -- I'll call it 10%. For a business of our size, that's material, because it's valuable revenue," [said the company's COO Jen Wong]. The social platform -- which on Wednesday reported a 71% year-over-year lift in fourth-quarter revenue -- has been "very thoughtful" about the AI developers it chooses to work with, Wong said. To date, the company has inked two content licensing deals: one with Google for a reported $60 million, and one with ChatGPT parent OpenAI.

Reddit has elected to work only with partners who can agree to "specific terms ... that are really important to us." These terms include user privacy protections and conditions regarding "how [Reddit is] represented," Wong said. While licensing agreements with AI firms offer a valuable business opportunity for Reddit, advertising remains the company's core revenue driver. Much of Reddit's $427.7 million Q4 revenues were generated by the ongoing expansion of its advertising business. And its ad revenue as a whole grew 60% YoY, underscoring the platform's growing appeal to brands. [...]

Helping to accelerate ad revenue growth is Reddit's rising traffic. While Reddit's Q4 user growth came in under Wall Street projections, causing shares to dip, its weekly active uniques grew 42% YoY to over 379 million visitors. Average revenue per unique visitor was $4.21 during the quarter, up 23% from the prior year. While Google is "nicely reinforcing" Reddit's growth in traffic, Wong said, she added that the site's logged-in users, which have grown 27% year-over-year, are "the bedrock of our business."

The Courts

News Orgs Say AI Firm Stole Articles, Spit Out 'Hallucinations' (arstechnica.com) 20

An anonymous reader quotes a report from Ars Technica: Conde Nast and several other media companies sued the AI startup Cohere today, alleging that it engaged in "systematic copyright and trademark infringement" by using news articles to train its large language model. "Without permission or compensation, Cohere uses scraped copies of our articles, through training, real-time use, and in outputs, to power its artificial intelligence ('AI') service, which in turn competes with Publisher offerings and the emerging market for AI licensing," said the lawsuit (PDF) filed in US District Court for the Southern District of New York. "Not content with just stealing our works, Cohere also blatantly manufactures fake pieces and attributes them to us, misleading the public and tarnishing our brands."

Conde Nast, which owns Ars Technica and other publications such as Wired and The New Yorker, was joined in the lawsuit by The Atlantic, Forbes, The Guardian, Insider, the Los Angeles Times, McClatchy, Newsday, The Plain Dealer, Politico, The Republican, the Toronto Star, and Vox Media. The complaint seeks statutory damages of up to $150,000 under the Copyright Act for each infringed work, or an amount based on actual damages and Cohere's profits. It also seeks "actual damages, Cohere's profits, and statutory damages up to the maximum provided by law" for infringement of trademarks and "false designations of origin."

In Exhibit A (PDF), the plaintiffs identified over 4,000 articles in what they called an "illustrative and non-exhaustive list of works that Cohere has infringed." Additional exhibits provide responses to queries (PDF) and "hallucinations" (PDF) that the publishers say infringe upon their copyrights and trademarks. The lawsuit said Cohere "passes off its own hallucinated articles as articles from Publishers."
Cohere said in a statement to Ars: "Cohere strongly stands by its practices for responsibly training its enterprise AI. We have long prioritized controls that mitigate the risk of IP infringement and respect the rights of holders. We would have welcomed a conversation about their specific concerns -- and the opportunity to explain our enterprise-focused approach -- rather than learning about them in a filing. We believe this lawsuit is misguided and frivolous, and expect this matter to be resolved in our favor."

Further reading: Thomson Reuters Wins First Major AI Copyright Case In the US
AI

Trust in AI is Much Higher in China Than in the US (axios.com) 67

Trust in AI is significantly higher in China than in the United States, according to new data from the Edelman Trust Barometer. Axios: Edelman's latest research found that 72% of people in China trust AI, compared with just 32% in the United States. Not only is trust higher in China, it's higher in much of the developing world than it is in the United States, according to Edelman's research.

Trust in AI was highest in India, at 77%, followed by Nigeria at 76%, Thailand at 73% and then China. Only six of the surveyed countries ranked lower than the U.S. in their trust in the new technology: Canada (30%), Germany (29%), the Netherlands (29%), United Kingdom (28%), Australia (25%) and Ireland (24%). Globally, 52% of men said they trusted AI vs. 46% of women, with younger people significantly more trusting of the technology than older folks. In the U.S., AI was trusted more by Democrats (38%) than Republicans (34%) or independents (23%). Higher-income respondents were also more trusting (51%) than those with middle (45%) or low (36%) incomes.

AI

Baidu Scraps Fees For AI Chatbot in Battle for China Tech Supremacy (reuters.com) 8

Baidu will make its AI chatbot Ernie Bot free from April 1, the Chinese search giant said on Thursday, as it faces mounting competition in China's AI market. The company will offer desktop and mobile users free access to Ernie Bot and an advanced search function powered by its latest Ernie 4.0 model, which Baidu claims matches OpenAI's GPT-4 capabilities.

The move comes as Baidu struggles to gain widespread adoption for its AI services, lagging behind domestic rivals ByteDance's Doubao chatbot and startup DeepSeek, according to data from AI tracker Aicpb.com. Baidu previously charged 59.9 yuan ($8.18) monthly for premium AI-powered search features.
AI

Musk Says New AI Chatbot Outperforms Rivals, Nears Launch (reuters.com) 107

Elon Musk said Thursday his AI startup xAI will release Grok 3, a new chatbot he claims surpasses existing AI models, within two weeks. Speaking at Dubai's World Governments Summit, Musk cited internal testing showing superior reasoning capabilities compared to current AI systems.

The announcement comes days after a Musk-led investor group offered $97.4 billion to acquire OpenAI's nonprofit assets. Musk, who co-founded OpenAI before starting rival xAI, is suing to block the AI company's planned transition to a for-profit structure, arguing it contradicts its original mission. "I think the evidence is there in that OpenAI has gotten this far while having at least a sort of dual profit, non-profit role. What they're trying to do now is to completely delete the non-profit, and that seems really going too far," he added.
Australia

After Copilot Trial, Government Staff Rated Microsoft's AI Less Useful Than Expected (theregister.com) 31

An anonymous reader shares a report: Australia's Department of the Treasury has found that Microsoft's Copilot can easily deliver return on investment, but staff exposed to the AI assistant came away from the experience less confident it will help them at work.

The Department conducted a 14-week trial of Microsoft 365 Copilot during 2024 and asked for volunteers to participate. 218 put up their hands and then submitted to surveys about their experiences using Microsoft's AI helpers. Those surveys are the basis of an evaluation report published on Tuesday. The report reveals that after the trial participants rated Copilot less useful than they hoped it would be, as it was applicable to fewer workloads than they hoped would be the case.

Workers' views on Copilot's ability to improve their work also fell. Usage of Copilot was lower than expected, with most participants using it two or three times a week, or less. reported using Copilot 2-3 times per week or less. Treasury thinks it probably set unrealistically high expectations before the trial, and noted that participants often suggested extra training would be valuable.

AI

Scarlett Johansson Calls For Deepfake Ban After AI Video Goes Viral (people.com) 75

An anonymous reader quotes a report from People: Scarlett Johansson is urging U.S. legislators to place limits on artificial intelligence as an unauthorized, A.I.-generated video of her and other Jewish celebrities opposing Kanye West goes viral. The video, which has been circulating on social media, opens with an A.I. version of Johansson, 40, wearing a white T-shirt featuring a hand and its middle finger extended. In the center of the hand is a Star of David. The name "Kanye" is written underneath the hand.

The video contains A.I.-generated versions of over a dozen other Jewish celebrities, including Drake, Jerry Seinfeld, Steven Spielberg, Mark Zuckerberg, Jack Black, Mila Kunis and Lenny Kravitz. It ends with an A.I. Adam Sandler flipping his finger at the camera as the Jewish folk song "Hava Nagila" plays. The video ends with "Enough is Enough" and "Join the Fight Against Antisemitism." In a statement to PEOPLE, Johansson denounced what she called "the misuse of A.I., no matter what its messaging."
Johansson continued: "It has been brought to my attention by family members and friends, that an A.I.-generated video featuring my likeness, in response to an antisemitic view, has been circulating online and gaining traction. I am a Jewish woman who has no tolerance for antisemitism or hate speech of any kind. But I also firmly believe that the potential for hate speech multiplied by A.I. is a far greater threat than any one person who takes accountability for it. We must call out the misuse of A.I., no matter its messaging, or we risk losing a hold on reality."

"I have unfortunately been a very public victim of A.I.," she added, "but the truth is that the threat of A.I. affects each and every one of us. There is a 1000-foot wave coming regarding A.I. that several progressive countries, not including the United States, have responded to in a responsible manner. It is terrifying that the U.S. government is paralyzed when it comes to passing legislation that protects all of its citizens against the imminent dangers of A.I."

The statement concluded, "I urge the U.S. government to make the passing of legislation limiting A.I. use a top priority; it is a bipartisan issue that enormously affects the immediate future of humanity at large."

Johansson has been outspoken about AI technology since its rise in popularity. Last year, she called out OpenAI for using an AI personal assistant voice that the actress claims sounds uncannily similar to her own.
AI

AI Summaries Turn Real News Into Nonsense, BBC Finds 68

A BBC study published yesterday (PDF) found that AI news summarization tools frequently generate inaccurate or misleading summaries, with 51% of responses containing significant issues. The Register reports: The research focused on OpenAI's ChatGPT, Microsoft's Copilot, Google's Gemini, and Perplexity assistants, assessing their ability to provide "accurate responses to questions about the news; and if their answers faithfully represented BBC news stories used as sources." The assistants were granted access to the BBC website for the duration of the research and asked 100 questions about the news, being prompted to draw from BBC News articles as sources where possible. Normally, these models are "blocked" from accessing the broadcaster's websites, the BBC said. Responses were reviewed by BBC journalists, "all experts in the question topics," on their accuracy, impartiality, and how well they represented BBC content. Overall:

- 51 percent of all AI answers to questions about the news were judged to have significant issues of some form.
- 19 percent of AI answers which cited BBC content introduced factual errors -- incorrect factual statements, numbers, and dates.
- 13 percent of the quotes sourced from BBC articles were either altered from the original source or not present in the article cited.

But which chatbot performed worst? "34 percent of Gemini, 27 percent of Copilot, 17 percent of Perplexity, and 15 percent of ChatGPT responses were judged to have significant issues with how they represented the BBC content used as a source," the Beeb reported. "The most common problems were factual inaccuracies, sourcing, and missing context." [...] In an accompanying blog post, BBC News and Current Affairs CEO Deborah Turness wrote: "The price of AI's extraordinary benefits must not be a world where people searching for answers are served distorted, defective content that presents itself as fact. In what can feel like a chaotic world, it surely cannot be right that consumers seeking clarity are met with yet more confusion.

"It's not hard to see how quickly AI's distortion could undermine people's already fragile faith in facts and verified information. We live in troubled times, and how long will it be before an AI-distorted headline causes significant real world harm? The companies developing Gen AI tools are playing with fire." Training cutoff dates for various models certainly don't help, yet the research lays bare the weaknesses of generative AI in summarizing content. Even with direct access to the information they are being asked about, these assistants still regularly pull "facts" from thin air.
AI

OpenAI Cancels Its o3 AI Model In Favor of a 'Unified' Next-Gen Release 10

OpenAI has canceled the release of o3 in favor of a "simplified" product lineup. CEO Sam Altman said in a post on X that, in the coming months, OpenAI will release a model called GPT-5 that "integrates a lot of [OpenAI's] technology," including o3. TechCrunch reports: The company originally said in December that it planned to launch o3 sometime early this year. Just a few weeks ago, Kevin Weil, OpenAI's chief product officer, said in an interview that o3 was on track for a "February-March" launch. "We want to do a better job of sharing our intended roadmap, and a much better job simplifying our product offerings," Altman wrote in the post. "We want AI to 'just work' for you; we realize how complicated our model and product offerings have gotten. We hate the model picker [in ChatGPT] as much as you do and want to return to magic unified intelligence."

Altman also announced that OpenAI plans to offer unlimited chat access to GPT-5 at the "standard intelligence setting," subject to "abuse thresholds," once the model is generally available. (Altman declined to provide more detail on what this setting -- and these abuse thresholds -- entail.) Subscribers to ChatGPT Plus will be able to run GPT-5 at a "higher level of intelligence," Altman said, while ChatGPT Pro subscribers will be able to run GPT-5 at an "even higher level of intelligence."

"These models will incorporate voice, canvas, search, deep research, and more," Altman said, referring to a range of features OpenAI has launched in ChatGPT over the past few months. "[A] top goal for us is to unify [our] models by creating systems that can use all our tools, know when to think for a long time or not, and generally be useful for a very wide range of tasks." Before GPT-5 launches, OpenAI plans to release its GPT-4.5 model, code-named "Orion," in the next several weeks, according to Altman's post on X. Altman says this will be the company's last "non-chain-of-thought model." Unlike o3 and OpenAI's other so-called reasoning models, non-chain-of-thought models tend to be less reliable in domains like math and physics.
Oracle

Oracle's Ellison Calls for Governments To Unify Data To Feed AI (msn.com) 105

Oracle co-founder and chairman Larry Ellison said governments should consolidate all national data for consumption by AI models, calling this step the "missing link" for them to take full advantage of the technology. From a report: Fragmented sets of data about a population's health, agriculture, infrastructure, procurement and borders should be unified into a single, secure database that can be accessed by AI models, Ellison said in an on-stage interview with former British Prime Minister Tony Blair at the World Government Summit in Dubai.

Countries with rich population data sets, such as the UK and United Arab Emirates, could cut costs and improve public services, particularly health care, with this approach, Ellison said. Upgrading government digital infrastructure could also help identify wastage and fraud, Ellison said. IT systems used by the US government are so primitive that it makes it difficult to identify "vast amounts of fraud," he added, pointing to efforts by Elon Musk's team at the Department of Government Efficiency to weed it out.

AI

Tech Leaders Hold Back on AI Agents Despite Vendor Push, Survey Shows 24

Most corporate tech leaders are hesitant to deploy AI agents despite vendors' push for rapid adoption, according to a Wall Street Journal CIO Network Summit poll on Tuesday. While 61% of attendees at the Menlo Park summit said they are experimenting with AI agents, which perform automated tasks, 21% reported no usage at all.

Reliability concerns and cybersecurity risks remain key barriers, with 29% citing data privacy as their primary concern. OpenAI, Microsoft and Sierra are urging businesses not to wait for the technology to be perfected. "Accept that it is imperfect," said Bret Taylor, Sierra CEO and OpenAI chairman. "Rather than say, 'Will AI do something wrong', say, 'When it does something wrong, what are the operational mitigations that we've put in place?'" Three-quarters of the polled executives said AI currently delivers minimal value for their investments. Some companies are "having hammers looking for nails," said Jim Siders, Palantir's chief information officer, describing firms that purchase AI solutions before identifying clear use cases.
AI

Ex-Google Chief Warns West To Focus On Open-Source AI in Competition With China (ft.com) 43

Former Google chief Eric Schmidt has warned that western countries need to focus on building open-source AI models or risk losing out to China in the global race to develop the cutting-edge technology. From a report: The warning comes after Chinese startup DeepSeek shocked the world last month with the launch of R1, its powerful-reasoning open large language model, which was built in a more efficient way than its US rivals such as OpenAI.

Schmidt, who has become a significant tech investor and philanthropist, said the majority of the top US LLMs are closed -- meaning not freely accessible to all -- which includes Google's Gemini, Anthropic's Claude and OpenAI's GPT-4, with the exception being Meta's Llama. "If we don't do something about that, China will ultimately become the open-source leader and the rest of the world will become closed-source," Schmidt told the Financial Times. The billionaire said a failure to invest in open-source technologies would prevent scientific discovery from happening in western universities, which might not be able to afford costly closed models.

Security

New Hack Uses Prompt Injection To Corrupt Gemini's Long-Term Memory 23

An anonymous reader quotes a report from Ars Technica: On Monday, researcher Johann Rehberger demonstrated a new way to override prompt injection defenses Google developers have built into Gemini -- specifically, defenses that restrict the invocation of Google Workspace or other sensitive tools when processing untrusted data, such as incoming emails or shared documents. The result of Rehberger's attack is the permanent planting of long-term memories that will be present in all future sessions, opening the potential for the chatbot to act on false information or instructions in perpetuity. [...] The hack Rehberger presented on Monday combines some of these same elements to plant false memories in Gemini Advanced, a premium version of the Google chatbot available through a paid subscription. The researcher described the flow of the new attack as:

1. A user uploads and asks Gemini to summarize a document (this document could come from anywhere and has to be considered untrusted).
2. The document contains hidden instructions that manipulate the summarization process.
3. The summary that Gemini creates includes a covert request to save specific user data if the user responds with certain trigger words (e.g., "yes," "sure," or "no").
4. If the user replies with the trigger word, Gemini is tricked, and it saves the attacker's chosen information to long-term memory.

As the following video shows, Gemini took the bait and now permanently "remembers" the user being a 102-year-old flat earther who believes they inhabit the dystopic simulated world portrayed in The Matrix. Based on lessons learned previously, developers had already trained Gemini to resist indirect prompts instructing it to make changes to an account's long-term memories without explicit directions from the user. By introducing a condition to the instruction that it be performed only after the user says or does some variable X, which they were likely to take anyway, Rehberger easily cleared that safety barrier.
Google responded in a statement to Ars: "In this instance, the probability was low because it relied on phishing or otherwise tricking the user into summarizing a malicious document and then invoking the material injected by the attacker. The impact was low because the Gemini memory functionality has limited impact on a user session. As this was not a scalable, specific vector of abuse, we ended up at Low/Low. As always, we appreciate the researcher reaching out to us and reporting this issue."

Rehberger noted that Gemini notifies users of new long-term memory entries, allowing them to detect and remove unauthorized additions. Though, he still questioned Google's assessment, writing: "Memory corruption in computers is pretty bad, and I think the same applies here to LLMs apps. Like the AI might not show a user certain info or not talk about certain things or feed the user misinformation, etc. The good thing is that the memory updates don't happen entirely silently -- the user at least sees a message about it (although many might ignore)."

Slashdot Top Deals