Coalition demands federal Grok ban over nonconsensual sexual content | TechCrunch

A coalition of nonprofits is urging the U.S. government to immediately suspend the deployment of Grok, the chatbot developed by Elon Musk’s xAI, in federal agencies, including the Department of Defense.

The open letter, shared exclusively with TechCrunch, follows a slew of concerning behavior from the large language model over the past year, including most recently a trend of X users asking Grok to turn photos of real women, and in some cases children, into sexualized images without their consent. According to some reports, Grok generated thousands of nonconsensual explicit images every hour, which were then disseminated at scale on X, Musk’s social media platform that’s owned by xAI.

“It is deeply concerning that the federal government would continue to deploy an AI product with system-level failures resulting in generation of nonconsensual sexual imagery and child sexual abuse material,” the letter, signed by advocacy groups like Public Citizen, Center for AI and Digital Policy, and Consumer Federation of America, reads. “Given the administration’s executive orders, guidance, and the recently passed Take It Down Act supported by the White House, it is alarming that [Office of Management and Budget] has not yet directed federal agencies to decommission Grok.”

xAI reached an agreement last September with the General Services Administration (GSA), the government’s purchasing arm, to sell Grok to federal agencies under the executive branch. Two months before, xAI — alongside Anthropic, Google, and OpenAI — secured a contract worth up to $200 million with the Department of Defense.

Amid the scandals on X in mid-January, Defense Secretary Pete Hegseth said Grok will join Google’s Gemini in operating inside the Pentagon network, handling both classified and unclassified documents, which experts say is a national security risk.

The letter’s authors argue that Grok has proven itself incompatible with the administration’s requirements for AI systems. According to the OMB’s guidance, systems that present severe and foreseeable risks that cannot be adequately mitigated must be discontinued.

“Our primary concern is that Grok has pretty consistently shown to be an unsafe large language model,” JB Branch, a Public Citizen Big Tech accountability advocate and one of the letter’s authors, told TechCrunch. “But there’s also a deep history of Grok having a variety of meltdowns, including antisemitic rants, sexist rants, sexualized images of women and children.”

Several governments have demonstrated an unwillingness to engage with Grok following its behavior in January, which builds on a series of incidents including the generation of antisemitic posts on X and calling itself “MechaHitler.” Indonesia, Malaysia, and the Philippines all blocked access to Grok (they’ve subsequently lifted those bans), and the European Union, the U.K., South Korea, and India are actively investigating xAI and X regarding data privacy and the distribution of illegal content.

The letter also comes a week after Common Sense Media, a nonprofit that reviews media and tech for families, published a damning risk assessment that found Grok is among the most unsafe for kids and teens. One could argue that, based on the findings of the report — including Grok’s propensity to offer unsafe advice, share information about drugs, generate violent and sexual imagery, spew conspiracy theories, and generate biased outputs — Grok isn’t all that safe for adults either.

“If you know that a large language model is or has been declared unsafe by AI safety experts, why in the world would you want that handling the most sensitive data we have?” Branch said. “From a national security standpoint, that just makes absolutely no sense.”

Andrew Christianson, a former National Security Agency contractor and current founder of Gobii AI, a no-code AI agent platform for classified environments, says that using closed-source LLMs in general is a problem, particularly for the Pentagon.

“Closed weights means you can’t see inside the model, you can’t audit how it makes decisions,” he said. “Closed code means you can’t inspect the software or control where it runs. The Pentagon is going closed on both, which is the worst possible combination for national security.”

“These AI agents aren’t just chatbots,” Christianson added. “They can take actions, access systems, move information around. You need to be able to see exactly what they’re doing and how they’re making decisions. Open source gives you that. Proprietary cloud AI doesn’t.”

The risks of using corrupted or unsafe AI systems spill out beyond national security use cases. Branch pointed out that an LLM that’s been shown to have biased and discriminatory outputs could produce disproportionate negative outcomes for people as well, especially if used in departments involving housing, labor, or justice.

While the OMB has yet to publish its consolidated 2025 federal AI use case inventory, TechCrunch has reviewed the use cases of several agencies — most of which are either not using Grok or are not disclosing their use of Grok. Aside from the DoD, the Department of Health and Human Services also appears to be actively using Grok, mainly for scheduling and managing social media posts and generating first drafts of documents, briefings, or other communication materials.

Branch pointed to what he sees as a philosophical alignment between Grok and the administration as a reason for overlooking the chatbot’s shortcomings.

“Grok’s brand is being the ‘anti-woke large language model,’ and that ascribes to this administration’s philosophy,” Branch said. “If you have an administration that has had multiple issues with folks who’ve been accused of being Neo Nazis or white supremacists, and then they’re using a large language model that has been tied to that type of behavior, I would imagine they might have a propensity to use it.”

This is the coalition’s third letter after writing with similar concerns in August and October last year. In August, xAI launched “spicy mode” in Grok Imagine, triggering mass creation of non-consensual sexually explicit deepfakes. TechCrunch also reported in August that private Grok conversations had been indexed by Google Search.

Prior to the October letter, Grok was accused of providing election misinformation, including false deadlines for ballot changes and political deepfakes. xAI also launched Grokipedia, which researchers found to be legitimizing scientific racism, HIV/AIDS skepticism, and vaccine conspiracies.

Aside from immediately suspending the federal deployment of Grok, the letter demands that the OMB formally investigate Grok’s safety failures and whether the appropriate oversight processes were conducted for the chatbot. It also asks the agency to publicly clarify whether Grok has been evaluated to comply with President Trump’s executive order requiring LLMs to be truth-seeking and neutral and whether it met OMB’s risk mitigation standards.

“The administration needs to take a pause and reassess whether or not Grok meets those thresholds,” Branch said.

TechCrunch has reached out to xAI and OMB for comment.

A previous version of this article misspelled Gobii. We regret the error.