Twitter Inc.’s former head of security filed a whistleblower complaint against the company, accusing it of failing to protect sensitive user data and lying about its security problems, just weeks ahead of the social-networking platform’s courtroom battle with Elon Musk.

Twitter's Ex-Security Head Files Whistleblower Complaint
Twitter's Ex-Security Head Files Whistleblower Complaint© Matt McClain/The Washington Post/Getty Images

Peiter Zatko, who was fired as Twitter’s head of security earlier this year, submitted the complaint last month to the Securities and Exchange Commission, according to a representative of Whistleblower Aid, an organization that helped file the claims. His submission says that he “uncovered extreme, egregious deficiencies by Twitter in every area of his mandate,” including privacy, digital and physical security, platform integrity and content moderation.

Among Mr. Zatko’s claims are that Twitter executives, including CEO Parag Agrawal, deliberately undercounted the prevalence of spam on the platform. Those claims could further complicate Twitter’s battle with Mr. Musk, whom the company sued in July to enforce a $44 billion takeover deal. Mr. Musk has alleged Twitter misrepresented its business, particularly as it relates to the level of spam or bot accounts—claims Twitter denies.

A five-day nonjury trial is slated to begin in October.

The existence of the whistleblower complaint was earlier reported by the Washington Post and CNN.

A Twitter spokeswoman said Mr. Zatko was fired “for ineffective leadership and poor performance” and that the complaint “is riddled with inconsistencies and inaccuracies and lacks important context.”

A lawyer for Mr. Musk said: “We have already issued a subpoena for Mr. Zatko, and we found his exit and that of other key employees curious in light of what we have been finding.”

Twitter shares were down roughly 5% in Tuesday intraday trading.

Mr. Zatko, a former hacker who is known as “Mudge,” has been a noted computer security researcher for decades. He was a member of a Boston cybersecurity collective that came to prominence in 1998 when it offered warnings about the state of national cybersecurity in testimony to the U.S. Senate. During one Senate hearing, the group famously told lawmakers they could take down the internet in 30 minutes.

Related video: Twitter whistleblower was on CNN 22 years ago. Here's what he had to say

Every day there's something new. There's new computers. There's new
Loaded: 8.88%
Current Time 0:02
Duration 3:22
Twitter whistleblower was on CNN 22 years ago. Here's what he had to say
0
View on Watch 

He was hired by Twitter in late 2020 after a career that included other corporate roles.

Mr. Zatko was brought into Twitter by co-founder Jack Dorsey in the wake of a high-profile hack by a teenager who bypassed the company’s securities systems. Mr. Dorsey “specifically recruited Mudge for his reputation of speaking truth to power,” according to the complaint.

Mr. Dorsey, however, was only a sporadic presence at the company, and the new hire—who had hundreds of staff reporting to him—was quickly overwhelmed by the task at hand, according to the complaint. At one point, Mr. Agrawal told his team, “Twitter has 10 years of unpaid security bills,” per the complaint.

The relationship between Mr. Zatko and Twitter’s leadership deteriorated over the subsequent months, according to both parties. Mr. Zatko helped oversee a critical report on Twitter’s ability to fight misinformation and spam, which other executives watered down, according to the complaint, which said Mr. Zatko was told by a Twitter lawyer that the changes were intended to hide the findings and prevent them from leaking internally or externally.

Much of the complaint deals with fake or spam accounts, a topic that Mr. Musk drew attention to in his takeover bid for Twitter.

Like the Tesla Inc. CEO, Mr. Zatko alleges that Twitter miscounts such users by focusing only on what are known as monetizable daily users, or MDAU, rather than all total daily users. The former category counts only those accounts that are thought to view advertising.

“There are many millions of active accounts that are not considered ‘mDAU,’ either because they are spam bots, or because Twitter does not believe it can monetize them,” Mr. Zatko’s complaint says. “These millions of non-mDAU accounts are part of the median user’s experience on the platform.”

Twitter has said it has a system for measuring users and spam that entails multiple human reviews of thousands of accounts sampled at random over time.

Mr. Zatko’s complaint said he attempted to formally notify Twitter’s board of his concerns but was steered off by Mr. Agrawal.

Twitter in 2011 reached an agreement with the Federal Trade Commission to maintain rigorous security, including limiting the number of employees with access to its key security and privacy controls. Mr. Zatko alleges that the company is in violation of that accord. The FTC didn’t respond to a request for comment.

Copies of the complaint were sent to the Senate Judiciary and Intelligence committees, aides of each panel said.

Democrats and Republicans have raised concerns about Twitter and other social-media firms in recent years over how they use and protect customer data, and have considered legislation that could require firms to adhere to certain data transparency or security standards. “If these claims are accurate, they may show dangerous data privacy and security risks for Twitter users around the world,” Sen. Dick Durbin (D., Ill.), chairman of the Judiciary Committee, said in a statement.

Corrections & Amplifications Parag Agrawal is the CEO of Twitter. An earlier version of this article incorrectly spelled his last name as Agarwal. (Corrected on Aug. 23)

Write to Sarah E. Needleman at sarah.needleman@wsj.com