The company's board of directors elected President and Chief Operating Officer Tim Sloan to succeed him as CEO, while Lead Director Stephen Sanger will serve as the board's non-executive chairman.
The news comes after it was revealed that employees in Wells Fargo's community banking division opened about 2 million accounts without customer authorization, which resulted in the bank paying $185 million in penalties. Stumpf was grilled on Capitol Hill as he defended the bank's sales practices.
Stumpf said that he was "grateful for the opportunity to have led Wells Fargo" and is optimistic about the bank's future.
"While I have been deeply committed and focused on managing the Company through this period, I have decided it is best for the Company that I step aside," he said in a statement.
A spokesman for Wells Fargo said that there will be "no severance payment or agreement related" to Stumpf's departure.
"There are some retirement benefits that are detailed in our proxy statement. They are not accessible for the next 6 months. That is a normal lag time," the spokesman said in a statement.
Sanger said in a statement that while the retiring CEO helped create "create one of the strongest and most well-known financial services companies in the world," Stumpf "believes new leadership at this time is appropriate to guide Wells Fargo through its current challenges."
The board also said that Sloan was elected to the board and that independent director Elizabeth Duke will serve as its vice chair.
Last month, Sen. Elizabeth Warren accused Stumpf of "gutless leadership" and said he had not held himself accountable for the bank's actions. Before appearing on Capitol Hill, Stumpf told Jim Cramer on "Mad Money" that he would not resign.
The stock was last up more than 1 percent in extended trading.