Jebali had threatened to quit if his plan for a non-partisan cabinet of technocrats to lead the north African country into early elections foundered.
In the end it was his own party, Ennahda, that rejected the proposal, prolonging the political stand-off that has cast a shadow over Tunisia's fledgling democracy and deepened an economic crisis.
"I vowed that if my initiative did not succeed, I would resign and ... I have already done so," Jebali told a news conference after meeting with President Moncef Marzouki.
Tunisia's deepest political crisis since the 2011 Arab Spring uprising that toppled President Zine El Abidine Ben Ali began when leading secular opposition politician Chokri Belaid was gunned down outside his home in Tunis on Feb. 6.
No one claimed responsibility for the killing, but it deepened the misgivings of secularists who believe Jebali's government has failed to deal firmly enough with religious extremists threatening the country's stability.
Protesters poured onto the streets in the following days and Marzouki's secularist party threatened to quit the coalition government.
Jebali said he would try to form a cabinet of apolitical technocrats to restore calm and take Tunisia to elections, but did not consult his Ennahda allies or their secular coalition partners before making the proposal.
Several secular politicians backed the plan but Ennahda, winner of most parliamentary seats in elections that followed Ben Ali's overthrow, opposed the idea, fearing it would be sidelined from power.
Country in 'real trouble'
Jebali bet his own job on the outcome, saying he would quit if he was rebuffed, and lost.
He quits 15 months into the job, although political experts said Marzouki was likely to re-appoint him as caretaker premier before a new leader is appointed.
Ennahda leader Rached Ghannouchi has said he wants to see Jebali head a new coalition. President Marzouki was due to meet Ghannouchi on Wednesday to ask him to name a prime minister.
But Jebali, announcing his resignation late on Tuesday, said he would not lead another government without assurances on the timing of fresh elections and a new constitution.
No government would be viable without Ennahda's blessing given its strength in parliament.
Ghannouchi has said it is essential that Islamists and secular parties share power now and in the future, and that his party was willing to compromise over control of important ministries such as foreign affairs, justice and interior.
"Ennahda is in negotiations with political parties to form a national coalition government", said Fethi Ayadi, a senior Ennahda official.
Iyed Dahmani, a leader of the secular Republican Party, said some kind of agreement was vital.
"We are in real trouble, politically and economically," he said.
The crisis has disrupted efforts to revitalize an economy hit hard by the disorder that followed the overthrow of veteran strongman Ben Ali.
Tunisia has been negotiating with the International Monetary Fund for a $1.78 billion loan and politicians said Jebali's inability to re-establish a functioning government had slowed efforts to restore normality.
Credit rating service Standard and Poor's said on Tuesday it had lowered its long-term foreign and local currency sovereign credit rating on Tunisia, citing "a risk that the political situation could deteriorate further amid a worsening fiscal, external and economic outlook."