After 90 percent of the votes were counted, Netanyahu's center-right Likud party was expected to win 31 seats, while its main rival the centrist Blue and White party was pushing ahead on 32 seats, according to the Central Election Committee.
Both Netanyahu's right-wing bloc and Blue and White's centrist and left-wing block were set to fall short of the 61 seats needed to form a governing coalition in the 120-seat parliament, known as the Knesset.
Final results are expected Wednesday and could swing either way.
A close ally of President Donald Trump, Netanyahu is likely to continue to pursue his hard-line stance toward Palestinians if he remains prime minister, putting a two-state solution further out of reach.
Blue and White leader Benny Gantz, a former army chief of staff, has campaigned on a promise of clean government and social harmony. He has called for pursuing peace with Palestinians while maintaining Israeli security.
Also in play is the fact that Netanyahu could potentially be indicted in three corruption cases. If he remains prime minister, Netanyahu may be able to pass legislation that would grant him immunity, but if he loses he may have to appear in court and even face time in jail.
Early Wednesday morning, Netanyahu refused to concede defeat and vowed to form a new government that excludes Arab parties.
Let our news meet your inbox. The news and stories that matters, delivered weekday mornings.
“In the coming days we will convene negotiations to assemble a strong Zionist government and to prevent a dangerous anti-Zionist government,” he said. He claimed that Arab parties “negate the existence of Israel as a Jewish and democratic state” and “glorify bloodthirsty murderers.”
Amid reports Wednesday afternoon that Netanyahu had canceled his trip to the United Nations General Assembly in New York, an Israel government official told NBC News: "Due to political circumstances, the prime minister is not traveling to UNGA."
Facing the end of his political career, and potentially even jail, Netanyahu pulled out all the stops throughout the campaign.
Once again this time both main parties could struggle to form a government with like-minded partners, potentially delivering another inconclusive result less than six months after the last one.
In order to form the next government, one party will have to cobble together a coalition that commands 61 out of the 120 seats in the Israeli parliament. It is usually the leader of the largest party that gets to attempt to form a government within the 42 days allocated.
No party has ever won a majority in Israeli politics.
However if two parties tie, as happened in April and may happen this time around, Israel’s president will ask the leader he believes has the greatest chance of forming a government after consulting with party leaders.
If the election produces a deadlock, one option would be a national unity government involving both Likud and the Blue White Party. But Gantz has previously said that he would not join a government with Netanyahu at the helm when he is facing potential corruption charges.
Israel’s attorney general is expected to decide whether to formally charge the prime minister by the end of the year, after a pre-trial hearing in October.
Speaking to reporters outside his house early Wednesday Lieberman, who has emerged as the country's unlikely kingmaker, said he saw a national unity government as the only solution to the potential impasse with his party as the glue. He has called for an immediate start to negotiations.
“There is one option — a national unity government,” he said.
Partial results indicate that Lieberman, a former defense and foreign minister, is on course to win nine seats — four more than he had in the last Parliament — meaning all eyes remain on him as a potential solution to a deadlock.
Paul Goldman reported from Tel Aviv, and Saphora Smith from London.
Saphora Smith is a London-based reporter for NBC News Digital.
Paul Goldman is a Tel Aviv-based producer and video editor for NBC News.
Lawahez Jabari, Josh Lederman and Associated Press contributed.