U.S. officials have said they haven't been able to determine Levinson's whereabouts, but have repeatedly brought up his disappearance with Iranian officials. Iran denies any knowledge, and has offered to help search for Levinson.
Levinson was seen in images sent to his family wearing an orange jump suit and chains in 2010 and 2011, but no trace of him has shown up since.
Let our news meet your inbox. The news and stories that matters, delivered weekday mornings.
Levinson's son slammed the Obama administration Tuesday, saying they're not doing enough.
"We're not getting any answers," Daniel Levinson said. "We have been abandoned. It's the worst feeling in the world."
Levinson, who suffers from diabetes, was working with the CIA on an unauthorized intelligence gathering mission when he disappeared on March 9, 2007.
His family learned of the other Americans' release on Saturday through the media, his son said, adding that they didn't hear from administration officials for hours afterward.
"I found out later than everybody else. That was just a devastating moment. My heart sunk. I was shaking for days," Daniel said. "I have been numb since."
State Department spokesman John Kirby said the government had intended to call the Levinsons before the news broke.
"The Iranians leaked the information early, too early for us to have made the phone calls and notifications that we wanted to make," Kirby told MSNBC. "Believe me, nobody is happy about the way that went down."
Saturday's swap freed four Americans held in Iran in exchange for clemency given to seven Iranians in U.S. sanctions cases. The Americans were Washington Post reporter Jason Rezaian, former Marine Amir Hekmati, Christian preacher Saeed Abedini, and Nosratollah Khosravi-Roodsari.
A fifth American, Matthew Trevithick, was released separately.
Kirby said Secretary of State John Kerry speaks to his counterparts in Iran at "every opportunity" to get information on Levinson, and said the U.S. isn't giving up on finding him.
"All Americans that are unjustly detained overseas matter to us," he said. "All of them matter equally."