There is no mistaking the medium-security unit of the Butner Federal Correctional Complex, in Butner, N.C., for anything other than a prison. But Bernard Madoff could have done much worse. The grounds are carefully maintained, and some parts of the facility feel more like a college than a correctional institution.
That's where, on a crisp January morning this year, I spent two hours with Butner's most infamous inmate.
Madoff — who pleaded guilty in March 2009 to running a fraud of up to $65 billion at his investment firm — and I had been corresponding for several months by email, and in late November 2012, he agreed to allow me to visit. He insisted that our conversation be off the record, with the understanding that he would consider allowing me to report it at a later date.
A request to reveal the substance of our discussion now, on the fifth anniversary of his arrest, has gone unanswered. But I can say that the atmosphere of our meeting — at the corner of a large table in the warden's conference room — was unforgettable.
Three-and-a-half years into his 150-year sentence, the Bernie Madoff I met with that day looked more like an average 74-year-old man from Queens, N.Y., than a onetime Wall Street luminary.
His hair was shorter than in those infamous 2009 court appearances. The rimless glasses had been replaced by a tinted pair that appeared to have come from the prison. Madoff wore a standard-issue khaki prison uniform, with his shirtsleeves rolled halfway up his forearms. He carried a winter-weight outer jacket, also khaki, which he draped over a chair. I made a note that the coat seemed to be high quality, and comfortable. Madoff also appeared to have gained back the weight he lost before his sentencing.
But most striking was how utterly at ease he seemed. Always careful to profess his remorse, Madoff nonetheless was confident in his version of the facts — including who was and was not to blame.
At the time of our meeting, Madoff was engaged in a full-fledged effort to control the narrative. Unsatisfied with how he was portrayed in the handful of on-the-record interviews he had granted, his new strategy was an apparent attempt to spin a small group of reporters and business academics, mostly off the record.
But a handful of on-the-record emails he sent me in late 2012 and early 2013 — both before and after my in-person visit — were consistent with what I heard that day in January.
One email from Madoff less than a month after my visit to Butner is typical:
"From the day of my arrest I offered to assist in recovering the investment principal of my customers. I stated that I was confident that I would be able to convince those parties that were complicit in creating my financial problems, to return the money they withdrew from the investment advisory side of my firm. Those parties were well aware of the incriminating evidence I possessed about their complicit activity and wisely came forward with settlements. It was my belief that it was more important to use the evidence I had to pressure the complicit parties to settle, rather than to use this information for a lesser prison sentence for myself. As remorseful as I am for the pain and suffering I have shamefully caused, I take some comfort in the fact that my assistance will in fact accomplish what I have originally claimed, that with my assistance all of my customers will recover their original investment principal."
Trustee: Nearly $5 billion returned to customers, more to come
Irving Picard, the court-appointed trustee responsible for recovering Madoff's assets and returning them to investors, said Madoff has been no help in those efforts. Picard's counsel, David Sheehan, has accused Madoff of "aggrandizing" himself in an attempt to claim credit.
That accusation drew an angry email response from Madoff:
"I never said anything about not receiving 'credit.' My comments were simply stating that I am frustrated with both of these attorneys constantly making claims and statements that are false, and nobody in the media challenging them to present the evidence they claim they have. I would remind you all, that my statement at the start, that I believed I could help in recovering all my investors' original investment principal, were met with the accusations of my being 'DELUSIONAL.' "
According to Picard's office, as of Dec. 6, more than $9.5 billion has been recovered of roughly $17 billion in actual losses. About $4.9 billion has been returned to customers, and Picard said more money should begin flowing regularly to customers next spring.
"We're just hopeful that we're going to, every four to six months, be in a position to make some distributions," Picard, a partner at BakerHostetler, told CNBC in an interview.
A series of court rulings in his favor could free up some of the nearly $4.3 billion still held in the customer fund, as well as potentially collecting billions from other big targets Picard has sued. In fact, he now sees a clear path to recovering all the money.
"Even in the last couple of weeks, some of the defendants have come forward to inquire whether we'd be interested in settlements," said Picard, though he would not identify them.
"If they're reasonable and they're fair, we're always willing to talk," he said.
In late 2010, Picard sued Madoff's primary bank, JPMorgan Chase, for $19.9 billion, claiming that it knew or should have known about the fraud. JPMorgan denied the allegation, and most of Picard's claims have been dismissed. This October, he appealed to the U.S. Supreme Court, which has not determined if it will hear the case.
"Nobody protected the small investors"
As large as the amounts seem, they barely begin to make most victims whole — or at least what they believe to be whole.
Under Picard's formula (generally upheld by the courts), investors are entitled only to funds they invested with Madoff, minus any withdrawals. The returns they thought they had earned based on their statements are considered fictitious because Madoff has admitted to fabricating his trades.
The so-called net equity formula is devastating to investors such as Stephanie Halio of Boca Raton, Fla. She and her husband, both in their 70s, had to return to work — they started up a small limousine service — after learning they did not have nearly the nest egg they thought they had.
"I have to lift 50-pound suitcases, and that's the nature of the work," she said.
"If you had a passbook, and you went to your bank every month and you made deposits in your passbook and after 25 years you looked at your passbook and it showed a really nice tidy sum … you'd be pretty happy. Now all of a sudden the bank tells you, 'Oh no, you don't have that total amount. You only have what you deposited over 25 years.' That's shocking!" Halio said.
She would not quantify her losses but said she and her husband believed their funds were protected by the Securities and Exchange Commission and the Securities Investor Protection Corporation (SIPC).
"Nobody protected the small investors," Halio said.
But SIPC President Stephen Harbeck said the organization was not designed to cover investment losses but only to return securities and cash to investors in failed brokerage firms. In the Madoff case, in which the securities did not exist, the only fair solution is to return the original investments.
"These folks are victims," Harbeck said in an interview. "But the fact of the matter is that if you allow the measure of what is owed to people to be used by the last imaginary statement that Bernard Madoff pulled out of thin air, then you're letting the thief determine who wins and who loses."
In a February email to me, Madoff claimed that he has more information to offer in the quest to make investors whole.
"I have little doubt that the information I could provide would clearly demonstrate the vital role the major banks … played in the carrying out [of] my fraud, including their role in handling the accounts of my major customers."
Madoff's most recent email, in October, praised a New York Times report on the dismissal of a lawsuit in London against his sons Andrew and Mark, whom a judge ruled neither "knew of, or suspected, the fraud." The judge's ruling said "their honesty and integrity has been vindicated."
Madoff wrote that the inclusion of the judge's remarks in the article "will provide some comfort to our family."
Andrew Madoff, meanwhile, has not spoken to his father since Bernard Madoff's confession five years ago. Mark Madoff committed suicide in 2010, on the two-year anniversary of his father's arrest.
—By CNBC's Scott Cohn. Follow him on Twitter @ScottCohnCNBC.
(Read more: 10 white-collar fugitives on the FBI's wish list)