Eliot Spitzer, the New York attorney-general, has served notice on the Royal Bank of Canada, according to people familiar with the case.
Mr. Spitzer's action comes after a Financial Times investigation found banks are deliberately hanging on to overpayments worth hundreds of millions of dollars made by customers.
Individual overpayments may be small, but research shows they can mount up to large sums. They are also sometimes used to inflate earnings.
Regulators and governments in several countries are weighing enforcement and legislative action to deal with the practice, which happens far from public scrutiny in banks' back offices.
The Financial Services Authority, the U.K. financial watchdog, was warned more than a year ago that managers at RBC were withholding money from customers that had overpaid on foreign-exchange transactions.
Confidential RBC e-mails, memos, letters, account statements, transactions and bank-to-bank communications seen by the FT, as well as FSA documents, show warnings came from a member of the internal investigations team in RBC's London capital-markets operations.
She told the FSA that managers had tried to cover up evidence by altering records just before an internal audit into her claims. She also complained to the U.K. regulator that she felt intimidated and discriminated against for complaining.
The woman first raised the alarm in 1996, when she looked into a case showing that managers had siphoned off $50,000 a Turkish bank had overpaid on a $2.8 million foreign exchange deal transaction. Documents seen by the FT show RBC created a new account and the $50,000 was transferred to the account.
After further transactions emerged involving overpayments by U.S. fund manager Franklin Templeton, she took her claims to Gordon Nixon, RBC president and chief executive, turning to the FSA after being rebuffed.
RBC said it would not comment on particular transactions, citing client confidentiality. But it told the FT its daily foreign-exchange trading volumes — 15,000 transactions — were worth more than $1 billion and that "perhaps 5 percent or less involve some form of discrepancy."
It added: "To be very explicit about overpayments, RBC has a process for investigating them and upon completion of an investigation . . . if it is determined that there was a payment error made by the counterparty, the funds are remitted."
A spokesman denied knowledge of any FSA investigation. A confidential internal statement of the bank's position, prepared by an internal ombudsman last year, said: "The hiding of these amounts in the bank's books would have no formidable impact on overall results."
The FSA would not comment on whether any investigation was ongoing or whether any action was taken against the bank.