The epicenter of what may be the largest Ponzi scheme in history was the 17th floor of the Lipstick Building, an oval red-granite building rising 34 floors above Third Avenue in Midtown Manhattan.

A busy stock-trading operation occupied the 19th floor, and the computers and paperwork filled the 18th floor of Bernard L. Madoff Investment Securities.

But the 17th floor was Bernie Madoff’s sanctum, occupied by fewer than two dozen staff members and rarely visited by other employees. They called it the “hedge fund” floor, but U.S. prosecutors now say the work Madoff did there was actually a fraud scheme whose losses Madoff himself estimates at $50 billion.

The tally of reported losses climbed through the weekend to nearly $20 billion, with a giant Spanish bank, Banco Santander, reporting on Sunday that clients of one of its Swiss subsidiaries have lost $3 billion. Some of the biggest losers were members of the Palm Beach Country Club, where many of Madoff’s wealthy clients were recruited.

The list of prominent fraud victims grew as well. According to a person familiar with the business of the real estate and publishing magnate Mort Zuckerman, he is also on a list of victims that already included the owners of the New York Mets, a former owner of the Philadelphia Eagles and the chairman of GMAC.

And the 17th floor is now an occupied zone, as investigators and forensic auditors try to piece together what Madoff did with the billions entrusted to him by individuals, banks and hedge funds around the world.

So far, only Madoff, the firm’s 70-year-old founder, has been arrested in the scandal. He is free on a $10 million bond and cannot travel far outside the New York area.

But a question still dominates the investigation: How one person could have pulled off such a far-reaching, long-running fraud, carrying out all the simple practical chores the scheme required, like producing monthly statements, annual tax statements, trade confirmations and bank transfers.

Firms managing money on Madoff’s scale would typically have hundreds of people involved in these administrative tasks. Prosecutors say he claims to have acted entirely alone.

“Our task is to find the records and follow the money,” said Alexander Vasilescu, a lawyer in the New York office of the Securities and Exchange Commission. As of Sunday night, he said, investigators could not shed much light on the fraud or its scale. “We do not dispute his number — we just have not calculated how he made it,” he said.

Scrutiny is also falling on the many banks and money managers who helped steer clients to Madoff and now say they are among his victims.

While many investors were friends or met Madoff at country clubs or on charitable boards, even more had entrusted their money to professional advisory firms that, in turn, handed it on to Madoff — for a fee.

Investors are now questioning whether these paid advisers were diligent enough in investigating Madoff to ensure that their money was safe. Where those advisers work for big institutions like Banco Santander, investors will most likely look to them, rather than to the remnants of Madoff’s firm, for restitution.

Santander may face $3.1 billion in losses through its Optimal Investment Services, a Geneva-based fund of hedge funds that is owned by the bank. At the end of 2007, Optimal had 6 billion euros, or $8 billion, under management, according to the bank’s annual report — which would mean that its Madoff investments were a substantial part of Optimal’s portfolio.

A spokesman for Santander declined to comment on the case.

Other Swiss institutions, including Banque Bénédict Hentsch and Neue Privat Bank, acknowledged being at risk, with Hentsch confirming about $48 million in exposure.

BNP Paribas said it had not invested directly in the Madoff funds but had 350 million euros, or about $500 million, at risk through trades and loans to hedge funds. And the private Swiss bank Reichmuth said it had 385 million Swiss francs, or $327 million, in potential losses. HSBC, one of the world’s largest banks, also said it had made loans to institutions that invested in Madoff but did not disclose the size of its potential losses.

Losses of this scale simply do not seem to fit into the intimate business that Madoff operated in New York.

With just over 200 employees, it was tight-knit and friendly, according to current and former employees. Madoff was gregarious and empathetic, known for visiting sick employees in their hospitals and hosting warmly generous staff parties.

By the elevated standards of Wall Street, the Madoff firm did not pay exceptionally well, but it was loyal to employees even in bad times. Madoff’s family filled the senior positions, but his was not the only family at the firm — generations of employees had worked for Madoff.

Even before Madoff collapsed, some employees were mystified by the 17th floor. In recent regulatory filings, Madoff claimed to manage $17 billion for clients — a number that would normally occupy a staff of at least 200 employees, far more than the 20 or so who worked on 17.

One Madoff employee said he and other workers assumed that Madoff must have a separate office elsewhere to oversee his client accounts.

Nevertheless, Madoff attracted and held the trust of companies that prided themselves on their diligent investigation of investment managers.

One of them was Walter Noel Jr., who struck up a business relationship with Madoff 20 years ago that helped earn his investment firm, the Fairfield Greenwich Group, millions of dollars in fees.

Indeed, over time, one Fairfield’s strongest selling points for its largest fund was its access to Madoff.

But now, Noel and Fairfield are the biggest known losers in the scandal, facing potential losses of $7.5 billion, more than half its assets.

Jeffrey Tucker, a Fairfield co-founder and former U.S. regulator, said in a statement posted on the firm’s Web site: “We have worked with Madoff for nearly 20 years, investing alongside our clients. We had no indication that we and many other firms and private investors were the victims of such a highly sophisticated, massive fraudulent scheme.”

The huge loss comes at a time when the hedge fund industry has already been wounded by the volatile markets. Several weeks ago, Fairfield had halted investor redemptions at two of its other funds, citing the tough market conditions as dozens of hedge funds have done. The firm reported a drop of $2 billion in assets between September and November.

Fairfield was founded in 1983 by Noel, the former head of international private banking at Chemical Bank, and Tucker, a former Securities and Exchange Commission official. It grew dramatically over the years, attracting investors in Europe, Latin America and Asia.

Noel first met Madoff in the 1980s, and Fairfield’s fortunes grew along with the returns Madoff reported. The two men were very different: Madoff hailed from eastern Queens and was tied closely to the Jewish community, while Noel, a native of Tennessee, moved in the Greenwich social scene with his wife, Monica.

“Walter was always really confident in Bernie and the strategy he employed,” said one hedge fund manager who declined to be named because for fear of jeopardizing his relationship with Noel.

“He was a person of superb ethics, and this has to cut him to the quick,” said George Ball, a former executive at E. F. Hutton and Prudential-Bache Securities who knows Noel.

Fairfield touted its investigative skills. On its Web site, the firm claimed to investigate hedge fund managers for six to 12 months before investing. As part of the process, a team of examiners conducted personal background checks, audited brokerage records and trading reports and interviewed hedge fund executives and compliance officials.

In 2001, Madoff called Fairfield and invited the firm to inspect his books after two news reports questioned the validity of his returns, according to a person close to Fairfield. Outside auditors hired to inspect Madoff’s operations concluded that “everything checked out,” this person said.

“FGG performed comprehensive and conscientious due diligence and risk monitoring,” Marc Kasowitz, a lawyer for Fairfield, said in a statement. “FGG like so many other Madoff clients was a victim of a highly-sophisticated massive fraud that escaped the detection of top institutional and private investors, industry organizations, auditors, examiners, and regulatory authorities.”

Now, Fairfield is seeking to recover what it can from Madoff.

“It is our intention to aggressively pursue the recovery of all assets related to Bernard L. Madoff Investment Securities,” Tucker said in a statement.

Working alongside the U.S. investigators on Madoff’s 17th floor, staffers for Lee Richards 3d, the court-appointed receiver for the firm, are trying to determine what parts of the firm can keep operating to preserve assets for investors.

A hotline number had been posted on the company Web site, madoff.com, but on Sunday night, Richards said that there was little reason to call.

“We don’t have anything to report to investors at this time,” he said. “We are doing everything we can to protect the assets of the Madoff entities that are subject to the receivership, and to learn what we can about the operations of those entities.”