Kenneth Fisher, founder of Fisher Investments. Jonathan Fickies | Bloomberg | Getty Images Individual investors at Fisher Investments transferred $20 million from the firm the week after the

billionaire made lewd comments at a conference, according to a research note from Mercer, an advisor to institutional investors. The development was disclosed on a conference call Fisher

executives held on Oct. 14 with Mercer, to discuss the fallout from founder Ken Fisher's comments at the Tiburon CEO Summit, according to the note obtained by CNBC. When speaking at the

conference on Oct. 8, Fisher had likened winning new clients to picking up women at a bar. He had used similar language at another conference in 2018. Though the billionaire apologized,

with the Securities and Exchange Commission. Private and retail clients have been "largely unconcerned," said Jill Hitchcock, senior executive vice president — private client group

at Fisher, in the Mercer note. Approximately 1% of those clients contacted raised concerns, with about $20 million in lost assets as a result, according to Mercer. Mercer declined to

comment further. A SMALL DENT While the divestitures sound large, it's barely a dent in the overall amount Fisher Investments manages. "To date, less than one percent of our

private/retail clients have contacted us in relation to this matter," said John Dillard, senior vice president and director of global public relations at Fisher. "Since Oct. 9, our

U.S. Private Client Group has experienced a net growth both in number of clients and assets managed," Dillard said. "We have been hired by new institutional clients as well as

hundreds of new private clients, representing billions of dollars in new assets under management," he said. The firm managed  $115 billion as of Oct. 31, Dillard said. > It may 

happen over time when clients feel that the time is right, > but if the performance is good, moving is too disruptive and they're > happy with their advisor – why leave? > 

> Dennis Gallant > senior analyst for Aite Group's wealth management practice Nevertheless, Fisher took to the media to respond to the controversy. Just last week, the firm

published a full-page advertisement in The New York Times, touting the number of women working at the advisory firm. "Over 800 women strong, with women leading 63% of employees,"

the ad said. The firm also started a website featuring testimonials from women who work there. "I think in any situation, companies will do whatever they can to resolve the issue and

stabilize clients and asset flow," said Dennis Gallant, senior analyst for Aite Group's wealth management practice. "But the one thing that's really hard is how much does

this impact assets over time," he said. "Will this blow over and will we see clients longer term reconsidering?" ASSET RETENTION Nick M. Do | E+ | Getty Images Fisher's

pension clients surprised many industry observers with their rapid move to divest, as retirement plan fiduciaries must show they underwent a prudent process before hiring or firing managers.

Individual investors can leave when they want. However, the process of swapping advisors is often an onerous one. "If you asked the average investor out there, 'Where do you want

to move?' Most wouldn't know where to go," Gallant said. "You'd have to go through a process, so switching is harder in the retail market." For starters, they

need to think about where they'll reinvest their assets, including finding a new advisor with whom they are compatible. Further, if the money they have invested at their current firm is

in a taxable account, they also need to consider whether a withdrawal will leave them with a tax bill. MORE FROM PERSONAL FINANCE: Six moves to prepare you financially for retirement

Everything you need to know about your 2020 taxes Here are the hidden benefits of a Roth IRA conversion Even high fees alone may not be enough to get retail clients to leave their advisor,

assuming they're happy with the service. "We've seen fee compression on the asset management side, but little evidence of it on the wealth management side so far," said

Marina Shtyrkov, a research analyst at Cerulli. Consider that Fisher charges an asset management fee of 1.25% for the first $1 million, with lower fees — known as breakpoints — for larger

accounts. Smaller accounts in the firm's WealthBuilder service — those with at least $200,000 in investible assets — are billed 1.5%. The average fee registered investment advisors

charge is around 1%, Shtyrkov said. "It may happen over time when clients feel that the time is right, but if the performance is good, moving is too disruptive and they're happy

with their advisor – why leave?" said Gallant.