One of the oldest hedge-fund firms is seeing an exodus of investors after years of lackluster returns and executive turnover.

Paloma Partners had so many investors ask to pull out their money that it decided to pay them partly in cash but mostly through the equivalent of IOUs, according to a recent letter to investors, a rare measure for a hedge fund. The firm said it didn’t have enough easy-to-sell assets on hand to immediately satisfy redemption requests while also maintaining diversification in its investment mix.