Goldman Sachs is upending the way it does business with hedge funds, jettisoning less-profitable clients and increasing some fees on others as it adapts to new banking rules, people familiar with the matter said.

The bank has told hedge fund clients that the regulations have forced it to set aside more capital, crimping profits at its prime brokerage business, which executes and finances the funds' trades.