The Securities and Exchange Commission’s plan to ban a range of controversial activities by private-fund managers marks a major change in the agency’s approach and likely will face industry challenges, experts say.

It may also reflect a view held by some within the agency that its current approach to policing the buyout industry needs to change.