US regulators could potentially impose penalties worth more than £5bn on Royal Bank of Scotland over the state-controlled UK lender’s sale of mortgage-backed securities that cost buyers of those instruments dear, sources have told The Times.