The retail brokerage business in the U.S. has become a game about scale: It's dominated by a handful of large firms that have devoted themselves to delivering services cheaply.